using curl within foreach loop optimization? - php

Is it bad practice or will it be slower if I use curl within a foreach loop?
I'm planning on having an autocomplete input field, and the query in the input would be sent to an API call.
I'm getting an id from a certain link (ie: http://api.linke1.com/names)
foreach($json as j){
$id = $j->id; //from http://api.linke1.com/names
$url = "https://api.site/{$id}/photos";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$output = curl_exec($ch);
curl_close($ch);
$jsonDecode = json_decode($output);
$results = $jsonDecode->results;
foreach($results as $result)
{
$photoURL= $result->photo->url; //from https://api.site/{$id}/photos
}
}
So every time I type in a name, it will go into the foreach searching for an id from http://api.linke1.com/names, and then it will look for the photo url from the other link. I wanted to output a list of an array, so eventually i'll have a list of data to output showing information such as name, photo, etc...
Will this slow down dramatically because each letter typed in the input field it will run through this foreach loop. Would there be an easier way?
Thanks!

Initialize the curl and the things that doesn't change before the loop and close it afterwards.
That will speed up the thing a little bit.
and you can use curl_multi_*, which can fetch several URLs in parallel.
http://se2.php.net/manual/en/ref.curl.php

Related

PHP Update with Curl Please

I am trying to update my API with an update curl function but am struggling to work out why it isn't working
The areas where it may be wrong is key($id) I want it to
extract the ID column based on the key value for the ID array.
$URL I want to create the URL based on the const variables plus the resource name plus the value of the ID array that has been passed through rawurlencode.
So far this is my update code, but am wondering what area is wrong.
I can provide more info if needed and appreciate any help, thanks
<?php
function update(array $id,array $vaules, $resourcename)
$jsonData = json_encode($vaules);
key($id);
$url = DOMAIN.FOLDER.APIPATH.$resourcename.rawurlencode("/".$id);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER,array ('content-type: application/json'));
curl_setopt($ch, CURLOPT_CUSTOMREQUEST,PUT);
curl_setopt($ch,CURLOPT_POSTFIELDS,$jsonData);
curl_exec($ch);
curl_getinfo(CURLINFO_HTTP_CODE);
}
The function key() returns the current key in an array (according to the internet pointer). Right now you're not doing anything with it, you're calling the function and not assigning it anywhere.
Did you mean to write: rawurlencode("/".key($id).$vaules);?
As your code is right now, assuming $id is an array, you're trying to convert an array into a string, which I doubt is what you want.

Twitch TV channel information: doesnt work

I want to get information about a channel, is it online at the moment or not:
$stream_list = ...;
$mycurl = curl_init();
curl_setopt ($mycurl, CURLOPT_HEADER, 0);
curl_setopt ($mycurl, CURLOPT_RETURNTRANSFER, 1);
//Build the URL
$url = "http://api.justin.tv/api/stream/list.json?channel=" . $stream_list;
curl_setopt ($mycurl, CURLOPT_URL, $url);
$web_response = curl_exec($mycurl);
but thats always return with an empty array. I saw many example based on it - mine wont work, what am I doing wrong?
An empty array probably means nothing was found with the stream list you provided.
I used http://api.justin.tv/api/stream/list.json?channel=beyondthesummit,towelliee and was able to get an array from the API, and then I used http://api.justin.tv/api/stream/list.json?channel=foobar and got an empty JSON array back.
I'd make sure $stream_list has the value you expect. And if it does, try removing the channel filter completely to see if you get results.
It returns an empty array if the channel is not live.

Web automation using PHP/cURL

I want to access to the several pages of this list. My try is the code below with which I'm getting an xml file with the data of the hotels of the first page, but I want to access to the pages where the rest of the hotels are..how to do that?
As you can imagine the url for all the pages is the same.
<?php
//extract data from the post
extract($_POST);
//set POST variables
$url = 'http://www.turismovenezia.it/index.php';
$fields1 = array(
'ajax'=>'searchEngineTopdata',
'next_pair'=>'Dove Allogiare|*',
'lang'=>'it');
$fields2 = array(
'ajax'=>'xmlSearchEngineResponder',
'xml' => "%3C%3Fxml%20version%3D%221.0%22%3F%3E%3CSearchRequest%20xmlns%3D%22http%3A%2F%2Fwww.liberologico.com%2Fdbsite%2Fjolly-search%22%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema-instance%22%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FScope%3E%3CFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Eaptve_territorio%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3ETHESAURUS%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Efull_text_search%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Elang%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5Bit%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3EEQUAL%3C%2FOperation%3E%3C%2FFilters%3E%3C%2FFilters%3E%3CSubSearches%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BEventi%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BArte%20%26%20Cultura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BMare%20%26%20Natura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BPiatti%20%26%20Prodotti%20tipici%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BRelax%20%26%20Divertimento%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Mangiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BInformazioni%20Utili%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3C%2FSubSearches%3E%3C%2FSearch%3E%3CActiveResultSet%3E%3CTab%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FTab%3E%3CFirstItem%3E0%3C%2FFirstItem%3E%3CPagerSize%3E10%3C%2FPagerSize%3E%3C%2FActiveResultSet%3E%3C%2FSearchRequest%3E",
'force' => 'false');
//open connection
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields2);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
//execute post
$result = curl_exec($ch);
echo $result;
//close connection
curl_close($ch);
Javi
First of all why are you configuring curl twice?
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST, true);
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields2);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
Then when you do the first call to first page with $fields2 (The long XML string), the call should return an XML (like you say) and in this XML response there is a field ItemCount which contain the total number of hotels.
And if you look at the long string of XML you send with $fields2 there is a field call 'FirstItem' where contain 0 for the first call. This field is you offset, you can increment it to skip hotels.
Example:
$fields2 = array(
'ajax'=>'xmlSearchEngineResponder',
'xml' => "%3C%3Fxml%20version%3D%221.0%22%3F%3E%3CSearchRequest%20xmlns%3D%22http%3A%2F%2Fwww.liberologico.com%2Fdbsite%2Fjolly-search%22%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema-instance%22%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FScope%3E%3CFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Eaptve_territorio%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3ETHESAURUS%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Efull_text_search%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Elang%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5Bit%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3EEQUAL%3C%2FOperation%3E%3C%2FFilters%3E%3C%2FFilters%3E%3CSubSearches%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BEventi%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BArte%20%26%20Cultura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BMare%20%26%20Natura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BPiatti%20%26%20Prodotti%20tipici%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BRelax%20%26%20Divertimento%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Mangiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BInformazioni%20Utili%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3C%2FSubSearches%3E%3C%2FSearch%3E%3CActiveResultSet%3E%3CTab%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FTab%3E%3CFirstItem%3E0%3C%2FFirstItem%3E%3CPagerSize%3E10%3C%2FPagerSize%3E%3C%2FActiveResultSet%3E%3C%2FSearchRequest%3E",
'force' => 'false');
Return you the first 10 Results;
$fields2 = array(
'ajax'=>'xmlSearchEngineResponder',
'xml' => "%3C%3Fxml%20version%3D%221.0%22%3F%3E%3CSearchRequest%20xmlns%3D%22http%3A%2F%2Fwww.liberologico.com%2Fdbsite%2Fjolly-search%22%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2001%2FXMLSchema-instance%22%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FScope%3E%3CFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Eaptve_territorio%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3ETHESAURUS%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Efull_text_search%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5B%2A%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3ELIKE%3C%2FOperation%3E%3C%2FFilters%3E%3CFilters%20xsi%3Atype%3D%22FilterSpecType%22%3E%3CField%3Elang%3C%2FField%3E%3CValue%3E%3CSingleValue%3E%3C%21%5BCDATA%5Bit%5D%5D%3E%3C%2FSingleValue%3E%3C%2FValue%3E%3CMode%3EFREE_TEXT%3C%2FMode%3E%3COperation%3EEQUAL%3C%2FOperation%3E%3C%2FFilters%3E%3C%2FFilters%3E%3CSubSearches%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BEventi%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BArte%20%26%20Cultura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BMare%20%26%20Natura%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BPiatti%20%26%20Prodotti%20tipici%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BRelax%20%26%20Divertimento%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BDove%20Mangiare%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3CSearch%3E%3CScope%3E%3C%21%5BCDATA%5BInformazioni%20Utili%5D%5D%3E%3C%2FScope%3E%3C%2FSearch%3E%3C%2FSubSearches%3E%3C%2FSearch%3E%3CActiveResultSet%3E%3CTab%3E%3C%21%5BCDATA%5BDove%20Alloggiare%5D%5D%3E%3C%2FTab%3E%3CFirstItem%3E10%3C%2FFirstItem%3E%3CPagerSize%3E10%3C%2FPagerSize%3E%3C%2FActiveResultSet%3E%3C%2FSearchRequest%3E",
'force' => 'false');
Will return you the next 10 results. Also there is a field call PagerSize thant can allow you to retrieve more result at once.
So i would do the first call to get the total number of hotel, and the loop to get all the other pages.
//do first curl call
//then
$totalHotel = 5214; // To retrieve in the first call
$increment = 10; // the number of hotel to treat at once
$nbOfHotelimported = $increment;
while($totalHotel-$increment){
// do another curl call
// with FirstItem set to $nbOfHotelimported
// and pageSizer set to $increment
$nbOfHotelimported += $increment;
}
Instead of scraping the initial HTML page itself, you should be fetching the URLs that the site itself accesses using AJAX. You can find out exactly how you should structure your request by using your browser's developer tools to snoop on the AJAX requests made when you request another result page and "copy" them.
By the way, depending on the why and how this might not be the most ethical or legal thing to do.

CURL grab information and log in with it

i got a problem.
i want to log in a website with CURL, but the page generates a key, which is in a hidden field. So I have to grab the value of the hidden field... after that i have to submit the password, the email AND the the grabbed key.
Is that possible?
hope you understand
Thanks
----edit----
if the page reload, there's a new key in the hidden field.. and the old one do not work
Yes, it's possible. The basic steps would be:
1. Fetch form
2. Run returned data through DOMdocument to extract the hidden form field's value
3. Post login data, including the key value from step #2
4. ???
5. Profit
Yes, it is.
Load the page contents into a variable (eg. $contents = file_get_contents(...);).
Now, parse it so you can get the hidden key (eg. $matches = preg_match(..., ...); $key = $matches[1];).
Now, post the request using the hidden key (eg. $context = streamcontextcreate(...); $data = file_get_contents(..., false, $context);).
Yes, it is possible.
You can even grab the page with
$source = htmlspecialchars(file_get_contents($url));
Then use strpos, substr and so on to get information of that field (that's the easiest way)
Then just POST it with this function
$urltopost = $url
$datatopost = array ($name => $value);
$ch = curl_init ($urltopost);
curl_setopt ($ch, CURLOPT_POST, true);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $datatopost);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$returndata = htmlspecialchars(curl_exec ($ch));

website query, php

How can I query a particular website with some fields and get the results to my webpage using php?
let say website xyz.com will give you the name of the city if you give them the zipcode. How can I acehive this easliy in php? any code snap shot will be great.
If I understand what you mean (You want to submit a query to a site and get the result back for processing and such?), you can use cURL.
Here is an example:
<?php
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "example.com");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
?>
You can grab the Lat/Long from this site with some regexp like this:
if ( preg_match_all( "#<td>\s+-?(\d+\.\d+)\s+</td>#", $output, $coords ) ) {
list( $lat, $long ) = $coords[1];
echo "Latitude: $lat\nLongitude: $long\n";
}
Just put that after the curl_close() function.
That will return something like this (numbers changed):
Latitude: 53.5100
Longitude: 60.2200
You can use file_get_contents (and other similar fopen-class functions) to do this:
$result = file_get_contents("http://other-site.com/query?variable=value");
Do you mean something like:
include 'http://www.google.com?q=myquery'; ? or which fields do you want to get?
Can you be a bit more specific pls :)
If you want to import the html to your page and analyze it, you probably want to use cURL.
You have to have the extensions loaded to your page (it's usually part of PHP _ I think it has to be compiled in? The manual can answer that)
Here is a curl function. Set up your url like
$param='fribby';
$param2='snips';
$url="www.example.com?data=$param&data2=$param2";
function curl_page($url)
{
$response =false;
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_FAILONERROR,true);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch,CURLOPT_TIMEOUT,30);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
$page_data=curl_page($url);
Then, you can get data out of the page using the DOM parsing or grep/sed/awk type stuff.

Categories