I want to CURL to Google to see how many results it returns for a certain search.
I've tried this:
$url = "http://www.google.com/search?q=".$strSearch."&hl=en&start=0&sa=N";
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible;)");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
$response = curl_exec($ch);
curl_close($ch);
But it just returns a 405 Method Allowed google error.
Any ideas?
Thanks
Use a GET request instead of a POST request. That is, get rid of
curl_setopt($ch, CURLOPT_POST, true);
Or even better, use their well defined search API instead of screen-scraping.
Scrapping Google is a very easy thing to do. However, if you don't require more than the first 30 results, then the search API is preferable (as others have suggested). Otherwise, here's some sample code. I've ripped this out of a couple of classes that I'm using so it might not be totally functional as is, but you should get the idea.
function queryToUrl($query, $start=null, $perPage=100, $country="US") {
return "http://www.google.com/search?" . $this->_helpers->url->buildQuery(array(
// Query
"q" => urlencode($query),
// Country (geolocation presumably)
"gl" => $country,
// Start offset
"start" => $start,
// Number of result to a page
"num" => $perPage
), true);
}
// Find first 100 result for "pizza" in Canada
$ch = curl_init(queryToUrl("pizza", 0, 100, "CA"));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_USERAGENT, $this->getUserAgent(/*$proxyIp*/));
curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$response = curl_exec($ch);
Note: $this->_helpers->url->buildQuery() is identical to http_build_query except that it will drop empty parameters.
Use the Google Ajax API.
http://code.google.com/apis/ajaxsearch/
See this thread for how to get the number of results. While it refers to c# libraries, it might give you some pointers.
Before scrapping data please read https://support.google.com/websearch/answer/86640?rd=1
Against google terms
Automated traffic includes:
Sending searches from a robot, computer program, automated service, or search scraper
Using software that sends searches to Google to see how a website or webpage ranks on Google
CURLOPT_CUSTOMREQUEST => ($post)? "POST" : "GET"
Related
I am currently trying to implement apple pay for the web which is quite the troublesome topic it seems. When I try to start a session according to the docs (https://developer.apple.com/documentation/apple_pay_on_the_web/apple_pay_js_api/requesting_an_apple_pay_payment_session) using Curl in PHP I receive a handshake error.
I don't know how to solve that. There is a very limited set of parameters available (link above) and nothing I tried so far worked.
My domain has TLS 1.3 and an overall ranking of A when I check via ssllabs.com
My call is currently looking like this
$url = 'https://apple-pay-gateway.apple.com/paymentservices/startSession';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query(['merchantIdentifier' => '##my-merchant-id##', 'displayName' => 'something', 'initiative' => 'web', 'initiativeContext' => '##my-domain-name##']));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
I am sending a call to the YouTube API using search.list (retrieve the first search result for a keyword) but I am getting nothing in return.
Here is my call:
$api_url = 'https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=1&type=video&key=MYKEY&&format=json&q=SEARCHTERMS;
Note: MYKEY is my API key I got from Google (currently active) and SEARCHTERMS is any word to search.
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $api_url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$response = curl_exec($ch);
curl_close($ch);
$data = json_decode($response);
$value = json_decode(json_encode($data), true);
echo $value;
The URL works correctly (tested on browser) and I'm not getting any kind of error (console), but still this curl request isn't echoing any data from YouTube. Isn't it properly set?
P.s. My quota isn't exceeded.
SOLVED!
Watch out: youtube API doesn't read spaces in url (eg. in search terms), so you gotta streplace all of them with "+"
well, if you're confused so am i, so to make it better for you to understand what i'd like to do this is what i think
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://google.com/a-good-coffee');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
//sleep(6);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'blablabla123abc');
echo $func = curl_exec($ch);
let's say $func will return this
{ "id": "great_coffee", "object": "amazing", "billing_details": google }
what i do want to achieve is this
i want $func only to read the text in the second " " which is "great_coffee", the result will randomize each time i curl the same page (lets say it's gonna be a_good_life next time i curl)
so i dont want to get a specific value, ?
*Is this possible ? and how could i do it if it is ?
Thank you for your responses, but if i use json_decode i will get the id yes, but i will still get some blank lines
Yes, the first thing you'd want to do is decode the result into an array using json_decode Then you can get the array values only by using the array_values function. From there you just need to get the second index.
i tried getting the contents of this url
http://www.google.com/trends/fetchComponent?q=ex%20machina&cid=TIMESERIES_GRAPH_0&export=3
with file_get_contents and curl but it returns nothing, however when i went directly to the link it shows the content so its not a problem with api limits etc.
It worked once, but not after subsequent calls
heres the curl method i used
function get_data($url) {
$ch = curl_init();
//$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXYPORT, 3128);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I had a similar problem with the Google Maps Geocoder API in combination with file_get_contents, you can find the solution here:
Google Maps Geocode API only works via browser
Try this url:
http://www.google.com/trends/fetchComponent?q=ex+machina&cid=TIMESERIES_GRAPH_0&export=3
I have been trying to post some variables to a site using POST method, using curl to get some results. I am posting to this link.
http://www.rasta.pk/Lhr/Lhr_Traffic.aspx
At this page you will see a drop down menu .. onchange some values are returned in "Yellow" colored table.
I have monitored this site and trying to get those results by making a post request to that link. But, I am getting "Bad Header" error. I have tried changing things but ubable to find a solution.
Here is my code:
"Canal Bank Rd",
"ScriptManager1 " => "UpdatePanelDDLRoads|DropDownRoads",
"__EVENTARGUMENT" => "",
"__EVENTTARGET" => "DropDownRoads"
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headerz);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_REFERER, "http://www.rasta.pk/Lhr/Lhr_Traffic.aspx");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
?>
There's too little details presented for us to know for sure.
A guess is that you're doing the wrong kind of post, since when you pass in a hash array to CURLOPT_POSTFIELDS it will do a multipart formpost which might not be what the server expects. Pass in a string instead to make a "normal" POST.
If that is not enough, use LiveHTTPHeaders or similar in a browser to figure out exactly what is sent in a "manual" session and then you make sure that your curl program mimics that operation as closely as possible.