php scraping excat search with quotation marks "batman kill a human" - php

I have a little PHP script for scraping google. I wnat to google the exact search "batman kill a human" I pass the param with simple quoation marks "batman kill a human"
and I can see a correct URL generated.
I call the script with the param '"batman kill a human"', and I get >Error 400 (Bad Request)
example
root#ubuntu:/var/www/html# php ejemplo.php '"batman kill a human"'
------------- [https://www.google.es/search?q="batman kill a human"]
------------- [
Error 400 (Bad Request)!!1 *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding
....
...
-------------------------------------------
if i try to use this url in firefox, https://www.google.es/search?q="batman kill a human" y get a correct answer from google.
Why i don't get a correct answer when I'm trying it from the php scritp
this is the source code.
<?php
include('simple_html_dom.php');
function file_get_contents_curl($url) {
/*
This is a file_get_contents replacement function using cURL
One slight difference is that it uses your browser's idenity
as it's own when contacting google.
*/
$ch = curl_init();
// curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION , 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$busca=$argv[1];
$cadena="https://www.google.es/search?q=$busca";
print "\n------------- [$cadena]";
$data=file_get_contents_curl($cadena);
$html = str_get_html($data);
print "\n------------- [$html]";
$html->clear(); exit();
?>

Check out the PHP manual on urlencode()
$busca=$argv[1];
becomes
$busca=urlencode($argv[1]);
Which renders as:
https://www.google.es/search?q=batman+kills+a+human

Related

PHP - get the returned value with CURL between two sites

I have two PHP sites, an API site (x.php) and another, which calls the API site with CURL (y.php)
The x.php looks like this:
if (isset($_POST['testconnection'])) {
return "ok";
}
And the y.php like this:
$host = "https://there.is.the/x.php";
$command = array (
"testconnection" => "testconnection"
);
$ch=curl_init($host);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($command));
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_TIMEOUT,180);
$response = curl_exec($ch);
curl_close($ch);
var_dump($response);
As you can see in the example, I would like to get the return string from x.php in the y.php, but I get an empty string in the answer: string(0) ""
I think it would be: string(2) "ok"
I have replaced the return to echo in x.php but without success.
Sorry if it is a noob question, I'm quite new in curl.
Replace
return "ok";
With
echo "ok";
Your script is exiting with return value, but nothing is output as a response.
A thing to note is that web server must have permissions to access the x.php. Verify that the script is able to execute. Permission problems often result in a blank page.
Tip: Use Postman to test REST APIs. You could shoot the post query against your x.php and see what comes back, eliminating the curl part being wrong.

Why the header function is not working after CURL call?

Following is the call to an URL using CURL :
<?php
ini_set('display_startup_errors',1);
ini_set('display_errors',1);
error_reporting(-1);
$link = $_GET['link'];
$url = "http://www.complexknot.com/user/verify/link_".$link."/";
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
The variable $url contains one URL which I'm hitting using CURL.
The logic written in the file(present in a variable $url) is working absolutely fine.
After executing the code I want the control to be redirected to one URL. For it I've written following code :
header('Location: http://www.complexknot.com/login.php');
exit;
The following code is not working. The URL http://www.complexknot.com/login.php is not opening and a blank white page appears. This is the issue I'm facing.
If I don't use the CURL and hit the URL i.e. the URL contained in $url then it gets redirect to the URL http://www.complexknot.com/login.php that means header function works fine when I hit the URL in browser.
Why it's not working when I call it from CURL?
Please someone help me.
Thanks in advance.
This is happening because CURL is outputting the data. You must use curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); in order to let CURL returning data instead of outputting it.
<?php
ini_set('display_startup_errors', 0);
ini_set('display_errors', 0);
error_reporting(0);
$link = $_GET['link'];
$url = "http://www.complexknot.com/user/verify/link_$link/";
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
header('Location: http://www.complexknot.com/login.php');
You can use
<?php echo "<script>window.location.href = 'http://www.complexknot.com/login.php';</script>";die; ?>

Query data using curl

Why is it not working code, why - I do not understand. Code gets a response from curl and looking (must look) in this response word yes, if it is found - that displays the text - if not, then the other. The code:
<?PHP
// CURL
$ch = curl_init('http://dev.local/phpwhois-4.2.2/example.php?query=domain.ru&output=object');
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0");
curl_setopt ($ch, CURLOPT_HEADER, false);
$curl = curl_exec($ch);
echo $curl;
curl_close($ch);
if(preg_match('~\s*yes\s*~u', $curl))
echo 'Ok';
else
echo 'Else text';
?>
Error strange, more precisely, its not quite there, but - if curl sends text yes, that does not work, then writes that else, and if it does not give a text - too else. If all the text that simply gives curl himself put in the variable it works.
That's what gives the script to curl `e (this answer in writing what else):
regrinfo->Array disclaimer->Array 0->By submitting a query to RIPN's
Whois Service 1->you agree to abide by the following terms of use:
2->#3.2 (in Russian)
3-#3.2 (in English).
domain->Array name->hashcode.ru nserver->Array
ns1.nameself.com->81.176.95.18 ns2.nameself.com->88.212.207.45
status->REGISTERED, DELEGATED, VERIFIED created->2010-11-05
expires->2014-11-05 source->TCI registered->yes regyinfo->Array
referrer-> registrar->RUCENTER-REG-RIPN
servers->Array 0->Array server->ru.whois-servers.net
args->hashcode.ru port->43 type->domain rawdata->Array 0->% By
submitting a query to RIPN's Whois Service 1->% you agree to abide by
the following terms of use: 2->%
(in Russian) 3->%
(in English). 4->
5->domain: 6->nserver: . 7->nserver:
. 8->state: REGISTERED, DELEGATED, VERIFIED
9->person: Private Person 10->registrar: REGTIME-REG-RIPN
11->admin-contact: 12->created: 2010.11.05
13->paid-till: 2014.11.05 14->free-date: 2014.12.06 15->source: TCI
16-> 17->Last updated on 2014.07.27 12:31:31 MSK 18->
You have forgotten to set flag return transfer
<?PHP
// CURL
$ch = curl_init('http://dev.local/phpwhois-4.2.2/example.php?query=domain.ru&output=object');
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$curl = curl_exec($ch);
echo $curl;
curl_close($ch);
if(preg_match('~\s*yes\s*~u', $curl))
echo 'Ok';
else
echo 'Else text';
?>
Take care also about timeouts in the future. Good luck.

HTML entity '&times' and php function

I need a function that return me the Timezone of a specific location, so i use the
Google Time Zone API.
function timezoneLookup($lat, $lng){
$url = 'https://maps.googleapis.com/maps/api/timezone/json?location='.$lat.','.$lng.'&timestamp='.time().'&sensor=false';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
The function doesn't work because if i return $url i can see that GET variable "&timestamp=" is transformed into "×tamp=".
If i run the script outside the function it works.
WHY??
----UPDATE----
I resolved the problem, the curl doesn't work with https://, so i add:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
See this for more information PHP cURL Not Working with HTTPS
The function works fine. The reason you're seeing ×tamp= is because &times is being converted to ×. If you view source you'll see the correct url(instead of viewing the converted entity on the web page).
Why ; is not required
There is no problem with this function. If you echo that URL you will get the multiplication sign because it is being filtered through html and recognizing the ascii code. This only happens when you view it though and html viewer (browser), if you view source you will see the original string.
To confirm that this conversion will not occur when passed through curl_setopt(), I ran your code on my server and got an expected result.
echo timezoneLookup(52.2023913, 33.2023913);
function timezoneLookup($lat, $lng){
$url = 'https://maps.googleapis.com/maps/api/timezone/json?location='.$lat.','.$lng.'&timestamp='.time().'&sensor=false';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
Returned...
{ "dstOffset" : 3600, "rawOffset" : 7200, "status" : "OK", "timeZoneId" : "Europe/Kiev", "timeZoneName" : "Eastern European Summer Time" }
If this code is not working for you then it could be a networking issue. Try doing curl with another webpage and see what happens. Also, with a simple api call like this you could easily use file_get_contents()

need help converting php curl code to C language

my service provider has given me following piece of PHP code for accessing his service. I need help in converting to C lang code for use in my application. The code is using curl module to post on to a site.
pls advise.
<?php
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, "http://api.mVaayoo.com/mvaayooapi/MessageCompose?user="myusername":"mypassword"&senderID=TEST SMS&receipientno="phonenum"&msgtxt=This is a test from mVaayoo API&state=4");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "user=$user&senderID=$senderID&receipientno=$receipientno&cid=$cid&msgtxt=$msgtxt");
$buffer = curl_exec($ch);
if(empty ($buffer))
{ echo " buffer is empty "; }
else
{ echo $buffer; }
curl_close($ch);
?>
Use libcurl with it's C-interface. The remainer is good old C-style-string-handling.
Your example libcurl program in the comment looks good, except that for a POST you need to install a CURLOPT_READFUNCTION, not a CURLOPT_WRITEFUNCTION. But if you just want to post a static buffer, use CURLOPT_POSTFIELDS instead of a callback function.

Categories