I have this code
<?php
$ua = array(
"Mozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts)",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; XH; rv:8.578.498) fr, Gecko/20121021 Camino/8.723+ (Firefox compatible)",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1",
"Mozilla/5.0 (compatible, MSIE 11, Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko",
"Mozilla/5.0 (X11; U; Linux i686; fr-fr) AppleWebKit/525.1+ (KHTML, like Gecko, Safari/525.1+) midori/1.19",
"Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16",
"Mozilla/5.0 (Linux; U; Android 4.0.3; de-ch; HTC Sensation Build/IML74K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30");
$uar = array_rand($ua);
$url = "sometestserverisetup";
$ip = '127.0.0.1';
$port = '9051';
$auth = 'mypwwhateveritis';
$command = 'signal NEWNYM';
$fp = fsockopen($ip,$port,$error_number,$err_string,10);
if(!$fp) { echo "ERROR: $error_number : $err_string";
return false;
} else {
fwrite($fp,"AUTHENTICATE \"".$auth."\"\n");
$received = fread($fp,512);
fwrite($fp,$command."\n");
$received = fread($fp,512);
}
fclose($fp);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, "127.0.0.1:9050");
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch,CURLOPT_USERAGENT,$ua[$uar]);
$response = curl_exec($ch);
echo $response;
?>
everything works fine. With my test site and it displays correctly. However certain sites (google.com, amazon.com, youtube, facebook. only display a blank page for echo response.
Is there some curl set opt that needs to be enabled for pages to display properly.
Looking at a var_dump(curl_getinfo($ch)); after calling curl_exec can be helpful.
I tested your code and found in some cases the sites send a 302 Moved response with a Location header to redirect the browser which would result in an empty response on a successful request.
Adding
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
made it so that every site you mentioned always returned a response in my tests. And depending on what you are doing (searches, logins, form submissions) you will probably find redirects are common so you need to tell cURL to follow them with that option.
Beyond that, you can set CURLOPT_HEADER to true so you can look at the response headers sent to see what's going on in addition to curl_getinfo to make sure the connection was successful (either through Tor or to the site).
Related
I need to setup some custom headers start with ":".
$option['headers'][] = ":authority: example.com"; //<-- Here is the problem
$option['headers'][] = "accept-encoding: gzip, deflate, br";
$option['post'] = json_encode(array("Domain"=>"example.com"));
$url = "https://www.google.com";
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36");
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch,CURLOPT_COOKIEFILE,"file.cookie");
curl_setopt($ch,CURLOPT_COOKIEJAR,"file.cookie");
curl_setopt($ch,CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch,CURLOPT_HEADER,0);
curl_setopt($ch,CURLOPT_VERBOSE, true);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $option['post']);
curl_setopt($ch, CURLOPT_HTTPHEADER, $option['headers']);
$getdata = curl_exec($ch);
I try to replace the ":" with chr(58) but same problem. I get error 55 and from log "* Failed sending HTTP POST request". If I comment first line is working, but I really need that header. I'm stuck here. Any solutions?
:authority: looks like an HTTP/2 psuedo header, and you can't set them like this with curl. curl will however pass it on itself and it will use the same content as it would set for Host: to make it work the same way, independently of which HTTP version that is eventually used (it will also work with HTTP/3).
With this code can return the online users only on localhost. When sending to webhost crashes, even seeking the token code example in the second call. It only works with the second parameter tokem generated on site page source code. How to run this script from the webhost?
if (!function_exists('getHistats')) {
function getHistats($sid = 0, $cc = '') {
if (empty($sid) || empty($cc))
return 'error';
$url = 'http://www.histats.com/viewstats/HST_GET_SUMMARY.php';
$result = '';
$ualist = array(
'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)',
'Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.23 (KHTML, like Gecko) Ubuntu/10.04 Chromium/11.0.688.0 Chrome/11.0.688.0 Safari/534.23',
'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.04 (lucid) Firefox/3.6.14 GTB7.1',
'Opera/9.80 (X11; Linux i686; U; en) Presto/2.7.62 Version/11.01',
'Midori/0.2.2 (X11; Linux i686; U; en-us) WebKit/531.2+',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:15.0) Gecko/20100101 Firefox/15.0',
'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1150.1 Iron/20.0.1150.1 Safari/536.11'
);
if (function_exists('curl_init')) {
$http_headers = array();
$http_headers[] = 'Host: www.histats.com';
$http_headers[] = 'Referer: www.histats.com/viewstats/?sid='. $sid .'&act=2&f=1';
$http_headers[] = 'X-Requested-With: XMLHttpRequest';
$opts = array();
$opts[CURLOPT_URL] = $url;
$opts[CURLOPT_HTTPHEADER] = $http_headers;
$opts[CURLOPT_CONNECTTIMEOUT] = 5;
$opts[CURLOPT_TIMEOUT] = 10;
$opts[CURLOPT_USERAGENT] = $ualist[rand(0, count($ualist) - 1)];
$opts[CURLOPT_HEADER] = FALSE;
$opts[CURLOPT_RETURNTRANSFER] = TRUE;
$opts[CURLOPT_POST] = 1;
$opts[CURLOPT_POSTFIELDS] = 'AR_REQ[sid]='. $sid .'&AR_REQ[CC]='. $cc .'&dbg=1';
# Initialize PHP/CURL handle
$ch = curl_init();
curl_setopt_array($ch, $opts);
# Create return array
$result = curl_exec($ch);
curl_close($ch);
} elseif (ini_get('allow_url_fopen')) {
$result = file_get_contents($url);
}
if (empty($result) || ($result == 'error=11') || ($result == 'err:1'))
return 'error';
$obj = json_decode($result);
return isset($obj->livearray->livesummary->cur_online) ? $obj->livearray->livesummary->cur_online : 0;
}
}
$html = file_get_contents('http://histats.com/viewstats/?sid=3041076&act=2&f=1');
preg_match("/OBJ_summary.sockTOKEN = '(.*?)'/i", $html, $match);
echo 'Online: '. getHistats('3041076', 'bjh1NStBTVZyMFJzRENTODFHTHNQamJyV0FvY2l4TGRNSk5FczQyYnR3dERlaUhWczJZNUtWQk5lU2p6STlyRTZCQXZUd2t6MWJzS3Z2cWs2d1g4aXc9PQ==');
echo '<br />';
echo 'Token: '. $match[1];
echo '<br />';
echo 'Online: '. getHistats('3041076', $match[1]);
My code isn't working, tried a few things but I'm new to php so yeah... here's what I got, always returns me a blank page.
<?php
ini_set('display_errors',1);
error_reporting(E_ALL);
$rnd = $_GET['rnd'];
$ch = curl_init("http://chat.website.com/script/login.php?rnd=".$rnd);
$request_headers = array();
$request_header[] = (
'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36',
'Content-Type: application/x-www-form-urlencoded',
'onLoad: [type Function]',
'p: password',
'u: username',
'owner: [object Object]
');
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
$userdata = curl_exec($ch);
echo $userdata;
?>
you are passing $request_headers but the data you have in $request_header and better see your array is fine.
or may be try something like this:
$request_header[] = array('User-Agent'=>'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36',
'Content-Type'=> 'application/x-www-form-urlencoded',
'onLoad'=>'[type Function]',
'p'=>'username',
'u'=>'password',
'owner'=>'[object Object]
');
I found my error, I wasn't making the request in POST.
Here's the code that is working if anyone needs it:
<?php
ini_set('display_errors',1);
error_reporting(E_ALL);
$rnd = 1;
$rnd = $_GET['rnd'];
$ch = curl_init("http://chat.website.com/scripts/login.php?rnd=".$rnd);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "onLoad=%5Btype%20Function%5D&p=password&u=username&owner=%5Bobject%20Object%5D");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$userdata = curl_exec($ch);
echo $userdata;
?>
How can I scrape a site using a User-Agent for Ipad?
I have this code below using curl in PHP which outputs the source but can't find the tags still. On Ipad or Safari browser using an Ipad User-Agent, the tags displays when the site is loaded.
Thanks!
<?php
$useragent= "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10')";
$ch = curl_init ("http://www.cbsnews.com/video/watch/?id=7370279n&tag=mg;mostpopvideo");
curl_setopt ($ch, CURLOPT_USERAGENT, $useragent); // set user agent
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
// curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
echo $output = curl_exec ($ch);
curl_close($ch);
?>
Try using curl from the command line, with a perl script such as this:
my $ua = "Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10";
my $curl = "curl -A '$ua'";
my $server = "http://www.cbsnews.com";
my $startpage = "$server/video/watch/?id=7370279n&tag=mg;mostpopvideo";
my $path = "/path/to/download/to";
open(f, "$curl -L $startpage |") or die "Cannot open website: $!";
while (<f>)
{
if (/<a\s+[^>]*href=\"$server\/([^\"\/])*\"/)
{
my $file = $2;
system("$curl -e $startpage $server/$file > $path/$file");
next;
}
if (/<a\s+[^>]*href=\"$server\/([^\"]+)\/([^\"\/])*\"/)
{
my $folder = $1;
my $file = "$folder/$2";
system("mkdir -p $path/$folder");
system("$curl -e $startpage $server/$file > $path/$file");
next;
}
}
close(f);
Thanks for looking at my question.
I want to get the mobile version by the use of either file_get_contents() or cURL. I know that it can be done by the help of modifying the HTTP headers in the request. Can you please give me a simple example to do so?
Thanks again!
Regards,
Sanket
As an alternative, file_get_contents and stream_context_create can also be used:
$opts = array('http' =>
array(
'header' => 'User-agent: Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3B48b Safari/419.3',
)
);
$context = stream_context_create($opts);
$result = file_get_contents($url, false, $context);
Is this what you are looking for ?
curl -A "Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3" http://example.com/your-url
You need to set the user agent string:
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420.1 (KHTML, like Gecko) Version/3.0 Mobile/3B48b Safari/419.3');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$page = curl_exec($ch);
curl_close($ch);