i have been working on a scraper tool that rips google search results and then crawls the results websites looking to match specific items.
I'm having an issue with cURL though. I have come accross a site that is causing curl to go into an infinite loop.
website in question.
http://www.darellyelectrical.com/
when i open up my packet sniffer and look through tcp http packets ive found the same request is being sent over and over again.
i can not pinpoint the reason why, I have no trouble with any other websites.
I have tried setting the following curl options
curl_setopt($this->sessions[$key], CURLOPT_TIMEOUT, $timeout);
curl_setopt($this->sessions[$key], CURLOPT_MAXREDIRS, 2);
curl_setopt($this->sessions[$key], CURLOPT_CONNECTTIMEOUT, 1);
be great if someone could test that url with curl and let me know if the issue persists.
thanks
EDIT**
function sck_send()
{
$host = "www.darellyelectrical.com";
$path = "";
$fp = fsockopen($host, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /".$path." HTTP/1.1\r\n";
$out .= "Host: ".$host."\r\n";
$out .= "Connection: Close\r\n\r\n";
$data = "";
fwrite($fp, $out);
while (!feof($fp))
{
$data .= fgets($fp, 128);
}
fclose($fp);
echo $data;
}
}
sck_send();
this will produce the loop same as curl.
That server needs the User-Agent header to be included or it does not respond. PHP's curl doesn't set this by default and it wouldn't be included in a socket request unless you specify it. The code below works for me:
<?php
function sck_send() {
$host = "www.darellyelectrical.com";
$path = "";
$fp = fsockopen($host, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /".$path." HTTP/1.1\r\n";
$out .= "Host: ".$host."\r\n";
$out .= "User-Agent: Mozilla/5.0 \r\n";
$out .= "Connection: Close\r\n\r\n";
$data = "";
fwrite($fp, $out);
while (!feof($fp)) {
$data .= fgets($fp, 128);
}
fclose($fp);
echo $data;
}
}
sck_send();
Related
I'm trying to connect to an API using PHP Sockets.
I've tried it in many ways, but I can't get it working. I know there is other answers, but no one works. I really can't achieve it. I'm trying it at least two days.
Here is the error:
php_network_getaddresses: getaddrinfo failed: Name or service not known
Code (Sockets - It does not work):
var $url = 'api.site.com.br/';
public function getUsers(){
$path = "users/?accesstoken=c24f506fe265488435858925096bc4ad7ba0";
$packet= "GET {$path} HTTP/1.1\r\n";
$packet .= "Host: {$url}\r\n";
$packet .= "Connection: close\r\n";
if (!($socket = fsockopen($this->url,80,$err_no,$err_str))){
die( "\n Connection Failed. $err_no - $err_str \n");
}
fwrite($socket, $packet);
return stream_get_contents($socket);
}
Code (Curl - It Works!):
public function getUsers2(){
$path = "users/?accesstoken=c24f506fe265488435858925096bc4ad7ba0";
$method = 'GET';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $this->url . $path);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
$output = curl_exec($ch);
$curl_error = curl_error($ch);
curl_close($ch);
return $output;
}
I can't give to you the correct URL, but the URL is working with Curl, so it should work with sockets.
I solved it by doing the following:
public function getUsers(){
$path = "users/?accesstoken=c24f506fe265488435858925096bc4ad7ba0";
$fp = fsockopen($this->url, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /{$path} HTTP/1.1\r\n";
$out .= "Host: {$this->url}\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
echo fgets($fp, 128);
}
fclose($fp);
}
}
Connecting to a host via socket is not connecting to a URL.
Wrong: api.site.com.br/
Wrong: http://api.site.com.br/
Wrong: https//api.site.com.br/api/
Right: api.site.com.br
You connect to a host. This can be a domainname or an IP. Neither of those has a slash in it.
I want to send a request (POST or GET) and don't wait for answer
I have the code:
$fp = stream_socket_client("tcp://requestb.in:80", $errno, $errstr, 30, STREAM_CLIENT_ASYNC_CONNECT|STREAM_CLIENT_CONNECT);
$out = "GET /15b3x0v1 HTTP/1.1\r\n";
$out .= "Host: requestb.in\r\n";
$out .= "Connection: Close\r\n\r\n";
fputs($fp, $out);
//usleep(500);
fflush($fp);
fclose($fp);
Without the usleep, it doesn't work (the server doesn't receive the request). Why have i to wait 500ms before close the socket ?
<?php
$host = 'www.yourtargeturl.com';
$service_uri = '/detect_referal.php';
$vars ='additional_option1=yes&additional_option2=un';
$header = "Host: $host\r\n";
$header .= "User-Agent: PHP Script\r\n";
$header .= "Content-Type: application/x-www-form-urlencoded\r\n";
$header .= "Referer: http://www.google.com/search?hl=en&q=jigh&btnG=Google+Search \r\n";
$header .= "Content-Length: ".strlen($vars)."\r\n";
$header .= "Connection: close\r\n\r\n";
$fp = fsockopen("".$host,80, $errno, $errstr);
if (!$fp) {
echo "$errstr ($errno)<br/>\n";
echo $fp;
} else {
fputs($fp, "POST $service_uri HTTP/1.1\r\n");
fputs($fp, $header.$vars);
fwrite($fp, $out);
while (!feof($fp)) {
echo fgets($fp, 128);
}
fclose($fp);
}
?>
This is changing the $_SERVER['HTTP_REFERER']. How can I change $_SERVER['REMOTE_ADDR']. What code should I append in $header?
You can't do that. The IP address is determined at the beginning of the TCP connection, not in the HTTP headers. (It is possible to spoof [though not remotely like that], but then you won't get the response back.)
You can't. The IP address you connect out from isn't some header... it comes from the underlying TCP connection made from your server to the other server.
I am totally new to php. I had 2 pages let A.php and B.php want to run Page B while I am at Page A.For that I had created a Socket Connection and able to run The Page B but problem is that when I send an array of objects to page B.page A send only the First object listed in the array and other values of the array were nothing
here is my some code
in A.php
$arr = array("message" => $pushData,
"messageType" => $messageType,
"displayGroupID" => $displayGroupID,
"db"=>$db);
$fp1 = $this->JobStartAsync('localhost', 'B.php', $arr);
$r1 = $this->JobPollAsync($fp1);
function JobStartAsync($server, $url, $data, $port=80,$conn_timeout=10, $rw_timeout=86400)
{
$errno = '';
$errstr = '';
$data = http_build_query($data);
set_time_limit(0);
$fp = fsockopen($server, $port, $errno, $errstr, $conn_timeout);
if (!$fp)
{
echo "$errstr ($errno)<br />\n";
return false;
}
$out = "POST $url HTTP/1.1\r\n";
$out .= "Host: $server\r\n";
$out .= "Content-type: application/x-www-form-urlencoded\r\n";
$out .= "Content-length: ". strlen($data) ."\r\n";
$out .= "Connection: Close\r\n\r\n";
$out .= "$data";
stream_set_blocking($fp, false);
stream_set_timeout($fp, $rw_timeout);
fwrite($fp, $out);
//fwrite($fp, $data);
return $fp;
}
function JobPollAsync(&$fp)
{
if ($fp === false)
return false;
if (feof($fp))
{
fclose($fp);
$fp = false;
return false;
}
return fread($fp, 10000);
}
IN B.php
fileWrite ($_POST['displayGroupID'],$_POST['message'],$_POST['messageType']);
it get only "message" as it is listed as 1st element in array
Take a look at CURL. This is what you need and everyone else uses.
ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://www.example.com");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"postvar1=value1&postvar2=value2&postvar3=value3");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec ($ch);
curl_close ($ch);
For async variation take a look at a similar question answered before - How do I make an asynchronous GET request in PHP?
Hy all.
I need to get the content of multiple pages from a single domain.
Now for each page I use an fsockopen connection, and I get the content of the page this way:
<?php
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}
?>
My script wastes time, with reconnecting to the domain, to get the second page.
I was wondering, if is possible to use a single connection, and to get multiple pages, like this:
<?php
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
} $out = "GET /page2.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}
?>
But this method is returning the page1.html two times, I don't know why.
I tried to use: Connection: keep alive, or HTTP/1.0, but in this cases I didn't get anything from the server (infinite executing time of my script).
Any suggestion to solve this?
Thank you!
Try only sending the Connection: Close header on the last request.
EDIT: Clarification
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
// DON'T SEND Connection: Close HERE
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
$out = "GET /page2.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
// THIS IS THE LAST PAGE REQUIRED SO SEND Connection: Close HEADER
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}