Hy all.
I need to get the content of multiple pages from a single domain.
Now for each page I use an fsockopen connection, and I get the content of the page this way:
<?php
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}
?>
My script wastes time, with reconnecting to the domain, to get the second page.
I was wondering, if is possible to use a single connection, and to get multiple pages, like this:
<?php
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
} $out = "GET /page2.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}
?>
But this method is returning the page1.html two times, I don't know why.
I tried to use: Connection: keep alive, or HTTP/1.0, but in this cases I didn't get anything from the server (infinite executing time of my script).
Any suggestion to solve this?
Thank you!
Try only sending the Connection: Close header on the last request.
EDIT: Clarification
$fp = fsockopen("www.example.com", 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /page1.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
// DON'T SEND Connection: Close HERE
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
$out = "GET /page2.html HTTP/1.1\r\n";
$out .= "Host: www.example.com\r\n";
// THIS IS THE LAST PAGE REQUIRED SO SEND Connection: Close HEADER
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
while (!feof($fp)) {
fgets($fp, 128);
}
fclose($fp);
}
Related
I want to send a request (POST or GET) and don't wait for answer
I have the code:
$fp = stream_socket_client("tcp://requestb.in:80", $errno, $errstr, 30, STREAM_CLIENT_ASYNC_CONNECT|STREAM_CLIENT_CONNECT);
$out = "GET /15b3x0v1 HTTP/1.1\r\n";
$out .= "Host: requestb.in\r\n";
$out .= "Connection: Close\r\n\r\n";
fputs($fp, $out);
//usleep(500);
fflush($fp);
fclose($fp);
Without the usleep, it doesn't work (the server doesn't receive the request). Why have i to wait 500ms before close the socket ?
I grabbed the function curl_request_async from How do I make an asynchronous GET request in PHP?
// $type must equal 'GET' or 'POST'
function curl_request_async($url, $params, $type='POST')
{
foreach ($params as $key => &$val) {
if (is_array($val)) $val = implode(',', $val);
$post_params[] = $key.'='.urlencode($val);
}
$post_string = implode('&', $post_params);
$parts=parse_url($url);
$fp = fsockopen($parts['host'],
isset($parts['port'])?$parts['port']:80,
$errno, $errstr, 30);
// Data goes in the path for a GET request
if('GET' == $type) $parts['path'] .= '?'.$post_string;
$out = "$type ".$parts['path']." HTTP/1.1\r\n";
$out.= "Host: ".$parts['host']."\r\n";
$out.= "Content-Type: application/x-www-form-urlencoded\r\n";
$out.= "Content-Length: ".strlen($post_string)."\r\n";
$out.= "Connection: Close\r\n\r\n";
// Data goes in the request body for a POST request
if ('POST' == $type && isset($post_string)) $out.= $post_string;
fwrite($fp, $out);
fclose($fp);
}
It worked perfectly with my previous host but after I moved to a new host, it just does not work. I already installed php-curl but it's like the function just does not call the url.
There was no error given too. Please help I have been stuck with this for days.
One thing you could verify is if you can call fsockopen on your server, the code you posted does not use curl, but try to directly open a socket
I got my Paypal ipn to work. But I changed some things and now I can not remember what I changed. I used a series of INSERTS to monitor where the process of the ipn fails. It looks like it does not pass this part of my ipn. Here is the code:
fputs ($fp, $header . $req);
while (!feof($fp)) {
$res = fgets ($fp, 1024);
if (strcmp($res, "VERIFIED") == 0) {
Why is my Paypal ipn not working?
I had also struggled a lot with paypal ipn last week,below is my working code you can refer
header('HTTP/1.1 200 OK');
$req = 'cmd=_notify-validate';
foreach ($_POST as $key => $value) {
$value = urlencode(stripslashes($value));
$req .= "&$key=$value";
}
$fp = fsockopen('ssl://www.sandbox.paypal.com', 443, $errno, $errstr, 30);
fputs($fp, "POST /cgi-bin/webscr HTTP/1.1\r\n");
fputs($fp, "Host: www.sandbox.paypal.com\r\n");
fputs($fp, "Content-type: application/x-www-form-urlencoded\r\n");
fputs($fp, "Content-length: ".strlen($req)."\r\n");
fputs($fp, "Connection: close\r\n\r\n");
fputs($fp, $req. "\r\n\r\n");
$ipn_response = '';
while (!feof($fp)) {
$ipn_response .= fgets($fp, 1024);
}
fclose($fp);
if ( eregi("VERIFIED",$ipn_response) ) {}
i have been working on a scraper tool that rips google search results and then crawls the results websites looking to match specific items.
I'm having an issue with cURL though. I have come accross a site that is causing curl to go into an infinite loop.
website in question.
http://www.darellyelectrical.com/
when i open up my packet sniffer and look through tcp http packets ive found the same request is being sent over and over again.
i can not pinpoint the reason why, I have no trouble with any other websites.
I have tried setting the following curl options
curl_setopt($this->sessions[$key], CURLOPT_TIMEOUT, $timeout);
curl_setopt($this->sessions[$key], CURLOPT_MAXREDIRS, 2);
curl_setopt($this->sessions[$key], CURLOPT_CONNECTTIMEOUT, 1);
be great if someone could test that url with curl and let me know if the issue persists.
thanks
EDIT**
function sck_send()
{
$host = "www.darellyelectrical.com";
$path = "";
$fp = fsockopen($host, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /".$path." HTTP/1.1\r\n";
$out .= "Host: ".$host."\r\n";
$out .= "Connection: Close\r\n\r\n";
$data = "";
fwrite($fp, $out);
while (!feof($fp))
{
$data .= fgets($fp, 128);
}
fclose($fp);
echo $data;
}
}
sck_send();
this will produce the loop same as curl.
That server needs the User-Agent header to be included or it does not respond. PHP's curl doesn't set this by default and it wouldn't be included in a socket request unless you specify it. The code below works for me:
<?php
function sck_send() {
$host = "www.darellyelectrical.com";
$path = "";
$fp = fsockopen($host, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)<br />\n";
} else {
$out = "GET /".$path." HTTP/1.1\r\n";
$out .= "Host: ".$host."\r\n";
$out .= "User-Agent: Mozilla/5.0 \r\n";
$out .= "Connection: Close\r\n\r\n";
$data = "";
fwrite($fp, $out);
while (!feof($fp)) {
$data .= fgets($fp, 128);
}
fclose($fp);
echo $data;
}
}
sck_send();
<?php
$host = 'www.yourtargeturl.com';
$service_uri = '/detect_referal.php';
$vars ='additional_option1=yes&additional_option2=un';
$header = "Host: $host\r\n";
$header .= "User-Agent: PHP Script\r\n";
$header .= "Content-Type: application/x-www-form-urlencoded\r\n";
$header .= "Referer: http://www.google.com/search?hl=en&q=jigh&btnG=Google+Search \r\n";
$header .= "Content-Length: ".strlen($vars)."\r\n";
$header .= "Connection: close\r\n\r\n";
$fp = fsockopen("".$host,80, $errno, $errstr);
if (!$fp) {
echo "$errstr ($errno)<br/>\n";
echo $fp;
} else {
fputs($fp, "POST $service_uri HTTP/1.1\r\n");
fputs($fp, $header.$vars);
fwrite($fp, $out);
while (!feof($fp)) {
echo fgets($fp, 128);
}
fclose($fp);
}
?>
This is changing the $_SERVER['HTTP_REFERER']. How can I change $_SERVER['REMOTE_ADDR']. What code should I append in $header?
You can't do that. The IP address is determined at the beginning of the TCP connection, not in the HTTP headers. (It is possible to spoof [though not remotely like that], but then you won't get the response back.)
You can't. The IP address you connect out from isn't some header... it comes from the underlying TCP connection made from your server to the other server.