Writing multiple post requests using single connection - PHP - php

I am writing to a server using the following snippet.
$fp = connect();
$sent_requests = 0;
function connect() {
$addr = gethostbyname("example.com");
$fp = fsockopen("$addr", 80, $errno, $errstr);
socket_set_blocking( $fp, false );
if (!$fp) {
echo "$errstr ($errno)<br />\n";
exit(1);
} else{
echo "Connected\n";
return $fp;
}
}
function sendTestCalls($load){
global $fp, $sent_requests;
if(!$fp){
echo "reconnecting";
$sent_requests = 0;
//echo stream_get_contents($fp) . "\n";
fclose($fp);
$fp = connect();
}
$data = "POST /test HTTP/2.0\r\n";
$data.= "Host: example.com\r\n";
$data.= "Content-Type: application/json\r\n";
$data.= "Content-Length: ".strlen($load)."\r\n";
$data.= "Connection: Keep-Alive\r\n";
$data.= "xYtU87BVFluc6: 1\r\n";
$data.= "\r\n" . $load;
$bytesToWrite = strlen($data);
$totalBytesWritten = 0;
while ($totalBytesWritten < $bytesToWrite) {
$bytes = fwrite($fp, substr($data, $totalBytesWritten));
$totalBytesWritten += $bytes;
}
$sent_requests++;
}
$time = time();
for($i=0; $i<1000; $i++) {
sendTestCalls('{"justtesting": "somevalue"}');
}
fclose($fp);
$time_taken = time() - $time;//might be a bit inaccurate
echo "Time Taken: " . $time_taken . "\n";
When I check my access logs on my server less than 1000 post requests are received (in the range of 0 to 900). What am I doing wrong here?
EDIT1
I suppose my socket is timing out. What should I do to check if it has disconnected in such a scenario reconnect. I tried using stream_get_meta_data($fp) but it had no effect.

Try to insert this before each request:
$info = stream_get_meta_data($fp);
if ($info['timed_out']) {
fclose($fp);
$fp = connect();
}

I had found a solution to this at that point using php_curl and KEEP-ALIVE
Here is my updated version:
function sendCall(&$curl_handle, $data){
curl_setopt($curl_handle, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($curl_handle, CURLOPT_POSTFIELDS, $data);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt(
$curl_handle,
CURLOPT_HTTPHEADER,
array(
'Content-Type: application/json',
'Connection: Keep-Alive',
'Content-Length: ' . strlen($data)
)
);
$response = curl_exec($curl_handle); //Check response?
if($err = curl_error($curl_handle)) {
error_log("Error - $err Status - Reconnecting" );
$curl_handle = curl_init(curl_getinfo($curl_handle, CURLINFO_EFFECTIVE_URL));
sendCall($curl_handle, $data);
}
}
This function gives me an almost always alive connection. (Never got the error log in more than a week). Hope it helps anyone looking for the same.

Related

How to find the file size by url in php script [duplicate]

This question already has answers here:
check if php file exists on server
(2 answers)
Closed 4 years ago.
how can I make sure that the file exists on the server and find out its size on the URL without first downloading the file
$url = 'http://site.zz/file.jpg';
file_exists($url); //always is false
filesize($url); //not working
Help eny one worked exemple pls
The function file_exists() only works on file that exists on the server locally.
Similarly; filesize() function returns the size of the file that exists on the server locally.
If you are trying to load the size of a file for a given url, you can try this approach:
function get_remote_file_info($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
$data = curl_exec($ch);
$fileSize = curl_getinfo($ch, CURLINFO_CONTENT_LENGTH_DOWNLOAD);
$httpResponseCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return [
'fileExists' => (int) $httpResponseCode == 200,
'fileSize' => (int) $fileSize
];
}
Usage:
$url = 'http://site.zz/file.jpg';
$result = get_remote_file_info($url);
var_dump($result);
Example output:
array(2) {
["fileExists"]=>
bool(true)
["fileSize"]=>
int(12345)
}
Without any libraries and file openning
$data = get_headers($url, true);
$size = isset($data['Content-Length'])?(int) $data['Content-Length']:0;
Open remote files:
function fsize($path) {
$fp = fopen($path,"r");
$inf = stream_get_meta_data($fp);
fclose($fp);
foreach($inf["wrapper_data"] as $v) {
if (stristr($v, "content-length")) {
$v = explode(":", $v);
return trim($v[1]);
}
}
return 0;
}
Usage:
$file = "https://zzz.org/file.jpg";
$inbytes = fsize($filesize);
Use sockets:
function getRemoteFileSize($url){
$parse = parse_url($url);
$host = $parse['host'];
$fp = #fsockopen ($host, 80, $errno, $errstr, 20);
if(!$fp){
$ret = 0;
}else{
$host = $parse['host'];
fputs($fp, "HEAD ".$url." HTTP/1.1\r\n");
fputs($fp, "HOST: ".$host."\r\n");
fputs($fp, "Connection: close\r\n\r\n");
$headers = "";
while (!feof($fp)){
$headers .= fgets ($fp, 128);
}
fclose ($fp);
$headers = strtolower($headers);
$array = preg_split("|[\s,]+|",$headers);
$key = array_search('content-length:',$array);
$ret = $array[$key+1];
}
if($array[1]==200) return $ret;
else return -1*$array[1];
}
You can't access to filesize of a distant file.
You have to check with your local filepath.

Mailchimp api v1.3 returns HTTP/1.0 302 Moved Temporarily

Some of our customers still use api v1.3. MailChimp PHP API Wrapper 1.3 is used. From last 10 days this stopped working for unknown reasons. I know we should upgrade to api v3, but some customers won't see our e-mail so I am trying to find out what could be wrong.
I am enclosing the part of code that is calling their server which returns "http://HTTP/1.0 302 Moved Temporarily Server: AkamaiGHost...". I am trying with their support but even they don't know anything, it would be helpful if anybody can tell any opinion what could be the reason for this response so I can push this further with their support.
$dc = "us1";
if (strstr($this->api_key, "-")) {
list($key, $dc) = explode("-", $this->api_key, 2);
if (!$dc)
$dc = "us1";
}
$host = $dc . "." . $this->apiUrl["host"];
$params["apikey"] = $this->api_key;
$this->errorMessage = "";
$this->errorCode = "";
$sep_changed = false;
//sigh, apparently some distribs change this to & by default
if (ini_get("arg_separator.output") != "&") {
$sep_changed = true;
$orig_sep = ini_get("arg_separator.output");
ini_set("arg_separator.output", "&");
}
$post_vars = http_build_query($params);
if ($sep_changed) {
ini_set("arg_separator.output", $orig_sep);
}
$payload = "POST " . $this->apiUrl["path"] . "?" . $this->apiUrl["query"] . "&method=" . $method . " HTTP/1.0\r\n";
$payload .= "Host: " . $host . "\r\n";
$payload .= "User-Agent: MCAPI/" . $this->version . "\r\n";
$payload .= "Content-type: application/x-www-form-urlencoded\r\n";
$payload .= "Content-length: " . strlen($post_vars) . "\r\n";
$payload .= "Connection: close \r\n\r\n";
$payload .= $post_vars;
ob_start();
if ($this->secure) {
$sock = fsockopen("ssl://" . $host, 443, $errno, $errstr, 30);
} else {
$sock = fsockopen($host, 80, $errno, $errstr, 30);
}
if (!$sock) {
$this->errorMessage = "Could not connect (ERR $errno: $errstr)";
$this->errorCode = "-99";
ob_end_clean();
return false;
}
$response = "";
fwrite($sock, $payload);
stream_set_timeout($sock, $this->timeout);
$info = stream_get_meta_data($sock);
while ((!feof($sock)) && (!$info["timed_out"])) {
$response .= fread($sock, $this->chunkSize);
$info = stream_get_meta_data($sock);
}
fclose($sock);
ob_end_clean();
var_dump($info);
this return array(7) { ["timed_out"]=> bool(false) ["blocked"]=> bool(true) ["eof"]=> bool(false) ["stream_type"]=> string(14) "tcp_socket/ssl" ["mode"]=> string(2) "r+" ["unread_bytes"]=> int(0) ["seekable"]=> bool(false) }
var_dump($response);
this returns "http://HTTP/1.0 302 Moved Temporarily Server: AkamaiGHost Content-Length: 0 Location: https://us8.api.mailchimp.com/1.3/?output=php&method=listSubscribe Date: Fri, 10 Aug 2018 21:51:06 GMT Connection: close"
For me setting $this->secure = true didn't do the job because of the class' __construct()
function __construct($apikey, $secure=false) {
$this->secure = $secure;
$this->apiUrl = parse_url("https://api.mailchimp.com/" . $this->version . "/?output=php");
$this->api_key = $apikey;
}
So when you instantiate the class with just the API key $secure is false by default.
Fix:
$api = new MCAPI($apiKey, true);
Hope this helps any of you guys using a 7 year old repository ( myself included )...
It appeared they don't allow none secure connections anymore, this is why request was redirected and the php wrapper I used silently passed without any error and not adding e-mail to Mailchimp. Solution if to simply set $this->secure to true.

Why can I not download files from some sites like this?

This is my php source code:
<?php
$path = '/images/one.jpg';
$imm = 'http://www.allamoda.eu/wp-content/uploads/2012/05/calzedonia_290x435.jpg';
if( $content = file_get_contents($imm) ){
file_put_contents($path, $content);
echo "Yes";
}else{
echo "No";
}
?>
and I get this error:
Warning: file_get_contents(http://www.allamoda.eu/wp-content/uploads/2012/05/calzedonia_290x435.jpg) [function.file-get-contents]: failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden in /opt/lampp/htdocs/test/down.php on line 4
No
Why ?
There are some headers expected by the server(especially Accept and User-Agent). Use the stream_context -argument of file_get_contents() to provide them:
<?php
$path = '/images/one.jpg';
$opts = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Accept:image/png,image/*;q=0.8,*/*;q=0.5 \r\n".
"Host: www.allamoda.eu\r\n" .
"User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:12.0) Gecko/20100101 Firefox/12.0\r\n"
)
);
$context = stream_context_create($opts);
$imm = 'http://www.allamoda.eu/wp-content/uploads/2012/05/calzedonia_290x435.jpg';
if( $content = file_get_contents($imm,false,$context) ){
file_put_contents($path, $content);
echo "Yes";
}else{
echo "No";
}
?>
You are not allowed to download this file, the server allamoda.eu says (HTTP 403).
Nothing wrong with the code. The server simply is not letting you (either you have too much requests to it, or it just blocks all scripts scraping it).
You're not allowed to open the file directly. But you can try to fetch it's content by using sockets:
function getRemoteFile($url)
{
// get the host name and url path
$parsedUrl = parse_url($url);
$host = $parsedUrl['host'];
if (isset($parsedUrl['path'])) {
$path = $parsedUrl['path'];
} else {
// the url is pointing to the host like http://www.mysite.com
$path = '/';
}
if (isset($parsedUrl['query'])) {
$path .= '?' . $parsedUrl['query'];
}
if (isset($parsedUrl['port'])) {
$port = $parsedUrl['port'];
} else {
// most sites use port 80
$port = '80';
}
$timeout = 10;
$response = '';
// connect to the remote server
$fp = #fsockopen($host, '80', $errno, $errstr, $timeout );
if( !$fp ) {
echo "Cannot retrieve $url";
} else {
// send the necessary headers to get the file
fputs($fp, "GET $path HTTP/1.0\r\n" .
"Host: $host\r\n" .
"User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3\r\n" .
"Accept: */*\r\n" .
"Accept-Language: en-us,en;q=0.5\r\n" .
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n" .
"Keep-Alive: 300\r\n" .
"Connection: keep-alive\r\n" .
"Referer: http://$host\r\n\r\n");
// retrieve the response from the remote server
while ( $line = fread( $fp, 4096 ) ) {
$response .= $line;
}
fclose( $fp );
// strip the headers
$pos = strpos($response, "\r\n\r\n");
$response = substr($response, $pos + 4);
}
// return the file content
return $response;
}
Example:
$content = getRemoteFile('http://www.allamoda.eu/wp-content/uploads/2012/05/calzedonia_290x435.jpg');
Source

How do I test php code for twitter stream

I have the following function that is supposed to open up a twitter stream for me
<?php
set_time_limit(0);
$query_data = array('track' => 'facebook');
$user = 'xxx'; // replace with your account
$pass = 'xxx'; // replace with your account
$fp = fsockopen("ssl://stream.twitter.com", 443, $errno, $errstr, 30);
if(!$fp){
print "$errstr ($errno)\n";
} else {
$request = "GET /1/statuses/filter.json?" . http_build_query($query_data) . " HTTP/1.1\r\n";
$request .= "Host: stream.twitter.com\r\n";
$request .= "Authorization: Basic " . base64_encode($user . ':' . $pass) . "\r\n\r\n";
fwrite($fp, $request);
while(!feof($fp)){
$json = fgets($fp);
$data = json_decode($json, true);
if($data){
//
// Do something with the data!
//
echo $data . "<br />";
}
}
fclose($fp);
}
?>
This is my first time ever to use php at all, so I'm completely oblivious as to what to do with this $data variable, this echo line does not produce any output on the browser. I'm sure this is trivial but a first-timer for me.
EDIT: The question is, what do I do with $data to make sure the code works? do I just shoot the file in a browser? it keeps saying "waiting for localhost..." forever
Why not just check to see if it's an array after the json_decode?
if( is_array( $data ) and count( $data ) ) // Ensures that json_decode created an array and has more than one element
{
// do your processing, it should be okay
}
else
{
// it's broken, error out
}

Retrieving page with fsockopen adds numbers to returned string

This is very strange, on some pages it will return the HTML fine, others it will add numbers to the beginning and end of the returned string ($out).
function lookupPage($page, $return = true) {
$fp = fsockopen("127.0.0.1", 48580, $errno, $errstr, 5);
if (!$fp) {
return false;
}
else {
$out = "";
$headers = "GET /" . $page . " HTTP/1.1\r\n";
$headers .= "Host: www.site.com\r\n";
$headers .= "Connection: Close\r\n\r\n";
fwrite($fp, $headers);
stream_set_timeout($fp, 300);
$info = stream_get_meta_data($fp);
while (!feof($fp) && !$info['timed_out'] && ($line = stream_get_line($fp, 1024)) !== false) {
$info = stream_get_meta_data($fp);
if ($return) $out .= $line;
}
fclose($fp);
if (!$info['timed_out']) {
if ($return) {
$out = substr($out, strpos($out, "\r\n\r\n") + 4);
return $out;
}
else {
return true;
}
}
else {
return false;
}
}
}
e.g...
3565
<html>
<head>
...
</html>
0
It is called Chunked Transfer Encoding
It is part of the HTTP 1.1 protocol and you're decoding it in a HTTP 1.0 way. You can just check for the values and trim them if you want. They only show the length of the response so the browser knows it has the complete response.
Also maybe look at file_get_contents
My guess would be that the server responds with chunked data.
Have a look at RFC2616 Transfer codings and its introduction.

Categories