Unable to Download Remote File from server after using sleep in php - php

Hi there i am having a strange problem with downloading a remote file. I am using Flurry Data API to download some Reports. Thing is when first time we call the Flurry API it gives us XML/JSON response which contains the path of the Report for Download. It takes like 2 minutes for the report to get ready. I am having Problem with that thing. I wrote a Script which download the remote file if i just paste the link of Report directly to function which is already ready to download. It works like a charm but i have to automate the process of Downloading. So for that i First call the API and get the Report Download Link then i use sleep() function of PHP to stop execution for like 3 Minutes(Tried it with 2 also). Then i call the same function which i uses to download the report successfully doesn't work this time. Here is the File Download Method:
function get_file_and_save($file, $local_path, $newfilename) {
$err_msg = '';
$out = fopen($local_path . $newfilename . ".gz", 'wb');
if ($out == FALSE) {
print "File is not available<br>";
exit;
}
$ch = curl_init();
$headers = array('Content-type: application/x-gzip', 'Connection: Close');
curl_setopt($ch, CURLOPT_FILE, $out);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_URL, $file);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_exec($ch);
echo "<br>Error Occured:" . curl_error($ch);
curl_close($ch);
}
i also have tried giving the CURLOPT_TIMEOUT but it wasn't working either.
The code to Request to the Flurry note that download_path is working properly it just get the report link:
$query_url = 'http://api.flurry.com/rawData/Events?apiAccessCode=' . $ACCESS_CODE . '&apiKey=' . $API_KEY . '&startTime=' . $start_time . '&endTime=' . $end_time;
$response = download_path($query_url);
if ($response) {
$response_obj = json_decode($response, true);
if (isset($response_obj['report']['#reportUri'])) {
$report_link = $response_obj['report']['#reportUri'];
}
if (isset($response_obj['report']['#reportId'])) {
$report_id = $response_obj['report']['#reportId'];
}
if(isset($report_link) && !empty($report_link)){
echo "<br >Report Link: ".$report_link."<br >";
sleep(240);
$config = array(
'http' => array(
'header' => 'Accept: application/json',
'method' => 'GET'
)
);
$stream = stream_context_create($config);
$json= file_get_contents($report_link,false,$stream);
echo "<pre>";
print_r($http_response_header);
echo "</pre>";
if($http_response_header[3] == "Content-Type: application/octet-stream"){
get_file_and_save($report_link, "data-files/gz/", $current_time);
}
}
else{
echo "There was some error in downloading report";
}
} else {
$error = true;
echo "There was some error in genrating report";
}
is there something problem with sleep() or what i am stuck its been 2nd night i am unable to achieve it.

Check if your PHP script is timing out and being killed off. Both the webserver and PHP have max execution limits to prevent runaway scripts, and if your sleep surpasses that limit, it'll never continue beyond that.
http://www.php.net/manual/en/info.configuration.php#ini.max-execution-time
http://php-fpm.org/wiki/Configuration_File - request_terminate_timeout
http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_read_timeout
http://httpd.apache.org/docs/2.0/mod/core.html#timeout

Related

Check header response before download in single request with CURL

Is it possible to check the response headers (200=OK) and download a file in a single CURL request?
Here is my code. The problem with this is that it makes 2 requests, and hence the second request can be different and the saved file will be overwritten. This is a problem with rate limited API. I searched here on Stackoverflow but most solutions still make 2 requests.
// Check response first, we don't want to download the response error to the file
$urlCheck = checkRemoteFile($to_download);
if ($urlCheck) {
// Response is 200, continue
} else {
// Do not overwrite existing file
echo 'Download failed, response code header is not 200';
exit();
}
// File Handling
$new_file = fopen($downloaded, "w") or die("cannot open" . $downloaded);
// Setting the curl operations
$cd = curl_init();
curl_setopt($cd, CURLOPT_URL, $to_download);
curl_setopt($cd, CURLOPT_FILE, $new_file);
curl_setopt($cd, CURLOPT_TIMEOUT, 30); // timeout is 30 seconds, to download the large files you may need to increase the timeout limit.
// Running curl to download file
curl_exec($cd);
if (curl_errno($cd)) {
echo "the cURL error is : " . curl_error($cd);
} else {
$status = curl_getinfo($cd);
echo $status["http_code"] == 200 ? "File Downloaded" : "The error code is : " . $status["http_code"] ;
// the http status 200 means everything is going well. the error codes can be 401, 403 or 404.
}
// close and finalize the operations.
curl_close($cd);
fclose($new_file);
# FUNCTIONS
function checkRemoteFile($url) {
$curl = curl_init($url);
//don't fetch the actual page, you only want to check the connection is ok
curl_setopt($curl, CURLOPT_NOBODY, true);
//do request
$result = curl_exec($curl);
$ret = false;
//if request did not fail
if ($result !== false) {
//if request was ok, check response code
$statusCode = curl_getinfo($curl, CURLINFO_HTTP_CODE);
if ($statusCode == 200) {
$ret = true;
}
}
curl_close($curl);
return $ret;
}

How to solve "Connection timed out after 60051 milliseconds" cURL with PHP?

I'm trying to get JSON data from 51 URLs using cURL in Php & recently , I'm stuck with an issue of Connection Timeout & it displays data return bool(false).
Here is error message :
Curl Error no :28
Curl error message :Connection timed out after 60051 milliseconds
Count Value : 0
bool(false)
And Here is my Code :
foreach($this->state_list as $short_code => $state_name_value) {
// echo "Key=" .$short_code . ", Value=" . $state_name_value;
$legislator_of_state_Url = "https://openstates.org/api/v1/legislators/?state=".$short_code."&chamber=upper";
echo "URL : ".$legislator_of_state_Url."<br/>";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL,$legislator_of_state_Url);
curl_setopt($curl,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl , CURLOPT_HTTPHEADER, array ('Accept: application/json'));
curl_setopt($curl,CURLOPT_CONNECTTIMEOUT,60);
$feed = curl_exec($curl);
echo "Curl Info :".curl_getinfo($curl) . '<br/>';
echo "Curl Error no :".curl_errno($curl) . '<br/>';
echo "Curl error message :".curl_error($curl) . '<br/>';
curl_close($curl);
// print_r(json_decode($feed,true));
$jsonarray = json_decode($feed, true);
echo "Count Value : ".count($jsonarray);
if(count($jsonarray) == 0)
{
var_dump($feed);
}
}
Every time whenever I run this code 2 random URLs out of 51 returns this error hence I'm not getting any JSON data
What could be the solution ?
if you want to increase the time limit of the script execution then use this
set_time_limit(seconds);
but there can be some other issues like the url might not be responding at all,you can also check manually the 2 url which are giving the error

Curl caching a downloaded image?

I have a script that tails a log file for a song change event. When that happens a Tweet is auto generated and sent to Twitter. Problem is that, I recently updated my script so that it also includes the album art with the Tweet. I am downloading the image using curl, posting the media, then removing the file for the next Tweet. What is throwing me off is that when the image displayed in the Tweet is for the previous song playing and not the current one. I want to know if I download a file using curl, then use php unlink command, would that same image be downloaded again? Here is my code below.
// Analyze prepared tweet for issues with length
$tweeted = ($front . $fixedArtistNow . $hyphen . " " . $fixedTitleNow . $tag);
if (strlen($tweeted) <= 140) {
try {
// $path = ("/home/soundcheck/public_html/images/artwork.png");
// if(unlink($path));
// Grab album art from the song that is currently playing
$fresh = date("y-m-d-G-i-s");
$ch = curl_init('http://soundcheck.xyz:8000/playingart? sid=1?'.$fresh);
var_dump($ch);
$fp = fopen('/home/soundcheck/public_html/images/artwork.png', 'wb');
curl_setopt($fp, CURLOPT_FRESH_CONNECT, TRUE);
curl_setopt($fp, CURLOPT_FORBID_REUSE, TRUE);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
// copy("/home/soundcheck/public_html/images/artwork.png","/home/soundcheck/public_html/images/album.png");
// $twitter->send($front . $fixedArtistNow . $hyphen . " " . $fixedTitleNow . $tag);
$media1 = $twitter->upload('media/upload', ['media' => '/home/soundcheck/public_html/images/artwork.png']);
// $media2 = $connection->upload('media/upload', ['media' => '/path/to/file/kitten2.jpg']);
$parameters = [
'status' => $tweeted,
'media_ids' => implode(',', [$media1->media_id_string]),
];
$result = $twitter->post('statuses/update', $parameters);
// unlink("/home/soundcheck/public_html/images/artwork.png");
The url I am getting the image from never changes but the image does if that makes any difference. If anyone knows a better way to download the album art, please let me know. Thanks!

Urban Airship API Error Code 40700

I'm working with the Urban Airship (v3) API to push out messages to Android/iPhone/Blackberry and hopefully soon Windows phones. I'm not responsible for that; instead, I'm setting up the backend to allow users to send out a broadcast.
Another guy built the original backend, but I chose to rebuilt it from the bottom up to add in some additional functionality. Everything works in it, except the whole pushing of the broadcast part. Well, it sort of works; let me explain:
When a form is submitted, the data goes into the database via MYSQL and then with mysql_fetch_id() I get the new id and toss that id into a PHP function called sentBroadcast. It looks like the following:
function sentBroadcast($id){
$alertinfo = getAlertInfo($id);//this just gets all the data matching the id
$alert = mysql_fetch_assoc($alertinfo);
//these just get extra values
$organization = getOrganizationById($alert['broadcast_organization_id']);
$cityinfo = getCityInfo($organization['organization_city_id']);
$city = mysql_fetch_assoc($cityinfo);
// Create Airship object
$airship = new Airship(APP_KEY, APP_MASTER_SECRET);
$apiurl = "https://go.urbanairship.com/api/location/?q=".str_replace(" ","",strtolower($city['city_name']));
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $apiurl);
curl_setopt($ch, CURLOPT_USERPWD, APP_KEY.":".APP_MASTER_SECRET);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close ($ch);
$json = json_decode($output);
$locationid = "all";
if(count($json->features) > 0){
$locationid = $json->features[0]->id;
}
//send the message
$broadcasttype = "";
if($alert['broadcast_broadcasttypeother'] != ""){
$broadcasttype = $alert['broadcast_broadcasttypeother'];
}
else {
//this just gets data, nothing to see here
$broadcasttype = getCategoryInfo($alert['broadcast_broadcasttype_id'],'broadcasttype_name');
}
$message = html_entity_decode($broadcasttype)."\r\n".html_entity_decode($organization['organization_name'])."\r\n". html_entity_decode($alert['broadcast_subject']);
$blackberry_message = html_entity_decode($organization['organization_name'])."\r\n". html_entity_decode($alert['broadcast_subject']);
//calc as UTC
$timestamp = strtotime($alert['broadcast_sentdatetime']) + strtotime("+1 minute"); //add an hour
$offset = new DateTime(date("Y-m-d H:i:s T",$timestamp));
$offset->setTimezone(new DateTimeZone('UTC'));
$minutes_to_add = 10;
$time = new DateTime($alert['broadcast_sentdatetime']);
$time->add(new DateInterval('PT' . $minutes_to_add . 'S'));
$stamp = $time->format('Y-m-d H:i:s');
//echo $stamp;
$broadcast_message = array(
'schedule' => array("scheduled_time" => $stamp),
'push' => array("audience" => "all",
"notification" => array("alert" => $message),
"device_types" => array()
),
);
$device_types = array();
$device_types[] = "ios";
$device_types[] = "android";
$device_types[] = "blackberry";
$broadcast_message["push"]["device_types"] = $device_types;
if(in_array("ios", $device_types)){
$broadcast_message["push"]["notification"]["ios"] = array("sound" => "police.mp3", "extra" => array("id"=>$alert['broadcast_id']), "badge" => "+1");
}
if(in_array("android", $device_types)){
$broadcast_message["push"]["notification"]["android"] = array("extra"=>array("id"=>$alert['broadcast_id']));
}
if(in_array("blackberry", $device_types)){
$broadcast_message["push"]["notification"]["blackberry"] = array("content-type"=>"text/plain","body"=> json_encode(array("id"=>$alert['broadcast_id'], "body"=>$blackberry_message, "type"=>$broadcasttype)));
}
$data_string = json_encode($broadcast_message);
$apiurl = "https://go.urbanairship.com/api/schedules/";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $apiurl);
curl_setopt($ch, CURLOPT_USERPWD, APP_KEY.":".APP_MASTER_SECRET);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-type: application/json',
'Accept: application/vnd.urbanairship+json; version=3;',
'Content-Length: ' . strlen($data_string)
));
$json = curl_exec($ch);
$output = json_decode($json, true);
if(!is_array($output) || empty($output["ok"])){
echo "<h1>ERROR: (".(isset($output["error"]) ? $output["error"] : "An unknown error has occurred while trying to send your message.").")</h1>";
echo $data_string;
print_r($output);
$error = true;
}
curl_close ($ch);
$debug = false;
if($debug || $error){
if($error) echo "<!--";
var_dump($broadcast_message);
echo "<hr><br>";
echo $json."<hr><br>";
echo "<pre>";
print_r($output);
echo "</pre>";
if(!empty($output['ok'])){
//maybe we should save the status, or the json in the db.
echo 'yay it sent';
}
if($error) echo "-->";
}
if($error){
exit;
}
}//end sendBroadcast
When I do this query, I get hit by an error "Could not parse body request body". That wasn't very helpful, so I printed the response (look under "if(!is_array($output) || empty($output["ok"])){"). I get the following error message:
Array ( [ok] => [error] => Could not parse request body. [error_code] => 40700 [details] => Array ( [error] => Cannot schedule for the past 2013-10-12T06:46:00.000Z ) [operation_id] => 6fde4fa0-4b64-11e3-8903-90e2ba0253a0 )
The error I'm getting is "Cannot schedule for the past", however at the time of submitting this, it was the future. I began doing some research and read that I had to set it to UTC time. That being said, whatever my time is now, it will always be 6 hours into the past in UTC, so I have to convert it up to UTC.
So, I did that and the message went out and the phones received it and all went well. Except when we went to read the message: we then got an error that said the message was deleted.
We didn't delete it, so I think maybe (it hasn't been 6 hours yet) the users phone will get the new broadcast in the future, but they got informed of the alert now. That alert isn't visible yet, so it throws an error. At least that's what I think; it hasn't been 6 hours yet so I can't prove that.
My problem is this: How do I tell Urban Airship I want an immediate post to go out, without having to add 6 hours to the current time to make it in the "present", as well as actually having the phones get it at the correct time?
I contacted UA but they said to "expect a week delay in responding to you" (No rush, eh?) and I googled the error code (40700) and came up with nothing. I then emailed the guy who built the original and all he said was the "UTC was very important". Thank you for that.
If anybody can help me out I would be very thankful.
Thank you.
Oh, and if anybody is wondering, the json I'm submitting looks like the following:
{"schedule":{"scheduled_time":"2013-10-12 06:46:00"},"push":{"audience":"all","notification":{"alert":"Message\r\nKenton Industries\r\nKenton Test","ios":{"sound":"police.mp3","extra":{"id":"406"},"badge":"+1"},"android":{"extra":{"id":"406"}},"blackberry":{"content-type":"text\/plain","body":"{\"id\":\"406\",\"body\":\"Kenton Industries\\r\\nKenton Test\",\"type\":\"Message\"}"}},"device_types":["ios","android","blackberry"]}}
Thanks :)
I'm not sure I understand the second part of your question of, "having the phones get it at the correct time". In regards to the question of, "How do I tell Urban Airship I want an immediate post to go out, without having to add 6 hours to the current time to make it in the "present"":
If you want your users to receive the message as soon as possible then you shouldn't be scheduling your message. Via their curl examples an immediate push message curl request should look as thus:
curl -v -X POST -u "<AppKey>:<MasterSecret>" -H "Content-type: application/json" -H "Accept: application/vnd.urbanairship+json; version=3;" --data '{"audience" : "all", "device_types" : "all", "notification" : {"alert": "This is a broadcast"} }' https://go.urbanairship.com/api/push/

Get the filesize of a js file on another domain using php

How do I get the filesize of js file on another website. I am trying to create a monitor to check that a js file exists and that it is more the 0 bytes.
For example on bar.com I would have the following code:
$filename = 'http://www.foo.com/foo.js';
echo $filename . ': ' . filesize($filename) . ' bytes';
You can use a HTTP HEAD request.
<?php
$url = "http://www.neti.ee/img/neti-logo.gif";
$head = get_headers($url, 1);
echo $head['Content-Length'];
?>
Notice: this is not a real HEAD request, but a GET request that PHP parses for its Content-Length. Unfortunately the PHP function name is quite misleading. This might be sufficient for small js files, but use a real HTTP Head request with Curl for bigger file sizes because then the server won't have to upload the whole file and only send the headers.
For that case, use the code provided by Jakub.
Just use CURL, here is a perfectly good example listed:
Ref: http://www.php.net/manual/en/function.filesize.php#92462
<?php
$remoteFile = 'http://us.php.net/get/php-5.2.10.tar.bz2/from/this/mirror';
$ch = curl_init($remoteFile);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); //not necessary unless the file redirects (like the PHP example we're using here)
$data = curl_exec($ch);
curl_close($ch);
if ($data === false) {
echo 'cURL failed';
exit;
}
$contentLength = 'unknown';
$status = 'unknown';
if (preg_match('/^HTTP\/1\.[01] (\d\d\d)/', $data, $matches)) {
$status = (int)$matches[1];
}
if (preg_match('/Content-Length: (\d+)/', $data, $matches)) {
$contentLength = (int)$matches[1];
}
echo 'HTTP Status: ' . $status . "\n";
echo 'Content-Length: ' . $contentLength;
?>
Result:
HTTP Status: 302
Content-Length: 8808759
Another solution. http://www.php.net/manual/en/function.filesize.php#90913
This is just a two step process:
Crawl the the js file and store it to a variable
Check if the length of the js file is greater than 0
thats it!!
Here is how you can do it in PHP
<?php
$data = file_get_contents('http://www.foo.com/foo.js');
if(strlen($data)>0):
echo "yay"
else:
echo "nay"
?>
Note: You can use HTTP Head as suggested by Uku but then if you are seeking for the page content if js file has content then you would have to crawl again :(

Categories