I have to read an URL html content about ~1 MB, exactly is 926 KB.
And I already create 2 functions.
An URL content with filesize ~1 MB:
https://example.com/html_1MB_Content.html
And here are 2 functions that I was created:
function getContent1 ($url) {
$file_handle = fopen($url, "r");
while (!feof($file_handle)) {
$line = fgets($file_handle);
echo $line;
}
fclose($file_handle);
}
function getContent2 ($url) {
$handle = curl_init($url);
curl_setopt_array($handle, array(
CURLOPT_USERAGENT => $_SERVER['HTTP_USER_AGENT'],
CURLOPT_ENCODING => '',
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_FOLLOWLOCATION => 1
));
$curl_response = curl_exec($handle);
curl_close($handle);
return $curl_response;
}
$testUrl = 'https://example.com/html_1MB_Content.html';
$result1 = getContent1 ($testUrl);
$result2 = getContent2 ($testUrl);
What I want is fastest and less memory. Which is best in this case?
One more question is anyway to read page content from bottom to top, if found data stop reading?
if you want to know how long it takes to execute your code, you may use these couple of code..
//put this to the start of your code..
$time_start = microtime(true);
//here goes your code...
//and put this to the end of your code...
echo 'Total execution time in seconds: ' . (microtime(true) - $time_start).'. memory used in kb : '.echo memory_get_usage();
this will show you performance time in second and size of used memory in KB...
Related
I am trying to send curl request from source to destination in loop. Loop runs for 2 times. First request lasts for 32 seconds and second one for 50 seconds. Finally times out. Controlling timeout is not in my control as it is shared hosting.
Source section below is being run in the browser. the below error message shows after using up 120 seconds
Error Details: Fatal error: Maximum execution time of 120 seconds
exceeded
Question
I am assuming that the request should not timeout, since both requests are submitted separately through their own curl request. Still, it seems like it is getting consolidated to form total one request.
In case I run the loop for one time, then everything works as it takes 30 seconds.
Am I missing anything?
Source
for($i = 0; $i <= 200; $i+= 100) {
$postData = array(
'start' => $i,
'end' => $i + 100
);
$ch = curl_init('Server url');
curl_setopt_array($ch, array(
CURLOPT_POST => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json'
),
CURLOPT_POSTFIELDS => json_encode($postData)
));
$response = curl_exec($ch);
$responseData = json_decode($response, TRUE);
curl_close($ch);
echo $response;
}
Destination
public function methodname()
{
$json = json_decode(file_get_contents('php://input'), true);
// .
// .
// Logic that runs for 32 seconds
// .
// .
header('Content-type: application/json');
echo json_encode("message");
}
Try to add a sleep(1) function inner your loop. It could be that the server which you are requested dont like multiple POST request in a short time.
try using cURl's CURLOPT_TIMEOUT or similar configurations. More information https://www.php.net/manual/en/function.curl-setopt.php here
Answer: read the documentation
LE:
You could also use set_time_limit(0); // or value > 120 to increase your script execution timeout
I have a custom cURL function that has to download huge number of images from remote server. I was banned a couple of times before when I used file_get_contents(). I found that curl_multi_init() is better option as with 1 connection it can download for example 20 images at once.
I made a custom functions that uses curl_init() and I am trying to figure out how I can implement curl_multi_init() so in my LOOP where I grab the list of 20 URLs from the database I can call my custom function and at the last loop to use curl_close(). At the current situation my function generates connection for each url in the LOOP. Here is the function:
function downloadUrlToFile($remoteurl,$newfileName){
$errors = 0;
$options = array(
CURLOPT_FILE => fopen('../images/products/'.$newfileName, 'w'),
CURLOPT_TIMEOUT => 28800,
CURLOPT_URL => $remoteurl,
CURLOPT_RETURNTRANSFER => 1
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$imageString =curl_exec($ch);
$image = imagecreatefromstring($imageString);
if($imageString !== false AND !empty($imageString)){
if ($image !== false){
$width_orig = imagesx($image);
if($width_orig > 1000){
$saveimage = copy_and_resize_remote_image_product($image,$newfileName);
}else $saveimage = file_put_contents('../images/products/'.$newfileName,$imageString);
}else $errors++;
}else $errors++;
curl_close($ch);
return $errors;
}
There has to be a way to use curl_multi_init() and my function downloadUrlToFile because:
I need to change the file name on the fly
In my function I am also checking several things for the remote image.. In the sample function I check the size only and resize it if neccessary but there is much more things done by this function (I cutted that part for shorter, but I also use the function to pass more variables..)
How should the code be changed so during the LOOP to connect only once to the remote server?
Thanks in advance
Try this pattern for Multi CURL
$urls = array($url_1, $url_2, $url_3);
$content = array();
$ch = array();
$mh = curl_multi_init();
foreach( $urls as $index => $url ) {
$ch[$index] = curl_init();
curl_setopt($ch[$index], CURLOPT_URL, $url);
curl_setopt($ch[$index], CURLOPT_HEADER, 0);
curl_setopt($ch[$index], CURLOPT_RETURNTRANSFER, true);
curl_multi_add_handle($mh, $ch[$index]);
}
$active = null;
for(;;) {
curl_multi_exec($mh, $active);
if($active < 1){
// all downloads completed
break;
}else{
// sleep-wait for more data to arrive on socket.
// (without this, we would be wasting 100% cpu of 1 core while downloading,
// with this, we'll be using like 1-2% cpu of 1 core instead.)
curl_multi_select($mh, 1);
}
}
foreach ( $ch AS $index => $c ) {
$content[$index] = curl_multi_getcontent($c);
curl_multi_remove_handle($mh, $c);
//You can add some functions here and use $content[$index]
}
curl_multi_close($mh);
I can't allow for file_get_contents to work more than 1 second, if it is not possible - I need to skip to next loop.
for ($i = 0; $i <=59; ++$i) {
$f=file_get_contents('http://example.com');
if(timeout<1 sec) - do something and loop next;
else skip file_get_contents(), do semething else, and loop next;
}
Is it possible to make a function like this?
Actually I'm using curl_multi and I can't fugure out how to set timeout on a WHOLE curl_multi request.
If you are working with http urls only you can do the following:
$ctx = stream_context_create(array(
'http' => array(
'timeout' => 1
)
));
for ($i = 0; $i <=59; $i++) {
file_get_contents("http://example.com/", 0, $ctx);
}
However, this is just the read timeout, meaning the time between two read operations (or the time before the first read operation). If the download rate is constant, there should not being such gaps in the download rate and the download can take even an hour.
If you want the whole download not take more than a second you can't use file_get_contents() anymore. I would encourage to use curl in this case. Like this:
// create curl resource
$ch = curl_init();
for($i=0; $i<59; $i++) {
// set url
curl_setopt($ch, CURLOPT_URL, "example.com");
// set timeout
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
}
$ctx = stream_context_create(array(
'http' => array(
'timeout' => 1
)
)
);
file_get_contents("http://example.com/", 0, $ctx);
Source
I am using php 5.2 and I am fetching data from url using file_get_contents function. This is loop for 5000 and I have divided into 500 slots and set a script like this.
For 500 it is taking 3 hours to complete because for some url it is taking too much time and for some it is in 1 sec that is fine.
What I want if url is taking more than 30 sec then skip and go for next.
I want to stop fetch after 30 sec.
<?php
// Create the stream context
$context = stream_context_create(array(
'http' => array(
'timeout' => 1 // Timeout in seconds
)
));
// Fetch the URL's contents
echo date("Y-m-d H:i:s")."\n";
$contents = file_get_contents('http://example.com', 0, $context);
echo date("Y-m-d H:i:s")."\n";
// Check for empties
if (!empty($contents))
{
// Woohoo
// echo $contents;
echo "file fetched";
}
else
{
echo $contents;
echo "more than 30 sec";
}
?>
I have already done that it is not working for me because file_get_contents function is not stoping it will continue , then only thing now I am getting no result after 30 sec but time it is taking sameas u can see in output.
Output of php
2012-03-09 11:26:38
2012-03-09 11:26:40
more than 30 sec
You can set the HTTP timeout. (Not tested)
<?php
$ctx = stream_context_create(array(
'http' => array(
'timeout' => 30
)
));
file_get_contents("http://example.com/", 0, $ctx);
Source
Edit: I don't know why it isn't working with this code by you. But if you don't manage it to bring it to work with this you may also want to give CURL a try. This could be eventually also faster for that (but I don't know if that is really faster...).
If that would work for you, you could than use the curl_setopt function to set the timeout time with the CURLOPT_TIMEOUT flag.
There some info on the php manual about timeouts.
http://php.net/manual/en/function.file-get-contents.php
there is mention of the following as of php 5.2.1
ini_set('default_socket_timeout', 120);
$a = file_get_contents("http://abcxyz.com");
or adding a context which is more or less the same.
// Create the stream context
$context = stream_context_create(array(
'http' => array(
'timeout' => 3 // Timeout in seconds
)
));
// Fetch the URL's contents
$contents = file_get_contents('http://abcxyz.com', 0, $context);`
A third option is using PHP's fsockopen which has an explicit timeout option
http://www.php.net/manual/en/function.fsockopen.php
$timeout = 2; // seconds
$fp = fsockopen($url, 80, $errNo, $errString, $timeout);
/* stops connecting after 2 seconds,
stores the error Number in $errNo,
the error String in $errStr */
To save writing a lot of code, you could use it as a quick check if host is up.
ie:
if (pingLink($domain,$timeout)) {
file_get_contents()
}
function pingLink($domain,$timeout=30){
$status = 0; //default site is down
$file = fsockopen($domain,"r");
if ($file) {
$status = 1; // Site is up
fclose($file);
}
return $status;
}
Morning all,
Theres a few questions around this but none that really answer my question, as far as I ca understand. Basically I have a GD script that deals with resizing and caching images on our server, but I need to do the same with images stored on a remote server.
So, I'm wanting to save the image locally, then resize and display it as normal.
I've got this far...
$file_name_array = explode('/', $filename);
$file_name_array_r = array_reverse($file_name_array);
$save_to = 'system/cache/remote/'.$file_name_array_r[1].'-'.$file_name_array_r[0];
$ch = curl_init($filename);
$fp = fopen($save_to, "wb");
// set URL and other appropriate options
$options = array(CURLOPT_FILE => $fp,
CURLOPT_HEADER => 0,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_TIMEOUT => 60); // 1 minute timeout (should be enough)
curl_setopt_array($ch, $options);
curl_exec($ch);
curl_close($ch);
fclose($fp);
This creates the image file, but does not copy it accross? Am I missing the point?
Cheers guys.
More simple, you can use the file_put_contents instead of fwrite:
$file_name_array = explode('/', $filename);
$file_name_array_r = array_reverse($file_name_array);
$save_to = 'system/cache/remote/'.$file_name_array_r[1].'-'.$file_name_array_r[0];
file_put_contents($save_to, file_get_contents($filename));
or in just 2 lines :)
$file_name_array_r = array_reverse( explode('/', $filename) );
file_put_contents('system/cache/remote/'.$file_name_array_r[1].'-'.$file_name_array_r[0], file_get_contents($filename));
Well, I sorted it! After examing my images rather than my code a little closer, it turned out some of the images were erroring on their side, rather than mine. Once I selected an image that worked, my code worked too!
Cheers as always though guys :)
Personally i don't like to use curl functions that write into a file. Try this instead:
$file_name_array = explode('/', $filename);
$file_name_array_r = array_reverse($file_name_array);
$save_to = 'system/cache/remote/'.$file_name_array_r[1].'-'.$file_name_array_r[0];
$ch = curl_init($filename);
$fp = fopen($save_to, "wb");
// set URL and other appropriate options
$options = array(CURLOPT_HEADER => 0,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_TIMEOUT => 60,
CURLOPT_RETURNTRANSFER, true //Return transfer result
);
curl_setopt_array($ch, $options);
//Get the result of the request and write it into the file
$res=curl_exec($ch);
curl_close($ch);
fwrite($fp,$res);
fclose($fp);
But you can use something more simple without curl:
$file_name_array = explode('/', $filename);
$file_name_array_r = array_reverse($file_name_array);
$save_to = 'system/cache/remote/'.$file_name_array_r[1].'-'.$file_name_array_r[0];
$content=file_get_contents($filename);
$fp = fopen($save_to, "wb");
fwrite($fp,$content);
fclose($fp);