I'm Currently proxying an endpoint by running a cURL however the size of my cURL is about 10 times larger than the original API, Why is that, and how can I decrease the size? This is all JSON BTW.
Original API return size = 32.2kb
cURL return size = 488KB
And here is my cURL script:
$ch = curl_init();
// set url
$url = 'http://domain.com/api/v1';
// set options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 );
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
return $output;
ob_start('ob_gzhandler');
php output buffer controll was the fix to my problem. Thanks all that tried to help!
Related
This Code is Working Well
<?php
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "http://ipdev.in/");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
echo $output;
// close curl resource to free up system resources
curl_close($ch);
?>
But below two bunch of codes are not working. I am not getting an error but in these cases browser loading bar is just revolving and revolving and never stops. The page shows loading and loading for a long time but nothing loads from URL. Where is the problem ?
<?php
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "https://iiitd.ac.in/");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
echo $output;
// close curl resource to free up system resources
curl_close($ch);
?>
<?php
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "http://iiitd.ac.in");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
echo $output;
// close curl resource to free up system resources
curl_close($ch);
?>
the link https://iiitd.ac.in/ is redirecting to https://www.iiitd.ac.in/ so you need to modify your curl code. You need to set CURLOPT_FOLLOWLOCATION as true.
Have a look on below solution:
<?php
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "https://iiitd.ac.in/");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// added follow location
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
// $output contains the output string
$output = curl_exec($ch);
echo $output;
// close curl resource to free up system resources
curl_close($ch);
I'm doing a simple curl on this address: https://github.com/users/davidhariri/contributions_calendar_data
When i grab the result with this function:
function fetch_data($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);
$result = curl_exec($ch);
curl_close($ch);
print_r($result);
return $result;
}
The strings are correct, but the ints (the contributions) are wrong.
Results from curl
[...["2014/01/04",0],["2014/01/05",0],["2014/01/06",0],["2014/01/07",1],["2014/01/08",0]]
Results from just navigating to the address
[...["2014/01/04",0],["2014/01/05",0],["2014/01/06",1],["2014/01/07",5],["2014/01/08",5]]
Something during the curl process might be transforming ints to binary and back again? I have no idea what's happening here.
Check that you are not logged in in the browser. You may get different results if so.
I have this function that gets the html from a list of pages and once I run it for
two hours or so the script interrupts and shows that memory limit has been exceeded,
Now i've tried to unset/set to null some variables hopefully to free up some memory
but it's the same problem. Can you guys please take a look at the following piece of
code? :
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start();
return curl_exec($ch); // the line the script interrupts because of memory
ob_end_clean();
curl_close($ch);
ob_flush();
$site = null;
$ch = null;
}
Any suggestion is highly appreciated. I've set the memory limit to 128M, but before
increasing it (doesnt seem like the best option to me) I would like to know if there's
anything I can do to use less memory/free up memory while running the script.
Thank you.
You are indeed leaking memory. Remember that return immediately ends execution of the current function, so all your cleanup (most importantly ob_end_clean() and curl_close()) is never called.
return should be the very last thing the function does.
I know it's been a while, but others might run into a similar issue, so in case it helps anyone else...
To me the problem here is that curl is set to save the output to a string. [That's what happens with curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);] If the output gets too long, the script will run out of allowed memory for that string. [That returns an error like FATAL ERROR: Allowed memory size of 134217728 bytes exhausted (tried to allocate 130027520 bytes)] The way around this is to use one of the other output methods offered by curl: output to standard output, or output to file. In either case, ob-start shouldn't be needed at all.
Hence you could replace the content of the braces with either option below:
OPTION 1: Output to standard output:
$ch = curl_init();
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
curl_exec($ch);
curl_close($ch);
OPTION 2: Output to file:
$file = fopen("path_to_file", "w"); //place this outside the braces if you want to output the content of all iterations to the same file
$ch = curl_init();
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($curl, CURLOPT_FILE, $file);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
curl_exec($ch);
curl_close($ch);
fclose($file); //place this outside of the braces if you want to output the content of all iterations to the same file
For sure this is not a cURL issue. Use tools like xdebug to detect which part of your script is consuming memory.
Btw I would also change it not to run for two hours, I will move it to a cronjob that runs everyminute, check what it needs and then stops.
I'm building a video converter for someone. All I need to know is, When someone pastes in a URL to a video, how can I download that video to the server? Anything here would be helpful.
If you want to use it in php try curl:
function curl_download($Url){
// is cURL installed yet?
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
// OK cool - then let's create a new cURL resource handle
$ch = curl_init();
// Now set some options (most are optional)
// Set URL to download
curl_setopt($ch, CURLOPT_URL, $Url);
// Set a referer
curl_setopt($ch, CURLOPT_REFERER, "http://www.example.org/yay.htm");
// User agent
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
// Include header in result? (0 = yes, 1 = no)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Should cURL return or print out the data? (true = return, false = print)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Timeout in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
// Download the given URL, and return output
$output = curl_exec($ch);
// Close the cURL resource, and free system resources
curl_close($ch);
return $output;
}
[Source]
What kind of video? If you have the actual video sources then this is your best bet:
download-file-to-server-from-url
If you do not, then chances are you will have to write your own video ripper such as http://keepvid.com
I get files by their urls by this code
file_get_contents($_POST['url'];
Then I do something with them.
But I don't want to operate with big files, how do I limit size of received file?
It should throw an error if file is bigger than 500kb.
See my answer to this question. You need to have the cURL extension, with which you can make a HEAD HTTP request to the remote server. The response will let you know how big the file is, and you can then decide accordingly.
You are interested specifically in this line:
$size = curl_getinfo($ch, CURLINFO_CONTENT_LENGTH_DOWNLOAD);
Agree with #Jon
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url); //specify the url
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$head = curl_exec($ch);
$size = curl_getinfo($ch,CURLINFO_CONTENT_LENGTH_DOWNLOAD);
if(<limit the $size>){
file_get_contents($url);
}