I have this function that gets the html from a list of pages and once I run it for
two hours or so the script interrupts and shows that memory limit has been exceeded,
Now i've tried to unset/set to null some variables hopefully to free up some memory
but it's the same problem. Can you guys please take a look at the following piece of
code? :
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
ob_start();
return curl_exec($ch); // the line the script interrupts because of memory
ob_end_clean();
curl_close($ch);
ob_flush();
$site = null;
$ch = null;
}
Any suggestion is highly appreciated. I've set the memory limit to 128M, but before
increasing it (doesnt seem like the best option to me) I would like to know if there's
anything I can do to use less memory/free up memory while running the script.
Thank you.
You are indeed leaking memory. Remember that return immediately ends execution of the current function, so all your cleanup (most importantly ob_end_clean() and curl_close()) is never called.
return should be the very last thing the function does.
I know it's been a while, but others might run into a similar issue, so in case it helps anyone else...
To me the problem here is that curl is set to save the output to a string. [That's what happens with curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);] If the output gets too long, the script will run out of allowed memory for that string. [That returns an error like FATAL ERROR: Allowed memory size of 134217728 bytes exhausted (tried to allocate 130027520 bytes)] The way around this is to use one of the other output methods offered by curl: output to standard output, or output to file. In either case, ob-start shouldn't be needed at all.
Hence you could replace the content of the braces with either option below:
OPTION 1: Output to standard output:
$ch = curl_init();
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
curl_exec($ch);
curl_close($ch);
OPTION 2: Output to file:
$file = fopen("path_to_file", "w"); //place this outside the braces if you want to output the content of all iterations to the same file
$ch = curl_init();
if ($proxystatus == 'on'){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, TRUE);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
}
curl_setopt($curl, CURLOPT_FILE, $file);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_URL, $site);
curl_exec($ch);
curl_close($ch);
fclose($file); //place this outside of the braces if you want to output the content of all iterations to the same file
For sure this is not a cURL issue. Use tools like xdebug to detect which part of your script is consuming memory.
Btw I would also change it not to run for two hours, I will move it to a cronjob that runs everyminute, check what it needs and then stops.
In PHP, I see the word cURL in many PHP projects. What is it? How does it work?
Reference Link: cURL
cURL is a library that lets you make HTTP requests in PHP. Everything you need to know about it (and most other extensions) can be found in the PHP manual.
In order to use PHP's cURL functions
you need to install the ยป libcurl
package. PHP requires that you use
libcurl 7.0.2-beta or higher. In PHP
4.2.3, you will need libcurl version 7.9.0 or higher. From PHP 4.3.0, you will need a libcurl version that's
7.9.8 or higher. PHP 5.0.0 requires a libcurl version 7.10.5 or greater.
You can make HTTP requests without cURL, too, though it requires allow_url_fopen to be enabled in your php.ini file.
// Make a HTTP GET request and print it (requires allow_url_fopen to be enabled)
print file_get_contents('http://www.example.com/');
cURL is a way you can hit a URL from your code to get a html response from it. cURL means client URL which allows you to connect with other URLs and use their responses in your code.
CURL in PHP:
Summary:
The curl_exec command in PHP is a bridge to use curl from console. curl_exec makes it easy to quickly and easily do GET/POST requests, receive responses from other servers like JSON and download files.
Warning, Danger:
curl is evil and dangerous if used improperly because it is all about getting data from out there in the internet. Someone can get between your curl and the other server and inject a rm -rf / into your response, and then why am I dropped to a console and ls -l doesn't even work anymore? Because you mis underestimated the dangerous power of curl. Don't trust anything that comes back from curl to be safe, even if you are talking to your own servers. You could be pulling back malware to relieve fools of their wealth.
Examples:
These were done on Ubuntu 12.10
Basic curl from the commandline:
el#apollo:/home/el$ curl http://i.imgur.com/4rBHtSm.gif > mycat.gif
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 492k 100 492k 0 0 1077k 0 --:--:-- --:--:-- --:--:-- 1240k
Then you can open up your gif in firefox:
firefox mycat.gif
Glorious cats evolving Toxoplasma gondii to cause women to keep cats around and men likewise to keep the women around.
cURL example get request to hit google.com, echo to the commandline:
This is done through the phpsh terminal:
php> $ch = curl_init();
php> curl_setopt($ch, CURLOPT_URL, 'http://www.google.com');
php> curl_exec($ch);
Which prints and dumps a mess of condensed html and javascript (from google) to the console.
cURL example put the response text into a variable:
This is done through the phpsh terminal:
php> $ch = curl_init();
php> curl_setopt($ch, CURLOPT_URL, 'http://i.imgur.com/wtQ6yZR.gif');
php> curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
php> $contents = curl_exec($ch);
php> echo $contents;
The variable now contains the binary which is an animated gif of a cat, possibilities are infinite.
Do a curl from within a PHP file:
Put this code in a file called myphp.php:
<?php
$curl_handle=curl_init();
curl_setopt($curl_handle,CURLOPT_URL,'http://www.google.com');
curl_setopt($curl_handle,CURLOPT_CONNECTTIMEOUT,2);
curl_setopt($curl_handle,CURLOPT_RETURNTRANSFER,1);
$buffer = curl_exec($curl_handle);
curl_close($curl_handle);
if (empty($buffer)){
print "Nothing returned from url.<p>";
}
else{
print $buffer;
}
?>
Then run it via commandline:
php < myphp.php
You ran myphp.php and executed those commands through the php interpreter and dumped a ton of messy html and javascript to screen.
You can do GET and POST requests with curl, all you do is specify the parameters as defined here: Using curl to automate HTTP jobs
Reminder of danger:
Be careful dumping curl output around, if any of it gets interpreted and executed, your box is owned and your credit card info will be sold to third parties and you'll get a mysterious $900 charge from an Alabama one-man flooring company that's a front for overseas credit card fraud crime ring.
cURL is a way you can hit a URL from your code to get a HTML response from it. It's used for command line cURL from the PHP language.
<?php
// Step 1
$cSession = curl_init();
// Step 2
curl_setopt($cSession,CURLOPT_URL,"http://www.google.com/search?q=curl");
curl_setopt($cSession,CURLOPT_RETURNTRANSFER,true);
curl_setopt($cSession,CURLOPT_HEADER, false);
// Step 3
$result=curl_exec($cSession);
// Step 4
curl_close($cSession);
// Step 5
echo $result;
?>
Step 1: Initialize a curl session using curl_init().
Step 2: Set option for CURLOPT_URL. This value is the URL which we are sending the request to. Append a search term curl using parameter q=. Set option for CURLOPT_RETURNTRANSFER. True will tell curl to return the string instead of print it out. Set option for CURLOPT_HEADER, false will tell curl to ignore the header in the return value.
Step 3: Execute the curl session using curl_exec().
Step 4: Close the curl session we have created.
Step 5: Output the return string.
public function curlCall($apiurl, $auth, $rflag)
{
$ch = curl_init($apiurl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
if($auth == 'auth') {
curl_setopt($ch, CURLOPT_USERPWD, "passw:passw");
} else {
curl_setopt($ch, CURLOPT_USERPWD, "ss:ss1");
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$dt = curl_exec($ch);
curl_close($ch);
if($rflag != 1) {
$dt = json_decode($dt,true);
}
return $dt;
}
This is also used for authentication. We can also set the username and password for authentication.
For more functionality, see the user manual or the following tutorial:
http://php.net/manual/en/ref.curl.php
http://www.startutorial.com/articles/view/php-curl
Firstly let us understand the concepts of curl, libcurl and PHP/cURL.
curl: A command line tool for getting or sending files using URL syntax.
libcurl: a library created by Daniel Stenberg, that allows you to connect and communicate to many different types of servers with many different types of protocols. libcurl currently supports the http, https, ftp, gopher, telnet, dict, file, and ldap protocols. libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading (this can also be done with PHP's ftp extension), HTTP form based upload, proxies, cookies, and user+password authentication.
PHP/cURL: The module for PHP that makes it possible for PHP programs to use libcurl.
How to use it:
step1: Initialize a curl session use curl_init().
step2: Set option for CURLOPT_URL. This value is the URL which we are sending the request to.Append a search term "curl" using parameter "q=".Set option CURLOPT_RETURNTRANSFER, true will tell curl to return the string instead ofprint it out. Set option for CURLOPT_HEADER, false will tell curl to ignore the header in the return value.
step3: Execute the curl session using curl_exec().
step4: Close the curl session we have created.
step5: Output the return string.
Make DEMO :
You will need to create two PHP files and place them into a folder that your web server can serve PHP files from. In my case I put them into /var/www/ for simplicity.
1. helloservice.php and 2. demo.php
helloservice.php is very simple and essentially just echoes back any data it gets:
<?php
// Here is the data we will be sending to the service
$some_data = array(
'message' => 'Hello World',
'name' => 'Anand'
);
$curl = curl_init();
// You can also set the URL you want to communicate with by doing this:
// $curl = curl_init('http://localhost/echoservice');
// We POST the data
curl_setopt($curl, CURLOPT_POST, 1);
// Set the url path we want to call
curl_setopt($curl, CURLOPT_URL, 'http://localhost/demo.php');
// Make it so the data coming back is put into a string
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
// Insert the data
curl_setopt($curl, CURLOPT_POSTFIELDS, $some_data);
// You can also bunch the above commands into an array if you choose using: curl_setopt_array
// Send the request
$result = curl_exec($curl);
// Get some cURL session information back
$info = curl_getinfo($curl);
echo 'content type: ' . $info['content_type'] . '<br />';
echo 'http code: ' . $info['http_code'] . '<br />';
// Free up the resources $curl is using
curl_close($curl);
echo $result;
?>
2.demo.php page, you can see the result:
<?php
print_r($_POST);
//content type: text/html; charset=UTF-8
//http code: 200
//Array ( [message] => Hello World [name] => Anand )
?>
The cURL extension to PHP is designed to allow you to use a variety of web resources from within your PHP script.
cURL in PHP is a bridge to use command line cURL from the php language
cURL
cURL is a way you can hit a URL from your code to get a HTML response from it.
It's used for command line cURL from the PHP language.
cURL is a library that lets you make HTTP requests in PHP.
PHP supports libcurl, a library created by Daniel Stenberg, that allows you to connect and communicate to many different types of servers with many different types of protocols. libcurl currently supports the http, https, ftp, gopher, telnet, dict, file, and ldap protocols. libcurl also supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading (this can also be done with PHP's ftp extension), HTTP form based upload, proxies, cookies, and user+password authentication.
Once you've compiled PHP with cURL support, you can begin using the cURL functions. The basic idea behind the cURL functions is that you initialize a cURL session using the curl_init(), then you can set all your options for the transfer via the curl_setopt(), then you can execute the session with the curl_exec() and then you finish off your session using the curl_close().
Sample Code
// error reporting
error_reporting(E_ALL);
ini_set("display_errors", 1);
//setting url
$url = 'http://example.com/api';
//data
$data = array("message" => "Hello World!!!");
try {
$ch = curl_init($url);
$data_string = json_encode($data);
if (FALSE === $ch)
throw new Exception('failed to initialize');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array( 'Content-Type: application/json', 'Content-Length: ' . strlen($data_string)));
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
$output = curl_exec($ch);
if (FALSE === $output)
throw new Exception(curl_error($ch), curl_errno($ch));
// ...process $output now
} catch(Exception $e) {
trigger_error(sprintf(
'Curl failed with error #%d: %s',
$e->getCode(), $e->getMessage()),
E_USER_ERROR);
}
For more information, please check -
cURL
cURL Functions
Php curl function (POST,GET,DELETE,PUT)
function curl($post = array(), $url, $token = '', $method = "POST", $json = false, $ssl = true){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
if($method == 'POST'){
curl_setopt($ch, CURLOPT_POST, 1);
}
if($json == true){
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json','Authorization: Bearer '.$token,'Content-Length: ' . strlen($post)));
}else{
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-www-form-urlencoded'));
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSLVERSION, 6);
if($ssl == false){
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
}
// curl_setopt($ch, CURLOPT_HEADER, 0);
$r = curl_exec($ch);
if (curl_error($ch)) {
$statusCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$err = curl_error($ch);
print_r('Error: ' . $err . ' Status: ' . $statusCode);
// Add error
$this->error = $err;
}
curl_close($ch);
return $r;
}
Php curl class (GET,POST,FILES UPLOAD, SESSIONS, SEND POST JSON, FORCE SELFSIGNED SSL/TLS):
<?php
// Php curl class
class Curl {
public $error;
function __construct() {}
function Get($url = "http://hostname.x/api.php?q=jabadoo&txt=gin", $forceSsl = false,$cookie = "", $session = true){
// $url = $url . "?". http_build_query($data);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if($session){
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch , CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch , CURLOPT_COOKIEFILE, 'cookies.txt');
}
if($forceSsl){
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // 1, 2
}
if(!empty($cookie)){
curl_setopt($ch, CURLOPT_COOKIE, $cookie); // "token=12345"
}
$info = curl_getinfo($ch);
$res = curl_exec($ch);
if (curl_error($ch)) {
$this->error = curl_error($ch);
throw new Exception($this->error);
}else{
curl_close($ch);
return $res;
}
}
function GetArray($url = "http://hostname.x/api.php", $data = array("name" => "Max", "age" => "36"), $forceSsl = false, $cookie = "", $session = true){
$url = $url . "?". http_build_query($data);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if($session){
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch , CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch , CURLOPT_COOKIEFILE, 'cookies.txt');
}
if($forceSsl){
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // 1, 2
}
if(!empty($cookie)){
curl_setopt($ch, CURLOPT_COOKIE, $cookie); // "token=12345"
}
$info = curl_getinfo($ch);
$res = curl_exec($ch);
if (curl_error($ch)) {
$this->error = curl_error($ch);
throw new Exception($this->error);
}else{
curl_close($ch);
return $res;
}
}
function PostJson($url = "http://hostname.x/api.php", $data = array("name" => "Max", "age" => "36"), $forceSsl = false, $cookie = "", $session = true){
$data = json_encode($data);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if($session){
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch , CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch , CURLOPT_COOKIEFILE, 'cookies.txt');
}
if($forceSsl){
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // 1, 2
}
if(!empty($cookie)){
curl_setopt($ch, CURLOPT_COOKIE, $cookie); // "token=12345"
}
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Authorization: Bearer helo29dasd8asd6asnav7ffa',
'Content-Type: application/json',
'Content-Length: ' . strlen($data))
);
$res = curl_exec($ch);
if (curl_error($ch)) {
$this->error = curl_error($ch);
throw new Exception($this->error);
}else{
curl_close($ch);
return $res;
}
}
function Post($url = "http://hostname.x/api.php", $data = array("name" => "Max", "age" => "36"), $files = array('ads/ads0.jpg', 'ads/ads1.jpg'), $forceSsl = false, $cookie = "", $session = true){
foreach ($files as $k => $v) {
$f = realpath($v);
if(file_exists($f)){
$fc = new CurlFile($f, mime_content_type($f), basename($f));
$data["file[".$k."]"] = $fc;
}
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_SAFE_UPLOAD, false); // !!!! required as of PHP 5.6.0 for files !!!
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2) Gecko/20100115 Firefox/3.6 (.NET CLR 3.5.30729)");
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if($session){
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch , CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch , CURLOPT_COOKIEFILE, 'cookies.txt');
}
if($forceSsl){
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); // 1, 2
}
if(!empty($cookie)){
curl_setopt($ch, CURLOPT_COOKIE, $cookie); // "token=12345"
}
$res = curl_exec($ch);
if (curl_error($ch)) {
$this->error = curl_error($ch);
throw new Exception($this->error);
}else{
curl_close($ch);
return $res;
}
}
}
?>
Example:
<?php
$urlget = "http://hostname.x/api.php?id=123&user=bax";
$url = "http://hostname.x/api.php";
$data = array("name" => "Max", "age" => "36");
$files = array('ads/ads0.jpg', 'ads/ads1.jpg');
$curl = new Curl();
echo $curl->Get($urlget, true, "token=12345");
echo $curl->GetArray($url, $data, true);
echo $curl->Post($url, $data, $files, true);
echo $curl->PostJson($url, $data, true);
?>
Php file: api.php
<?php
/*
$Cookie = session_get_cookie_params();
print_r($Cookie);
*/
session_set_cookie_params(9000, '/', 'hostname.x', isset($_SERVER["HTTPS"]), true);
session_start();
$_SESSION['cnt']++;
echo "Session count: " . $_SESSION['cnt']. "\r\n";
echo $json = file_get_contents('php://input');
$arr = json_decode($json, true);
echo "<pre>";
if(!empty($json)){ print_r($arr); }
if(!empty($_GET)){ print_r($_GET); }
if(!empty($_POST)){ print_r($_POST); }
if(!empty($_FILES)){ print_r($_FILES); }
// request headers
print_r(getallheaders());
print_r(apache_response_headers());
// Fetch a list of headers to be sent.
// print_r(headers_list());
?>