extending php execution time - php

Question

Have tried to extend the execution time by these methods but the script keeps ending prematurely. It is basically cycling through a mysql database and doing something with each row. It should be lasting maybe five to ten minutes but stops consistently early at the same spot. I have tried the following:
set_time_limit(0);
curl_setopt($ch, CURLOPT_TIMEOUT, 2000000);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 200000);
None of them are working. Please help!
<?php
ini_set('max_execution_time', 500);
$user="5aabf73bdd2c7";
//$user = $_POST['user'];
$servername = "localhost";
$username = "placeprint_1";
$password = "JS313833";
$dbname = "placeprint_1";
// Create connection
$con = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($con->connect_error) {
die("Connection failed: " . $con->connect_error);
}
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 2000000);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 200000);
curl_setopt($ch,CURLOPT_BINARYTRANSFER, true);
$data = curl_exec($ch);
curl_close ( $ch );
return $data;
}
$sql = "SELECT link, id, cookie FROM rawlinks WHERE cookie='$user' ORDER by ID desc ";
$result = $con->query($sql);
if ($result->num_rows > 0) {
// output data of each row
while($row = $result->fetch_assoc()) {
$getimagefrom=$row["link"];
$id=$row["id"];
$cookie=$row["cookie"];
echo $getimagefrom;
echo "<br><br>ID:".$id."<br><br>";
$htmlaa = file_get_contents_curl($getimagefrom);
$docaa = new DOMDocument();
#$docaa->loadHTML($htmlaa);
$nodesaa = $docaa->getElementsByTagName('title');
$nodesxaa = $docaa->getElementsByTagName('img');
//get and display what you need:
$titlev = $nodesaa->item(0)->nodeValue;
$metasv = $docaa->getElementsByTagName('meta');
$asv = $docaa->getElementsByTagName('img');
for ($iv = 0; $iv < $metasv->length; $iv++)
{
$metav = $metasv->item($iv);
if($metav->getAttribute('name') == 'description')
$descriptionv = $metav->getAttribute('content');
if($metav->getAttribute('name') == 'keywords')
$keywordsv = $metav->getAttribute('content');
if($metav->getAttribute('property') == 'og:image');
$languagev = $metav->getAttribute('content');
}
for ($ivv = 0; $ivv < $asv->length; $ivv++)
{
$av = $asv->item($ivv);
echo $av->getAttribute('src');
echo "<br>";
$cvarv = $av->getAttribute('src');
echo " <img src='$cvarv' >";
$servername = "localhost";
$username = "placeprint_1";
$password = "JS313833";
$dbname = "placeprint_1";
// Create connection
$con = new mysqli($servername, $username, $password, $dbname);
// Check connection
if ($con->connect_error) {
die("Connection failed: " . $con->connect_error);
}
$getimagefrom = rtrim($getimagefrom, '/');
$cvarv = urlencode($cvarv);
$titlev = preg_replace('/[^\p{L}\p{N}\s]/u', '', $titlev);
$descriptionv = preg_replace('/[^\p{L}\p{N}\s]/u', '', $descriptionv);
}
}
}
$con->close();
echo "FINISHED!!!!";
?>

Answer 1

From php.ini try to increase the max_execution_time and restart apache server.
To locate where your php.ini is you can create a phpinfo.php file and insert the following:
<?php phpinfo();
More informarion here: https://mediatemple.net/community/products/dv/204643880/how-can-i-create-a-phpinfo.php-page
You can then get the location of the php.ini configuration file location when you visit the phpinfo.php on your browser.
From the code I can suggest that instead of doing an INSERT mysql request for every loop, you should prepare the insert values in the loop and then perform the actual mysql request only once after the loop is finished.
Another suggestion is to have curl_close ( $ch ); before the return in file_get_contents_curl()
I have updated your code in order to be able to work in chunks instead of doing all the execution at once. I have created a table import_progress where we will store the last imported/executed row and fetch 20 rows for every execution:
/**
* Get the last imported ID
*
* #param mysqli $con
*
* #return int
*/
function get_last_inserted_id( $con ) {
$sql = "SELECT imported_id
FROM import_progress
ORDER by ID desc
LIMIT 1";
$result = $con->query( $sql );
if ( $result->num_rows == 0 ) {
//Since you are fetching in descending order
//We need to return a very high number if we did not import anything yet
return 999999999999999999;
} else {
while ( $row = $result->fetch_assoc() ) {
return $row['imported_id'];
}
}
}
/**
* Get the last imported ID
*
* #param mysqli $con
* #param array $ids
*
* #return int
*/
function insert_imported_ids( $con, $ids ) {
$ids = array_unique( array_filter( $ids ) );
if ( empty( $ids ) ) {
return false;
}
$sql = "INSERT INTO import_progress (imported_id) VALUES";
//We now append the data to the mysql string
$insert = [];
foreach ( $ids as $id ) {
$insert[] = "($id)";
}
$sql .= implode( ',', $insert ) . ';';
return $con->query( $sql );
}
/**
* #param string $url
*
* #return mixed
*/
function file_get_contents_curl( $url ) {
$ch = curl_init();
curl_setopt( $ch, CURLOPT_HEADER, 0 );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
curl_setopt( $ch, CURLOPT_TIMEOUT, 2000000 );
curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, 200000 );
curl_setopt( $ch, CURLOPT_BINARYTRANSFER, true );
$data = curl_exec( $ch );
curl_close( $ch );
return $data;
}
$user = "5aabf73bdd2c7";
$servername = "db";
$username = "root";
$password = "root";
$dbname = "stack_overflow";
// Create connection
$con = new mysqli( $servername, $username, $password, $dbname );
// Check connection
if ( $con->connect_error ) {
die( "Connection failed: " . $con->connect_error );
}
$last_inserted_id = get_last_inserted_id( $con );
$sql = "SELECT link, id, cookie FROM rawlinks
WHERE cookie='$user'
AND ID < $last_inserted_id
ORDER by ID desc
LIMIT 20";
$result = $con->query( $sql );
if ( $result->num_rows > 0 ) {
$imported_ids = [];
// output data of each row
while ( $row = $result->fetch_assoc() ) {
$getimagefrom = $row["link"];
$id = $row["id"];
$cookie = $row["cookie"];
echo $getimagefrom;
echo "<br><br>ID:" . $id . "<br><br>";
$htmlaa = file_get_contents_curl( $getimagefrom );
$docaa = new DOMDocument();
#$docaa->loadHTML( $htmlaa );
$nodesaa = $docaa->getElementsByTagName( 'title' );
$nodesxaa = $docaa->getElementsByTagName( 'img' );
//get and display what you need:
$titlev = $nodesaa->item( 0 )->nodeValue;
$metasv = $docaa->getElementsByTagName( 'meta' );
$asv = $docaa->getElementsByTagName( 'img' );
for ( $iv = 0; $iv < $metasv->length; $iv ++ ) {
$metav = $metasv->item( $iv );
if ( $metav->getAttribute( 'name' ) == 'description' ) {
$descriptionv = $metav->getAttribute( 'content' );
}
if ( $metav->getAttribute( 'name' ) == 'keywords' ) {
$keywordsv = $metav->getAttribute( 'content' );
}
if ( $metav->getAttribute( 'property' ) == 'og:image' ) {
;
}
$languagev = $metav->getAttribute( 'content' );
}
for ( $ivv = 0; $ivv < $asv->length; $ivv ++ ) {
$av = $asv->item( $ivv );
echo $av->getAttribute( 'src' );
echo "<br>";
$cvarv = $av->getAttribute( 'src' );
echo " <img src='$cvarv' >";
$cvarv = urlencode( $cvarv );
$titlev = preg_replace( '/[^\p{L}\p{N}\s]/u', '', $titlev );
$descriptionv = preg_replace( '/[^\p{L}\p{N}\s]/u', '', $descriptionv );
}
$imported_ids[] = (int)$id;
}
insert_imported_ids($con, $imported_ids);
}
$con->close();
echo "FINISHED!!!!";

Answer 2

Curl doesn't make any difference. If set_time_limit doesn't work it means that a global configuration overrides it, try printing the following:
var_dump(ini_get('max_execution_time'));
One time before executing set_time_limit(0); and also after you've done it.
This could be editable by php.ini but also is typically a blocked feature in free or shared hosting providers to prevent users to over-load the shared server.

Answer 3

add
ini_set('max_execution_time', 500);
at the top of the page, here 500 is number of seconds.