I'm having some problems looping this script through a large database of 1m+ items. The script returns the size of an image in bytes from it's url and inserts the result into a database.
I get the browser error Error code: ERR_EMPTY_RESPONSE on my test attempt. This doesn't bode well. Am I trying to loop through too many records with a while loop? Any methods for a fix?
<?php
error_reporting(E_ALL);
mysql_connect('xxxx', 'xxxx', 'xxxx') or die("Unable to connect to MySQL");
mysql_select_db('xxxx') or die("Could not select database");
$result = mysql_query("SELECT * FROM items");
if (mysql_num_rows($result)) {
while ($row = mysql_fetch_array($result)) {
$ch = curl_init($row['bigimg']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
$data = curl_exec($ch);
$info = curl_getinfo($ch, CURLINFO_CONTENT_LENGTH_DOWNLOAD);
curl_close($ch);
mysql_query("UPDATE items SET imgsize = '" . $info . "' WHERE id=" . $row['id'] . " LIMIT 1");
}
}
?>
I think your issue might be related to the fact you are trying to call eachtime curl_exec. You might want to change your code to this in 2 parts: first retrieve the data from the database and then make the curl calls.
Related
I have a php script that is checking the result of a virustotal scan. If the scan returns positive for malicious code it changes the value to 0 in the db. I have another php script which checks the value and if it is 0 it removes the entry from the db and then removes the file from the directory. When I run this through the command line it works perfectly, however when cron runs it, it does remove the db entry as it should however it does not delete the file from the directory.
Any help would be much appreciated.
Here is the end of the php file with the unlink:
else{
// if not it deletes the image
$hash = $row['hash'];
$result2 = mysqli_query($connection, "DELETE FROM userUploads WHERE hash = '$hash' ");
// need due to dir structure
$pat = str_replace('../','', $row['fileName']);
unlink ($pat);
if (! $result2){
echo('Database error: ' . mysqli_error($connection));
}
For reference, here is the full file:
<?php
function VirusTotalCheckHashOfSubmitedFile($hash){
$debug = false;
$post = array('apikey' => '','resource'=>$hash);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.virustotal.com/vtapi/v2/file/report');
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate'); // please compress data
curl_setopt($ch, CURLOPT_USERAGENT, "gzip, My php curl client");
curl_setopt($ch, CURLOPT_VERBOSE, 1); // remove this if your not debugging
curl_setopt($ch, CURLOPT_RETURNTRANSFER ,true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$result = curl_exec ($ch);
$status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($status_code == 200) { // OK
$js = json_decode($result, true);
if($debug){echo "<pre>";}
if($debug){print_r($js);}
if($js["positives"]==0){
// clean
return true;
}else{
// malware
return false;
}
} else { // Error occured
print($result);
}
curl_close ($ch);
}
$connection = mysqli_connect("");
if (!$connection) {
trigger_error("Could not reach database!<br/>");
}
$db_selected = mysqli_select_db($connection, "seclog");
if (!$db_selected) {
trigger_error("Could not reach database!<br/>");
}
// Selecs images that have not been marked as clean by virus total
$result = mysqli_query($connection, "Select hash, fileName FROM userUploads WHERE scanResult = 0");
if (! $result){
echo('Database error: ' . mysqli_error($connection));
}
while ($row = mysqli_fetch_assoc($result)) {
// checks for results on scanned images
if(VirusTotalCheckHashOfSubmitedFile($row['hash'])){
// if report returns image is malware free we update its virusFree attribute to true
$hash = $row['hash'];
$result2 = mysqli_query($connection, "UPDATE userUploads SET scanResult = 1 WHERE hash = '$hash'");
if (! $result2){
echo('Database error: ' . mysqli_error($connection));
}
}else{
// if not it deletes the image
$hash = $row['hash'];
$result2 = mysqli_query($connection, "DELETE FROM userUploads WHERE hash = '$hash' ");
// need due to dir structure
$pat = str_replace('../','', $row['fileName']);
unlink ($pat);
if (! $result2){
echo('Database error: ' . mysqli_error($connection));
}
}
}
?>
The problem almost certainly is the path $pat = str_replace('../','', $row['fileName']);. Crons execute PHP cli, that is not the same PHP that Apache executes, also is another context. Try setting absolute path:
$pat = "/var/www/myfolder/myscript.some";
If for some reason you need a variable because folder structure depends of the context (e.g. development, production) you could pass the variable as a parameter when the cron executes PHP:
//this is the cron
30 17 * * 1 myscript.php myvar
Inside myscript.php $argv[1] is myvar.
I'm new to PHP, and I want to get latitude and longitude of a place and then add them to MySQL database.
I'm using Google Geo-code API to get them, this is what I do right-row
for ($i = 0; $i<1000; $i++) {
$sql = mysql_query("SELECT place_address FROM place_locator WHERE place_id =".$i, $this->db) or die('invalide request : ' . mysql_error());
if (mysql_num_rows($sql)) {
while ($place = mysql_fetch_assoc($sql)) {
//Encode the place string I got, to get rid of space
$encodePlace = str_replace(" ", "%20", $place["place_address"]);
//Use Google API
$url = 'http://maps.googleapis.com/maps/api/geocode/json?address='.$encodePlace.'&sensor=false';
//Use Curl to send the request
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
$obj = json_decode($response, true);
$updateSql = mysql_query("UPDATE `place_locator`.`place_locator` SET
`latitude` = '".$obj["results"][0]["geometry"]["location"]["lat"]."',
`longitude` = '".$obj["results"][0]["geometry"]["location"]["lng"]."' WHERE `place_locator`.`place_id` = ".$i, $this->db) or die('Invalide : ' . mysql_error());
curl_close($ch);
}
}
It works for a loop of 10,when going to 1000, it will take a lot of time and many results didn't updated to the database.
I think may be multi thread should help, but I don't really know how it works, please help me.
Thanks in advance
I had the same problem. Google limits the frequency of the requests! Try a sleep(1); in the loop and it will work but need much more time.
I am trying to execute a url and getting its response. Following is the code that executes the curl. I want the curl execution to return me a string in $result.
<?php
$fields = array
(
'username'=>urlencode($username),
'pwrd'=>urlencode($pwrd),
'customer_num'=>urlencode($customer_num)
);
$url = 'http://localhost/test200.php';
//open connection
set_time_limit(20);
$ch = curl_init();
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_POST,count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS,$fields_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//execute post
$result = curl_exec($ch);
//close connection
curl_close($ch);
echo $result; //I want $result to be "Successful"
?>
This is my test200.php on localhost:
<?php
$usernam = $_POST['username'];
$pass = $_POST['pwrd'];
$customer_num = $_POST['customer_num'];
echo "Successful!";
?>
What changes do I make in test200.php? Please help.
You should use the httpcode returned by the curl execution and not rely on a string that is returned
$res = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
Here - http://www.webmasterworld.com/forum88/12492.htm
Once the data is sent to test200.php do the appropriate manipulation like insert the posted values into a table and on success
echo "Successful!";
or print the same in your test200.php.. assuming you are doing an insert code in test200.php code would be like
<?php
$qry = "INSERT INTO `your_table` (`field_customer_name`, `field_username`, `field_password`) VALUES ($fields['customer_num'], $fields['username'], some_encrypt_fxn($fields['pwrd']))";
mysql_query($qry);
$err_flag = mysql_error($your_conn_link);
if($err_flag == '') {
echo "Successful!";
}
else {
echo "Failed, Error " . $err_flag;
}
?>
If the purpose of getting "Successful!" is to check if the cURL returns success then i suggest using Prateik's answer of using the returned status code
Somehow a simple print("Successful"); statement in test200.php worked well. The response i get now is as follows: HTTP Code: 0 Array ( [0] => [1] => Successful )
I have a little problem. I need to execute an script that execute 5000 URL in php.
$con = mysql_connect("localhost","user","psswd");
if (!$con) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db('db_name', $con);
print "connected";
$result = mysql_query ("SELECT name, uid FROM obinndocusers");
// I need to execute that url for each user
while ($row = mysql_fetch_array($result)) {
header (Location http:xxxxxxxx?q=user/" . $row['uid'] . "/edit&destination=admin/user/user);
}
Any idea??
Thx.
use CURL
LIKE :
$ch = curl_init();
while ($row = mysql_fetch_array($result)) {
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http://www.example.com?q=user/" . $row['uid'] . "/edit&destination=admin/user/user");
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);
}
// close cURL resource, and free up system resources
curl_close($ch);
Use cURL
<?php
// create a new cURL resource
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "http:xxxxxxxx?q=user/" . $row['uid'] . "/edit&destination=admin/user/user");
curl_setopt($ch, CURLOPT_HEADER, 0);
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
First thing: header() is a primitive to send http headers to browser. It must be called before any stdout output (like 'print' or 'echo').
Second thing: "Location: " header will tell your browser to redirect to that URL. You can not specify more that one URL.
If you need your script to do http queries, use curl or fopen, and do not call your script from your browser.
The best way would be with CURL (see other answer by Haim Evgi), but if the server doesn't have the curl extension, then this will also work.
<?
$con = mysql_connect("localhost","user","psswd");
if (!$con) {
die('Could not connect: ' . mysql_error());
}
mysql_select_db('db_name', $con);
print "connected";
$result = mysql_query ("SELECT name, uid FROM obinndocusers");
// I need to execute that url for each user
while ($row = mysql_fetch_array($result)) {
file_get_contents("http:xxxxxxxx?q=user/" . $row['uid'] . "/edit&destination=admin/user/user");
}
I am attempting to download images saved in a url to a local folder, and have made the following attempt at doing so with curl. I would like to know if it is necessary to include curl or download it seperateley, or if my function will just work. I would like to know if there are any obvious problems with my implementation below. I am aware of the sql vulnerability and I am switching to prepared statements. I have trimmed non relevant parts of the code for brevity.
edit: the function is out of the while loop. The page displays if I comment out the call the the function, otherwise I only get a blank page. Why is this
<?php
header("Content-Type: text/html; charset=utf-8");
if (isset($_GET["cmd"]))
$cmd = $_GET["cmd"];
else
die("You should have a 'cmd' parameter in your URL");
$pk = $_GET["pk"];
$con = mysql_connect("localhost","someuser","notreal");
if(!$con)
{
die('Connection failed because of' .mysql_error());
}
mysql_query('SET NAMES utf8');
mysql_select_db("somedb",$con);
if($cmd=="GetAuctionData")
{
$sql="SELECT * FROM AUCTIONS WHERE ARTICLE_NO ='$pk'";
$sql2="SELECT ARTICLE_DESC FROM AUCTIONS WHERE ARTICLE_NO ='$pk'";
$htmlset = mysql_query($sql2);
$row2 = mysql_fetch_array($htmlset);
$result = mysql_query($sql);
function savePicture($imageUrl) {
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $lastImg);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, 0);
$fileContents = curl_exec($ch);
curl_close($ch);
$newImg = imagecreatefromstring($fileContents);
return imagejpeg($newImg, "./{$pk}.jpg",100);
}
while ($row = mysql_fetch_array($result))
{
$lastImg = $row['PIC_URL'];
savePicture($lastImg);
<div id='rightlayer'>
<img src='./".$pk.".jpg' width='".$outputWidth."' height='".$outputHeight."'>
</div>
</div>
</div>";
}
}
mysql_free_result($result);
You’ll get an error if you declare a function inside a loop when the loop is run more than one time. So you should declare the savePicture function outside while.
I'd take the function definition out of the while block.
In my opinion your using curl for the sake of using curl here, a simpler method would be to use file get contents.