I have a cron job that runs once a week and checks a remote site for any updates, which may include new data, and may include new graphics.
I check my local folder to see if the graphic already exists, and if it doesnt then I want to save a local copy of that graphic.
My code seems to work when I run it from the browser, but when I check the folder every week, there are a lot of empty graphic files.
They have the correct filenames, but are all zero bytes.
My code inside the php file that the cron job runs is this:
if (!file_exists($graphic)) {
$imageString = file_get_contents($graphicurl);
file_put_contents($graphic, $imageString);
}
$graphic will be something like "filename.jpg"
$graphicurl will be something like "https://remotegraphicfile.jpg"
And I see many files such as "filename.jpg" that exist in my local folder, but with a zero byte filesize.
Is there any reason why this wouldnt work when called by a cron job?
I have now managed to get this working, by changing my original code from:
if (!file_exists($graphic)) {
$imageString = file_get_contents($graphicurl);
file_put_contents($graphic, $imageString);
}
To this:
if (!file_exists($graphic)) {
$file = fopen ($graphic, 'w');
$c = curl_init($graphicurl);
curl_setopt($c, CURLOPT_FILE, $file);
curl_exec($c);
curl_close($c);
fclose($file);
}
This now works, and graphics save to my folder when running the cron job.
Related
I have a little problem inside my application.
I have many process launched automatically, from the server with crontab, written in php to read file inside a folder.
Sometimes different process read the same file and create a problem inside the application.
Is there a way to manage this problem?
Actually I read all files inside a folder, read each of them and delete immediately, but sometimes another process read the same file before I delete It.
This is my script written with cakephp3 (so some classes like File is only for cakephp3 but isn't the point of the question) to read and delete:
$xml_files = glob(TMP . 'xml/*.xml');
foreach($xml_files as $fileXml)
{
//read the file and put into a string or array or object
$explStr = explode('/', $fileXml);
$filename = $explStr[count($explStr) - 1];
$path = TMP . '/xml/' . $filename;
$file = new File($path, false);
if($file->exists()){
$string = $file->read();
$file->close();
$file->delete();
}
}
Use flock() to obtain (or attempt to obtain) a file lock and act accordingly.
It's called a race condition, and when working with files you could lock the file when process A uses it, it locks it, then other processes would check if it's locked and if it is then do nothing. Then unlock the file when process A has finished with it.
My environment is: Windows, MsSQL and PHP 5.4.
My scenario:
I'm doing a small shell script that creates a full backup from my wanted database to a temp folder and then moves it to a new location.
The backup goes fine and the file is created to my temp folder. Then I rename it to the 2nd folder and sometimes it goes ok, sometimes it cannot find the source file.
Of course at this point I know that I could skip the temporary location alltogether, but the actual problem with not finding the file bothers me. Why is it so random and might it also affect other file functions I've written before this one... Also i need to be able to control how and when the files move to the destination.
The base code is simple as it should be (although this is a simplified version of my actual code, since I doubt anyone would be interested in my error handling/logging conditions):
$query = "use test; backup database test to disk '//server01/temp/backups/file.bak', COMPRESSION;";
if($SQLClass->query($query)) {
$source="////server01//temp//backups//file.bak";
$destination="////server02//storage//backups//file.bak";
if(!rename($source , $destination)) {
//handleError is just a class function of mine that logs and outputs errors.
$this->handleError("Moving {$source} to {$destination} failed.");
}
}
else {
die('backup failed');
}
What I have tried is:
I added a file_exists before it and it can't find the source file either, when rename can't.
As the file can't be found, copy() and unlink() will not work either
Tried clearstatcache()
Tried sleep(10) after the sql backup completes
None of these didn't help at all. I and google seem to be out of ideas on what to do or try next. Of course I could some shell_execing, but that wouldn't remove my worries about my earlier products.
I only noticed this problem when I tried to run the command multiple times in a row. Is there some sort of cache for filenames that clearstatcache() won't touch ? It seems to be related to some sort of ghost file phenomena, where php is late to refresh the file system contents or such.
I would appreciate any ideas on what to try next and if you read this far thank you :).
You may try calling system's copy command.
I had once problem like yours (on Linux box) when i had to copy files between two NFS shares. It just failed from time to time with no visible reasons. After i switched to cp (analog of Windows copy) problem has gone.
Surely it is not perfect, but it worked for me.
It might be cache-related, or the mysql process has not yet released the file.
mysql will dump the file into another temp file, first and finally moves it to your temp folder.
While the file is beeing moved, it might be inaccessible by other processes.
First I would try to glob() all the files inside temp dir, when the error appears. Maybe you notice, its still not finished.
Also have you tried to implemente something like 10 retry iterations, with some delay?
$notMoved = 0;
while($notMoved < 10){
$source="////server01//temp//backups//file.bak";
$destination="////server02//storage//backups//file.bak";
if(!rename($source , $destination)) {
//handleError is just a class function of mine that logs and outputs errors.
if ($notMoved++ < 10){
sleep(20);
} else {
$this->handleError("Moving {$source} to {$destination} failed.");
break;
}
}else{
break;
}
}
To bypass the issue:
Don't dump and move
Move then dump :-)
(ofc. your backup store would be one behind then)
$source="////server01//temp//backups//file.bak";
$destination="////server02//storage//backups//file.bak";
if(!rename($source , $destination)) {
//handleError is just a class function of mine that logs and outputs errors.
$this->handleError("Moving {$source} to {$destination} failed.");
}
$query = "use test; backup database test to disk '//server01/temp/backups/file.bak', COMPRESSION;";
if($SQLClass->query($query)) {
//done :-)
}
else {
die('backup failed');
}
Try
$source = "\\server01\temp\backups\file.bak";
$destination = "\\server02\storage\backups\file.bak";
$content = file_get_content($source);
file_put_contents($destination, $content);
I am trying to rename() a 900 KiB PDF file in PHP. It is taking a long time to rename it for some reason. I thought it should be instant.
This is on a CentOS server. While the file is being renamed I can get properties and it seems like rename() is copying and replacing the old file with new renamed file.
The old name and new name paths are in the same directory.
Has anyone stumbled upon this issue before?
Code:
//If exists change name and then return path
$pieces = explode("#", $filename);
$newName = $pieces[0].' '.$pieces[2];
rename($uidPath.$filename, $uidPath.$newName);
if (preg_match('/pdf/', $pieces[2]))
{
$result['status'] = '1';
$result['path'] = 'path to file';
}
else
{
$result['status'] = '1';
$result['path'] = 'path to file';
}
PHP is for some reason very slow to release file lock on fclose(), so if you are writing to the file prior to moving it you might have to wait for a bit. I've had this very problem with a low priority background job, so I didn't really look into why this happens or what I can do to prevent it - I just added 1 second sleep between fclose() and rename.
I am using this code to download a package from server A and put it in server B (copy)..but it does not work always, sometimes transfer does not get completed, file is not complete and some times it goes well. Can I improve this code in anyway or use cURL to do the same thing?
This is my code:
// from server a to server b
$filename = 'http://domain.com/file.zip';
$dest_folder = TEMPPATH.'/';
$out_file = #fopen(basename($filename), 'w');
$in_file = #fopen($filename, 'r');
if ($in_file && $out_file) {
while ($chunk = #fgets($in_file)) {
#fputs($out_file, $chunk);
}
#fclose($in_file);
#fclose($out_file);
$zip = new ZipArchive();
$result = $zip->open(basename($filename));
if ($result) {
$zip->extractTo($dest_folder);
$zip->close();
}
}
Problem is that it is not consistent. It does not get transfered all times, many times it comes missing and the script does not run well.
$filename = 'http://domain.com/file.zip';
echo `wget $filename`;
echo `unzip $filename`;
or
$ch = curl_init();
$timeout = 5;
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
$data = curl_exec($ch);
curl_close($ch);
fwrite(fopen($destfile,'w'),$data);
Really though, you need to figure out why it is failing. Is the zip operation killing it? Is the php script timing out because it took too long to execute? Is it running out of memory? Is the server on the other end timing out? Get some error reporting and debug data and try to figure out why it's not working. The code you have should be fine, and reliable.
Have you checked the timeout settings on your server. Maybe they are
causing your script to timeout before the code is executed
completely.
Make sure that you are allowed to open the external url by fopen in your servers settings. And you also have right access settings to get this file.
Make sure the firewall of the Server A is allowing Server B and not just blocking its ip.
Try to use curl or file_get_contents and file_put_contents. It is likely to work too and prevents the loops.
Check if the problem is with the ZipArchive class or getting the file itself.
The fact that you are having temperamental issues suggests you may be having the same issue I've encountered - which has nothing to do with the code.
I'm pulling my zip from a remote server using cURL and then extracting the locally saved zip. Sometimes it works, sometimes not... this caused some serious hair pulling to begin with.
I'm uploading my zip via filezilla and what I have found is it frequently crashes out, retries a few times and eventually works. The uploaded file has the correct file size and looks like it's successfully uploaded but if I download it again sometimes it's simply corrupted and can't be unzipped.
So long as I make sure my uploaded zip is fine my script works fine... so here it is:
$zip_url = "http://www.mydomain.com.au/";
$version = "1.0.1.zip"; // zip name
$ch = curl_init();
$tmp_zip = fopen($version, 'w'); // open local file for writing
curl_setopt($ch, CURLOPT_URL, "$zip_url$version"); // pull remote file
curl_setopt($ch, CURLOPT_FILE, $tmp_zip); // save to local file
$data = curl_exec($ch); // do execute
curl_close($ch);
fclose($tmp_zip); // close local file
// extract latest build
$zip = new ZipArchive;
$zip->open($version);
$result = $zip->extractTo("."); // extract to this directory
$zip->close();
if ($result) #unlink($version); // delete local zip if extracted
else echo "failed to unzip";
One big difference in my code from the previous answer is I'm using CURLOPT_FILE rather than CURLOPT_RETURNTRANSFER. You can read why CURLOPT_FILE is better for large transfers at:
www.phpriot.com/articles/download-with-curl-and-php
I have a directory with a number of subdirectories that users add files to via FTP. I'm trying to develop a php script (which I will run as a cron job) that will check the directory and its subdirectories for any changes in the files, file sizes or dates modified. I've searched long and hard and have so far only found one script that works, which I've tried to modify - original located here - however it only seems to send the first email notification showing me what is listed in the directories. It also creates a text file of the directory and subdirectory contents, but when the script runs a second time it seems to fall over, and I get an email with no contents.
Anyone out there know a simple way of doing this in php? The script I found is pretty complex and I've tried for hours to debug it with no success.
Thanks in advance!
Here you go:
$log = '/path/to/your/log.js';
$path = '/path/to/your/dir/with/files/';
$files = new RecursiveIteratorIterator(new RecursiveDirectoryIterator($path), RecursiveIteratorIterator::SELF_FIRST);
$result = array();
foreach ($files as $file)
{
if (is_file($file = strval($file)) === true)
{
$result[$file] = sprintf('%u|%u', filesize($file), filemtime($file));
}
}
if (is_file($log) !== true)
{
file_put_contents($log, json_encode($result), LOCK_EX);
}
// are there any differences?
if (count($diff = array_diff($result, json_decode(file_get_contents($log), true))) > 0)
{
// send email with mail(), SwiftMailer, PHPMailer, ...
$email = 'The following files have changed:' . "\n" . implode("\n", array_keys($diff));
// update the log file with the new file info
file_put_contents($log, json_encode($result), LOCK_EX);
}
I am assuming you know how to send an e-mail. Also, please keep in mind that the $log file should be kept outside the $path you want to monitor, for obvious reasons of course.
After reading your question a second time, I noticed that you mentioned you want to check if the files change, I'm only doing this check with the size and date of modification, if you really want to check if the file contents are different I suggest you use a hash of the file, so this:
$result[$file] = sprintf('%u|%u', filesize($file), filemtime($file));
Becomes this:
$result[$file] = sprintf('%u|%u|%s', filesize($file), filemtime($file), md5_file($file));
// or
$result[$file] = sprintf('%u|%u|%s', filesize($file), filemtime($file), sha1_file($file));
But bare in mind that this will be much more expensive since the hash functions have to open and read all the contents of your 1-5 MB CSV files.
I like sfFinder so much that I wrote my own adaption:
http://www.symfony-project.org/cookbook/1_0/en/finder
https://github.com/homer6/altumo/blob/master/source/php/Utils/Finder.php
Simple to use, works well.
However, for your use, depending on the size of the files, I'd put everything in a git repository. It's easy to track then.
HTH