Nowdays there are a lot of websites for files hosting (uploading websites) and it count for example point per complete download of certain file.
My question
I want to understand what is the idea they are using !
How does it only count on complete downloading of the file ?!
i mean if i canceled downloading of the file after it started , it won't count point!
how does it knew ! is there any php function that able to know if i canceled downloading certain exact file or not !
that question was all time in my mind and thinking about it but i can't understand how does it works or what is the idea behind it. ~ thanks
This can be done by using my other answer as base How can I give download access to files outside public_html directory? and replacing readfile( $filename )
with readfileWhileConnected( $filename ):
Read file until EOF or disconnect:
/** Read $filename until EOF or disconnect,
* if disconnect then error_log() count of bytes read already
*/
function readfileWhileConnected( $filename ) {
// Save and set ini values:
$user_abort = ignore_user_abort();
ignore_user_abort(false);
// Get file size and set bytes_sent to zero:
$fsize = filesize($filename);
$bytes_sent = 0;
// Open file:
$f = fopen($filename, 'r');
// Read file:
while($chunk = fread($f, 1024)) {
// Check if connection is still open:
if(!connection_aborted()) {
// Send $chunk to buffer (if any), then flush() buffers:
echo $chunk;
flush();
// Add $chunk length to $bytes_sent
$bytes_sent += strlen($chunk);
} else {
// Close file:
fclose($f);
error_log("Connection closed at $bytes_sent/$fsize");
exit();
}
// Close file:
fclose($f);
// Reset ini values:
ignore_user_abort($user_abort);
return $bytes_sent;
}
}
After you have your new shiny class myNewSuperDownloadHandlerClass { ... } ready, then make sure you only serve downloads through filedownload.php described here or if have done good myNewSuperDownloadHandlerClass(), then use that, just make sure that readfileWhileConnected() is used for every download requiring connection status polling.
You can easily add callback to be triggered if user closes connection, only 2 exit points here. (seen many functions that have every often return false; return true; return null; return false; return true; and so on..)
Related
I'm facing to a problem and I'm not really sure if this is the right way of doing this. I need to copy a file from a remote server to my server with php.
I use the following script :
public function download($file_source, $file_target) {
$rh = fopen($file_source, 'rb');
$wh = fopen($file_target, 'w+b');
if (!$rh || !$wh) {
return false;
}
while (!feof($rh)) {
if (fwrite($wh, fread($rh, 4096)) === FALSE) {
return false;
}
echo ' ';
flush();
}
fclose($rh);
fclose($wh);
return true;
}
but in the end, the file size remains at 0.
EDIT : I update my question, because there are still some things I didn't understand :
About fread, I used 2048mb. But it didn't work.
I found the script above, which uses 4096mb.
My question : How to determine which quantity of memory (?) to use in order no get the file downloaded anytime ? Because this one works on a specific machine (dedicated), but will it on a shared host, if I cannot modify the php.ini ?
Thanks again
filesize() expects a filename/path. You're passing in a filehandle, which means filesize will FAIL and return a boolean false.
You then use that false as the size argument for your fread, which gets translated to an integer 0. So essentially you're sitting there telling php to read a file, 0 bytes at a time.
You cannot reliably get the size of a remote file anyways, so just have fread some fixed number of bytes, e.g. 2048, at a time.
while(!feof($handle)) {
$contents = fread($handle, 2048);
fwrite($f, $contents);
}
and if that file isn't too big and/or your PHP can handle it:
file_put_contents('local.mp4', file_get_contents('http://whatever/foo.mp4'));
I want to have a temp file that gets updated from time to time.
What I was thinking of doing is:
<!-- language: lang-php -->
// get the contents
$s = file_get_contents( ... );
// does it need updating?
if( needs_update() )
{
$s = 'some new content';
file_put_contents( ... );
}
The issue that I could see happening is that whatever condition causes 'needs_update()' to return true could cause more than one process to update the same file at, (almost), the same time.
In an ideal situation, I would have one single process updating the file and prevent all other processes from reading the file until I am done with it.
So as soon as 'needs_update()' return true is called I would prevent others processes from reading the file.
<!-- language: lang-php -->
// wait here if anybody is busy writing to the file.
wait_if_another_process_is_busy_with_the_file();
// get the contents
$s = file_get_contents( ... );
// does it need updating?
if( needs_update() )
{
// prevent read/write access to the file for a moment
prevent_read_write_to_file_and_wait();
// rebuild the new content
$s = 'some new content';
file_put_contents( ... );
}
That way, only one process could possibly update the file and the files would all be getting the latest values.
Any suggestions on how I could prevent such a conflict?
Thanks
FFMG
You are looking for the flock function. flock will work as long as everyone that acess the file is using it. Example from php manual:
$fp = fopen("/tmp/lock.txt", "r+");
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
ftruncate($fp, 0); // truncate file
fwrite($fp, "Write something here\n");
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
} else {
echo "Couldn't get the lock!";
}
fclose($fp);
Manual: http://php.net/manual/en/function.flock.php
I'm trying to write a PHP script that I want to ensure only has a single instance of it running at any given time. All of this talk about different ways of locking, and race conditions, and etc. etc. etc. is giving me the willies.
I'm confused as to whether lock files are the way to go, or semaphores, or using MySQL locks, or etc. etc. etc.
Can anyone tell me:
a) What is the correct way to implement this?
AND
b) Point me to a PHP implementation (or something easy to port to PHP?)
One way is to use the php function flock with a dummy file, that will act as a watchdog.
On the beginning of our job, if the file raise a LOCK_EX flag, exit, or wait, can be done.
Php flock documentation: http://php.net/manual/en/function.flock.php
For this examples, a file called lock.txt must be created first.
Example 1, if another twin process is running, it will properly quit, without retrying, giving a state message.
It will throw the error state, if the file lock.txt isn't reachable.
<?php
$fp = fopen("lock.txt", "r+");
if (!flock($fp, LOCK_EX|LOCK_NB, $blocked)) {
if ($blocked) {
// another process holds the lock
echo "Couldn't get the lock! Other script in run!\n";
}
else {
// couldn't lock for another reason, e.g. no such file
echo "Error! Nothing done.";
}
}
else {
// lock obtained
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp); // Empty memory
Example 2, FIFO (First in, first out): we wants the process to wait, for an execution after the queue, if any:
<?php
$fp = fopen("lock.txt", "r+");
if (flock($fp, LOCK_EX)) { // acquire an exclusive lock
ftruncate($fp, 0); // truncate file
// Your job here
echo "Job running!\n";
// Leave a breathe
sleep(3);
fflush($fp); // flush output before releasing the lock
flock($fp, LOCK_UN); // release the lock
}
fclose($fp);
It is also doable with fopen into x mode, by creating and erasing a file when the script ends.
Create and open for writing only; place the file pointer at the
beginning of the file. If the file already exists, the fopen() call
will fail by returning FALSE
http://php.net/manual/en/function.fopen.php
However, into a Unix environment, for fine tuning, I found easier to list the PID's of every background scripts with getmypid() into a DB, or a separate JSON file.
When one task ends, the script is responsible to declare his state in this file (eq: success/failure/debug infos, etc), and then remove his PID. This allows from my view to create admins tools and daemons in a simpler way. And use posix_kill() to kill a PID from PHP if necessary.
Micro-Services are composed using Unix-like pipelines.
Services can call services.
https://en.wikipedia.org/wiki/Microservices
See also: Prevent PHP script using up all resources while it runs?
// borrow from 2 anwsers on stackoverflow
function IsProcessRunning($pid) {
return shell_exec("ps aux | grep " . $pid . " | wc -l") > 2;
}
function AmIRunning($process_file) {
// Check I am running from the command line
if (PHP_SAPI != 'cli') {
error('Run me from the command line');
exit;
}
// Check if I'm already running and kill myself off if I am
$pid_running = false;
$pid = 0;
if (file_exists($process_file)) {
$data = file($process_file);
foreach ($data as $pid) {
$pid = (int)$pid;
if ($pid > 0 && IsProcessRunning($pid)) {
$pid_running = $pid;
break;
}
}
}
if ($pid_running && $pid_running != getmypid()) {
if (file_exists($process_file)) {
file_put_contents($process_file, $pid);
}
info('I am already running as pid ' . $pid . ' so stopping now');
return true;
} else {
// Make sure file has just me in it
file_put_contents($process_file, getmypid());
info('Written pid with id '.getmypid());
return false;
}
}
/*
* Make sure there is only one instance running at a time
*/
$lockdir = '/data/lock';
$script_name = basename(__FILE__, '.php');
// The file to store our process file
$process_file = $lockdir . DS . $script_name . '.pid';
$am_i_running = AmIRunning($process_file);
if ($am_i_running) {
exit;
}
Use semaphores:
$key = 156478953; //this should be unique for each script
$maxAcquire = 1;
$permissions =0666;
$autoRelease = 1; //releases semaphore when request is shut down (you dont have to worry about die(), exit() or return
$non_blocking = false; //if true, fails instantly if semaphore is not free
$semaphore = sem_get($key, $maxAcquire, $permissions, $autoRelease);
if (sem_acquire($semaphore, $non_blocking )) //blocking (prevent simultaneous multiple executions)
{
processLongCalculation();
}
sem_release($semaphore);
See:
https://www.php.net/manual/en/function.sem-get.php
https://www.php.net/manual/en/function.sem-acquire.php
https://www.php.net/manual/en/function.sem-release.php
You can go for the solution that fits best your project, the two simple ways to achieve that are file locking or database locking.
For implementations of file locking, check http://us2.php.net/flock
If you already use a database, create a table, generate known token for that script, put it there, and just remove it after the end of the script. To avoid problems on errors, you can use expiry times.
Perhaps this could work for you,
http://www.electrictoolbox.com/check-php-script-already-running/
In case you are using php on linux and I think the most practical way is:
<?php
if(shell_exec('ps aux | grep '.__FILE__.' | wc -l')>3){
exit('already running...');
}
?>
Another way to do it is with file flag and exit callback,
the exit callback will ensures that the file flag will be reset to 0 in any case of php execution end also fatal errors.
<?php
function exitProcess(){
if(file_get_contents('inprocess.txt')!='0'){
file_put_contents('inprocess.txt','0');
}
}
if(file_get_contents('inprocess.txt')=='1'){
exit();
}
file_put_contents('inprocess.txt','1');
register_shutdown_function('exitProcess');
/**
execute stuff
**/
?>
I have a script i use that checks an IP address stored within my hosts.allow file against what IP is mapped to my dyndns hostname so i can log into my servers once i've synced my current IP to that hostname. For some reason though the script seems to cause really intermittent issues.
within my hosts.allow file i have a section like this:
#SOme.gotdns.com
sshd : 192.168.0.1
#EOme.gotdns.com
#SOme2.gotdns.com
sshd : 192.168.0.2
#EOme2.gotdns.com
I have a script running on a cron (every minute) that looks like this:
#!/usr/bin/php
<?php
$hosts = array('me.gotdns.com','me2.gotdns.com');
foreach($hosts as $host)
{
$ip = gethostbyname($host);
$replaceWith = "#SO".$host."\nsshd : ".$ip."\n#EO".$host;
$filename = '/etc/hosts.allow';
$handle = fopen($filename,'r');
$contents = fread($handle, filesize($filename));
fclose($handle);
if (preg_match('/#SO'.$host.'(.*?)#EO'.$host.'/si', $contents, $regs))
{
$result = $regs[0];
}
if($result != $replaceWith)
{
$newcontents = str_replace($result,$replaceWith,$contents);
$handle = fopen($filename,'w');
if (fwrite($handle, $newcontents) === FALSE) {
}
fclose($handle);
}
}
?>
The problem i have is that intermittently characters are being dropped (i assume during the replace) that causes future updates to fail as it inserts something like:
#SOme.gotdns.com
sshd : 192.168.0.1
#EOme.gotdn
note the missing "s.com"
This of course means i lose access to the server, any ideas why this would be happening?
Thanks.
that might be because of script execution time - can be too short- OR 1 min interval is too short. While cron is doing the job, another process of script starts and it may effect the first one.
This is almost certainly because the script hasn't finished executing within the one minute time period before it's started again via cron. You need to implement some sort of locking, or use a tool that only allows once instance of the script to be run. There are several tools available out there that can do this, for example lockrun.
I would say that in order to do this safely, you should acquire an exclusive lock on the file at the beginning of the script, read it all into memory once, modify it in memory, then write it back to the file at the end. This would also be considerably more efficient in terms of disk I/O.
You should also alter the cron job to run less frequently. It is likely that the reason you currently have this problem is because two processes are running at the same time - by locking the file, if this is the case, you risk having the processes stack up waiting to acquire a lock. Setting it for every 5 minutes should be good enough - your IP shouldn't change that often!
So do this (FIXED):
#!/usr/bin/php
<?php
// Settings
$hosts = array(
'me.gotdns.com',
'me2.gotdns.com'
);
$filename = '/etc/hosts.allow';
// No time limit (shouldn't be necessary with CLI, but just in case)
set_time_limit(0);
// Open the file in read/write mode and lock it
// flock() should block until it gets a lock
if ((!$handle = fopen($filename, 'r+')) || !flock($handle, LOCK_EX)) exit(1);
// Read the file
if (($contents = fread($handle, filesize($filename)) === FALSE) exit(1);
// Will be set to true if we actually make any changes to the file
$changed = FALSE;
// Loop hosts list
foreach ($hosts as $host) {
// Get current IP address of host
if (($ip = gethostbyname($host)) == $host) continue;
// Find the entry in the file
$replaceWith = "#SO{$host}\nsshd : {$ip}\n#EO{$host}";
if (preg_match("/#SO{$host}(.*?)#EO{$host}/si", $contents, $regs)) {
// Only do this if there was a match - otherise risk overwriting previous
// entries because you didn't reset the value of $result
if ($regs[0] != $replaceWith) {
$changed = TRUE;
$contents = str_replace($regs[0], $replaceWith, $contents);
}
}
}
// We'll only change the contents of the file if the data changed
if ($changed) {
ftruncate($handle, 0); // Zero the length of the file
rewind($handle); // start writing from the beginning
fwrite($handle, $contents); // write the new data
}
flock($handle, LOCK_UN); // Unlock
fclose($handle); // close
I'm using a PHP script to stream a live video (i.e. a file which never ends) from a remote source. The output is viewed in VLC, not a web browser. I need to keep a count of the number of bytes transferred. Here is my code:
<?php
ignore_user_abort(true);
$stream = $_GET['stream'];
if($stream == "vid1")
{
$count = readfile('http://127.0.0.1:8080/');
logThis($count);
}
function logThis($c)
{
$myFile = "bytecount.txt";
$handle = fopen($myFile,'a');
fwrite($handle,"Count: " . $c . "\n");
fclose($handle);
}
?>
However it appears that when the user presses the stop button, logThis() is never called, even though I've put in ignore_user_abort(true);
Any ideas on what I'm doing wrong?
Thanks
Update2: I've changed my code as I shoudn't be using ignore_user_abort(true) as that would continue to download the file forever even after the client has gone. I've changed my code to this:
<?php
$count = 0;
function bye()
{
//Create Dummy File with the filename of equal to count
}
register_shutdown_function('bye');
set_time_limit(0);
ignore_user_abort(false);
$stream = $_GET['stream'];
if($stream == "vid1")
{
$GLOBALS['count'] = readfile('http://127.0.0.1:8080/');
exit();
}
?>
My problem now is that when the script is aborted (i.e. user presses stop), readfile won't return a value (i.e. count remains at 0). Any ideas on how I can fix this?
Thanks
When a PHP script is running normally the NORMAL state, is active. If the remote client disconnects the ABORTED state flag is turned on. A remote client disconnect is usually caused by the user hitting his STOP button. If the PHP-imposed time limit (see set_time_limit()) is hit, the TIMEOUT state flag is turned on.
so setting the set_time_limit to 0 should help.
Ok folks I managed to fix this. The trick was to not use readfile() but read the video stream byte by byte. Ok it may not be 100% accurate, however a few bytes inaccuracy here or there is ok.
<?php
$count = 0;
function logCount()
{
//Write out dummy file with a filename equal to count
}
register_shutdown_function('logCount');
set_time_limit(0);
ignore_user_abort(false);
$stream = $_GET['stream'];
if($stream == "vid1")
{
$filename = 'http://127.0.0.1:8080/';
$f = fopen($filename, "rb");
while($chunk = fread($f, 1024)) {
echo $chunk;
flush();
if(!connection_aborted()) {
$GLOBALS['count'] += strlen($chunk);
}
else {
exit();
}
}
}
?>