I wrote a small script to generate email addresses based on a pattern:
<?php
$host = '#gmail.com';
$prefix = 'email';
$total = 200000;
for($i = 0; $i <$total; $i++)
{
echo $prefix . $i . $host . PHP_EOL;
}
Now, if I run this script as $php generate.php it takes around 15 seconds to complete the output echo. However, redirecting output ($php generate.php > file) completes within a second.
Is this something to do with PHP's buffering mechanism's or part of some Linux behavior?
Printing to the console is always much slower on any operating system than writing to a file directly, because there are a lot more drivers and operating system operations the output data must go through.
File operations are heavily optimized (caching!), so it's always much faster than writing to any visual output device.
This is Linux that is causing the slowdown. The same would happen if you printed the output in a windows console. Each flush of the output is obliging the program to halt, the operating system to update its window, and the program to be given control to continue again. If these flushes are numerous and often, it may indeed slow down things quite a bit.
Writing to a file still requires the operating system to step in, but there is no overhead beyond what you'd normally see with a program that reads and writes files. That said, writing to the console is a handy tool for debugging a program and you should use it. Just be sure that when it runs on the production machine, it writes to a file log instead.
Related
I'm working in PHP and creating a system with a lot of PHP-driven elements and I have noticed that some of my pages stop displaying text produced using the echo command.
I have made a small example of this. Of course, my program is not supposed to just print allt numbers from 1 to 10000, but this example demonstrates how the script just terminates without any warnings.
Example code:
<?php
for ($i = 1; $i <= 10000; $i++) {
echo $i, '<br>';
}
?>
Output:
1
2
More numbers...
8975
8976
8977
8
What is causing this? Is it a buffer issue, and how do I resolve it?
The fact that your code ran to completion on the cli suggests to me that your script is exceeding the ini.max_execution_time runtime configuration.
Note in the linked documentation that the value of this configuration on the cli is 0 which means it does not time out when run in that environment.
The default setting in the browser is 30 seconds.
You can show your current setting in the browser with:
echo ini_get('max_execution_time');
And you should be able to increase it with:
ini_set('max_execution_time', 0); // turns off timeout
If the script you have shown us behaves as you describe the there's something very wrong going on. If this is a Unix or Linux based system and its repeatedly exhibiting this behaviour then the kernel is terminating the script - unless it has been configured not to do so, the kernel will be forcing a core dump of the process.
Either go build a new system to run your code on or Google how to capture and diagnose a core dump on your operating system.
update
If xdebug is reporting the process is still running then it probably hasn't dumped its core, but "not producing output" != "not running". What state is the process in? What happens when you redirect the ouput? What is the end-to-end output channel when it misbehaves?
The problem did not lie directly with my PHP installation or the application itself, but somewhere in my IDE PHPStorm. When running the code with the same PHP interpreter outside of the IDE's wrappers, it all works fine. The procedures described by the many users here helped with that. Thank you.
I'm trying to develop an online management system for a very large FLAC music library for a radio station. It's got a beefy server and not many users, so I want to be able to offer a file download service where PHP transcodes the FLAC files into MP3/WAV depending on what the endpoint wants.
This works fine:
if($filetype == "wav") {
header("Content-Length: ". $bitrate * $audio->get_length());
$command = "flac -c -d ".$audio->get_filename().".flac";
}
ob_end_flush();
$handle = popen($command, "r");
while($read = fread($handle, 8192)) echo $read;
pclose($handle);
and allows the server to start sending the file to the user before the transcoding (well, decoding in this case) completes, for maximum speed.
However, the problem I'm getting is that while this script is executing, I can't get Apache to handle any other requests on the entire domain. It'll still work fine on other VirtualHosts on the same machine, but nobody can load any pages on this website while one person happens to be downloading a file.
I've also tried implementing the same thing using proc_open with no difference, and have played with the Apache settings for number of workers and the like.
Is the only way to stop this behaviour to use something like exec and waiting for the encoding process to finish before I start sending the user the file? Because that seems sub-optimal! :(
UPDATE: it seems that other people can still access the website, but not me - i.e. it's somehow related to sessions. This confuses me even more!
Use session_write_close() at some point before you start streaming... You may also want to stream_set_blocking(false) on the read pipe.
I have an object for tasks and on __deconstruct(), it's meant to run some of the lengthier cleanup tasks after the rest of the page has already loaded. Unfortunately, it buffers the output and won't send it until after the tasks are finished (nothing is printed in the tasks).
I've read through http://www.php.net/flush and tried all the suggestions there. Obviously I've tried disabling output_buffering in php.ini. I've disabled deflate_module, zlib compression is off, don't have mod_gzip. Calling flush() or ob_flush() has no effect, nor does enabling implicit_flush.
I'm just running XAMPP (currently apache 2.2.17, php 5.3.4) under Windows Server 2008 R2. PHP is being run as a module.
And yes, I could set up some little AJAX hack to run the task manager or even set up a scheduled task to run this specific task, but output buffering has been an issue elsewhere, too. Would just like it to be gone sometimes.
From a similar thread, someone suggested seeing what the following would do:
<?php
while (TRUE)
{
echo 'x';
flush();
sleep(1);
}
?>
As expected, the page displays nothing until the maximum execution time is reached, at which point it flushes the buffer.
This has become extremely frustrating. Anyone have any ideas what could still be causing it to buffer?
You're only sending a small amount of data. Browsers have their own buffer, which can be based on a number of bytes, by which elements have been received, or by something else.
In short, there is nothing you can do about this. The buffering is happening client-side, not server-side. You could try sending more data before your xs.
You can prove this by packet sniffing the connection between the server and the browser, with Wireshark or similar.
hmmm, interesting grabbed a snip of code I have used else where and it works as expected...
https://stackoverflow.com/a/9728519/632951
<?php
echo str_repeat('fillerbytes',20*1024/strlen('fillerbytes'));
echo '<body style="font-size:6px;font-family:arial;">';
echo str_repeat('<br>',2);
for($i=1; $i<=5000; $i++){
echo $i . ' ';
ob_flush();
flush();
usleep(2000); // 2 ms each = 10s total
}
?>
Watch my server count to 5000 http://atwebresults.com/texttest/new.php
(Doesn't work on some free hosts like freehostingeu.com.)
PHP Echo or Print functions does not show anything when php is busy with something (like when surfing the web with curl or something like that).
Later i discovered php does show the output when you execute your php on the command line:
php myscript.php
But right now i don't get any outputs from command line too!
Is there any kind of tricks or setting should be done to make php show the outputs?
Chances are, it's caching the results (both in PHP and the web server) and not actually sending them to the browser yet. the best suggestion I can give is this chunk from my code:
/**
* Act as a "breakpoint" in the code, forcing everything to be flushed to the browser
*/
function breakpoint() {
global $debug;
if ($debug) { // So that it doesn't slow down extracts
ob_flush();
flush();
sleep(.1);
}
}
The $debug stuff is if we're running the page in debug mode, specific to our website.
The two main thing you need are ob_flush(), which will send PHP's buffer to the web server's buffer, and flush() which will cause the server to dump to the browser. (Note: if the browser caches before displaying anything, nothing can prevent this) The sleep is there to help make sure it doesn't get overloaded, and has a chance to flush properly.
See:
http://ca.php.net/manual/en/function.ob-flush.php
and
http://ca.php.net/manual/en/function.flush.php
Both PHP and your web server are likely to be caching the output of echo and print. This will often result in no output until the script completes.
Try looking at flush() to force the output out of PHP, but it still may get held up at the web server, so may not help...
I'm trying to make a PHP script, I have the script finished but it takes like 10 minutes to finish the process it is designed to do. This is not a problem, however I presume I have to keep the page loaded all this time which is annoying. Can I have it so that I start the process and then come back 10mins later and just view the log file it has generated?
Well, you can use "ignore_user_abort(true)"
So the script will continue to work (keep an eye on script duration, perhaps add "set_time_limit(0)")
But a warning here: You will not be able to stop a script with these two lines:
ignore_user_abort(true);
set_time_limit(0);
Except you can directly access the server and kill the process there! (Been there, done an endless loop, calling itself over and over again, made the server come to a screeching stop, got shouted at...)
Sounds like you should have a queue and an external script for processing the queue.
For example, your PHP script should put an entry into a database table and return right away. Then, a cron running every minute checks the queue and forks a process for each job.
The advantage here is that you don't lock an apache thread up for 10 minutes.
I had lots of issues with this sort of process under windows; My situation was a little different in that I didn't care about the response of the "script"- I wanted the script to start and allow other page requests to go through while it was busy working away.
For some reason; I had issues with it either hanging other requests or timing out after about 60 seconds (both apache and php were set to time out after about 20 minutes); It also turns out that firefox times out after 5 minutes (by default) anyway so after that point you can't know what's going on through the browser without changing settings in firefox.
I ended up using the process open and process close methods to open up a php in cli mode like so:
pclose(popen("start php myscript.php", "r"));
This would ( using start ) open the php process and then kill the start process leaving php running for however long it needed - again you'd need to kill the process to manually shut it down. It didn't need you to set any time outs and you could let the current page that called it continue and output some more details.
The only issue with this is that if you need to send the script any data, you'd either do it via another source or pass it along the "command line" as parameters; which isn't so secure.
Worked nicely for what we needed though and ensures the script always starts and is allowed to run without any interruptions.
I think shell_exec command is what you are looking for.
However, it is disables in safe mode.
The PHP manual article about it is here: http://php.net/shell_exec
There is an article about it here: http://nsaunders.wordpress.com/2007/01/12/running-a-background-process-in-php/
There is another option which you can use, run the script CLI...It will run in the background and you can even run it as a cronjob if you want.
e.g
> #!/usr/bin/php -q
<?php
//process logs
?>
This can be setup as a cronjob and will execute with no time limitation....this examples is for unix based operation system though.
FYI
I have a php script running with an infinite loop which does some processing and has been running for the past 3 months non stop.
You could use ignore_user_abort() - that way the script will continue to run even if you close your browser or go to a different page.
Think about Gearman
Gearman is a generic application framework for farming out work to
multiple machines or processes. It allows applications to complete
tasks in parallel, to load balance processing, and to call functions
between languages. The framework can be used in a variety of
applications, from high-availability web sites to the transport of
database replication events.
This extension provides classes for writing Gearman clients and
workers.
- Source php manual
Offical website of Gearman
In addition to bastiandoeen's answer you can combine ignore_user_abort(true); with a cUrl request.
Fake a request abortion setting a low CURLOPT_TIMEOUT_MS and keep processing after the connection closed:
function async_curl($background_process=''){
//-------------get curl contents----------------
$ch = curl_init($background_process);
curl_setopt_array($ch, array(
CURLOPT_HEADER => 0,
CURLOPT_RETURNTRANSFER =>true,
CURLOPT_NOSIGNAL => 1, //to timeout immediately if the value is < 1000 ms
CURLOPT_TIMEOUT_MS => 50, //The maximum number of mseconds to allow cURL functions to execute
CURLOPT_VERBOSE => 1,
CURLOPT_HEADER => 1
));
$out = curl_exec($ch);
//-------------parse curl contents----------------
//$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
//$header = substr($out, 0, $header_size);
//$body = substr($out, $header_size);
curl_close($ch);
return true;
}
async_curl('http://example.com/background_process_1.php');
NB
If you want cURL to timeout in less than one second, you can use
CURLOPT_TIMEOUT_MS, although there is a bug/"feature" on "Unix-like
systems" that causes libcurl to timeout immediately if the value is <
1000 ms with the error "cURL Error (28): Timeout was reached". The
explanation for this behavior is:
[...]
The solution is to disable signals using CURLOPT_NOSIGNAL
pros
No need to switch methods (Compatible windows & linux)
No need to implement connection handling via headers and buffer (Independent from Browser and PHP version)
cons
Need curl extension
Resources
curl timeout less than 1000ms always fails?
http://www.php.net/manual/en/function.curl-setopt.php#104597
http://php.net/manual/en/features.connection-handling.php
Zuk.
I'm pretty sure this will work:
<?php
pclose(popen('php /path/to/file/server.php &'));
echo "Server started. [OK]";
?>
The '&' is important. It tells the shell not to wait for the process to exit.
Also You can use this code in your php (as "bastiandoeen" said)
ignore_user_abort(true);
set_time_limit(0);
in your server stop command:
<?php
$output;
exec('ps aux | grep -ie /path/to/file/server.php | awk \'{print $2}\' | xargs kill -9 ', $output);
echo "Server stopped. [OK]";
?>
Just call StartBuffer() before any output, and EndBuffer() when you want client to close connection. The code after calling EndBuffer() will be executed on server without client connection.
private function StartBuffer(){
#ini_set('zlib.output_compression',0);
#ini_set('implicit_flush',1);
#ob_end_clean();
#set_time_limit(0);
#ob_implicit_flush(1);
#ob_start();
}
private function EndBuffer(){
$size = ob_get_length();
header("Content-Length: $size");
header('Connection: close');
ob_flush();ob_implicit_flush(1);
}