I have a PHP script running via CLI that's working well, but it runs a couple long queries (2-5 minutes) that would ideally give you some idea that something is still happening. When iterating through results, I do have a function running that updates the progress, but when PHP is waiting for a query to return, silence ensues.
I don't need to know anything about when the query will complete, but some sort of indication on the CLI that it's doing something would be a huge gain (binking ..., or something). Possible?
I've found that using carriage returns \r without newlines to be extremely helpful. They reset the output to the beginning of the line, but do not move down a line, allowing you to overwrite the current text.
Please note that you'll need to pad the line to the full length, otherwise previous characters will still linger. For example:
$iteration = 0;
while (/* wait condition */) {
printf("Process still running%-5s\r", str_repeat('.', $iteration % 5));
sleep(1);
$iteration++;
}
echo "\n";
echo "Task completed!";
If you're using a for loop for processing, something like this would be much more useful:
// Display progress every X iterations
$update_interval = 1000000;
for ($i = 0; $i < $massive_number; $i++) {
// Do processing
if ($i % $update_interval == 0) {
printf("Progress: %.2f%%\r", (100 * $i / $massive_number));
}
}
Related
I've been trying to validate over 1 million randomly generated values (strings) with PHP and a client side programming language on an online form, but there are a few challenges I'm facing:
PHP
Link to the (editable) PHP code:https://3v4l.org/AtTkO
The PHP code:
<?php
function generateRandomString($length = 10) {
$characters = '0123456789abcdefghijklmnopqrstuvwxyz-_.';
$charactersLength = strlen($characters);
$randomString = '';
for ($i = 0; $i < $length; $i++) {
$randomString .= $characters[rand(0, $charactersLength - 1)];
}
return $randomString;
}
$unique = array();
for ($i = 0; $i < 9000000; $i++)
{
$u=$i+1;
$random = generateRandomString(5);
if(!in_array($random, $unique)){
echo $u.".m".$random."#[server]\n";
$unique[] = $random;
gc_collect_cycles();
}else{
echo "duplicate detected";
$i--;
}
}
echo memory_get_peak_usage();
What should happen:
New 5 character value gets randomly generated
Value gets checked if it already exists in the array
Value gets added to array
All randomly generated values are exported to a .txt file to be used for validating. (Not in the script yet)
What actually happens:
I hit either a memory usage limit or a server timeout for the execution time.
What I've tried
I've tried using sleep(3) during the for loop.
Setting Memory limit to -1 and timeout to 0. The unlimited memory doesn't make a difference and is too dangerous in a working environment.
Using gc_collect_cycles() during the for loop
Using echo memory_get_peak_usage(); -> I don't really understand
how I could use this for debugging.
What I need help with:
Memory management in PHP
Having pauses in the script that will reset the PHP execution timer
Client Side Programming language
This is where I have absolutely no clue which way I should go or which programming language I should use for this.
What I want to achieve
Load a webpage that has a form
Load the .txt with all randomly generated strings
fill in the form with the first string
submit the form:
If positive response from form > save string in special .txt file or array, go to the next value
If negative response from form > delete string from file, go to the next value | or just go to the next value
All values with a positive response are filtered out and easily accessible at the end.
I don't know which programming language I should use for this function. I've been thinking about Javascript and Python but I'm not sure how I could combine that with PHP. A nudge in the right direction would be appreciated.
I might be completely wrong for trying to achieve this with PHP, if so, please let me know what would be the better and easier option.
Thanks!
Interesting question, first of all whenever you think of a solution like this, one of the first things you need to consider is can it be async? If your answer is yes, then your implementation will likely be simple, else, you will likely have to pay huge server costs or render random cached results.
NB remove gc_collect_cycles. It does the opposite of what you want, and you hardly ever need to call it manually.
That being said, the approach I would recommend in your case is as follows:
Use a websocket which will be opened only once on the client browser, and then forward results in realtime from server to the browser. Of course, this code itself, can run completely on clientside via javascript, so if it's not just a PoC, you can convert the php code to javascript.
Change your code to yield items or forward results via websocket once a generated code has been confirmed as unique.
However, if you're really just doing only what the PHP code says, you can do that completely in javascript and save your server resources. See this answer for an example code to replace your generateRandomString function.
Assuming you have the ability to edit the php.ini:
Increase your memory limit as described here:
PHP MEMORY LIMIT INCREASE
For the 'memory limit' see here
and for the 'timeout for the execution time' add :
set_time_limit(0);
on the top of the PHP file.
Have you tried using sets? https://www.php.net/manual/en/class.ds-set.php
Sets are very efficient whenever you want to ensure a value isn't present twice.
Checking the presence of a value in a set it way way way faster that loop across all entries on the array.
I'm not a expert with PHP but it would look like something like that in Ruby
require 'set'
CHARS = '0123456789abcdefghijklmnopqrstuvwxyz-_.'.split('');
unique = Set.new()
def generateRandomString(l = 10)
Array.new(l) { CHARS.sample }.join
end
while unique.length < 1_000_000
random_string = generateRandomString
if !unique.include?(random_string)
unique.add(random_string)
end
end
hope it helps
The following loop is taking 13 seconds to run on a Windows i7 # 3.4Ghz 16GB.
The script is running from the command line - php loop.php
$start = microtime(true);
for($i = 0; $i <= 150000; $i++) {
$running_time = date('i:s', microtime(true) - $start);
echo "$i - $running_time\n";
}
If I take out the 'echo', it takes less than a second, why?
This has to do with lack of buffering of your output. If you run this in a Windows console, you'll find that the console is your bottleneck.
Hold the scroll bar and watch your program hang until you release it again, to prove this.
I ran into an interesting (well at least in my opinion) problem.
I have a PHP script that should generate the formatting (eg. the absolute positioning values of each image so they get displayed next to each other in a logical pattern) and the image sources when run. When completed it would load the appropriate image path from an sql db but currently I have a problem with this at this point.
Currently my script looks something like:
for ($i=0; $i<(866+1+866); $i++){
for ($j=0; $j<1001; $j++){
$data .= "<div id=\"tac-".$j."\"><img src=\"default_tactical.png\"/></div>";
}
}
As you can see it's rather basic at this point, as I only wanted to test if I can get the images in place.
Also the $data is a variable that my template simply echo-es to the browser.
The problem with all this is that my server runs out-of memory whenever I try to run this script.
So what's the problem? Or rather: how can I have a lot of images in a webpage without running out of memory?
Try changing it to:
for ($i=0; $i<(866+1+866); $i++){
for ($j=0; $j<1001; $j++){
echo "<div id=\"tac-".$j."\"><img src=\"default_tactical.png\"/></div>";
}
}
It should not run out of memory since it's not storing anything, just directly outputs it.
EDIT: Since you can't modify the code, just try raising the memory limit somewhere in the code (can be any PHP code that is executed before your loop).
#ini_set("memory_limit", "512M");
Look at it this way, you've got 2 nested loop, and are building a string inside.
866+1+66 = 1733 x 1002 = 17,364,66 iterations
17,364,666 iterations * 40 chars = ~70 megabytes
Either DON'T build the string at all once, or at least split it into chunks, e.g.
for ($i = ....) {
for ($j = ....) {
... build string here
}
echo $string
$string = ''; // reset to empty string and start over
}
While you haven't echo your $data, you haven't load image, it's just a string. It's the navigator which going to load each image after PHP processing. Your PHP is executed in the server and client load images. It's your variable $data which is out of memory.
Try like this :
for ($i=0; $i<(866+1+866); $i++){
for ($j=0; $j<1001; $j++){
echo "<div id=\"tac-".$j."\"><img src=\"default_tactical.png\"/></div>";
}
}
I have a php script that is running in CLI and I want to display the current percent progress so I was wondering if it is possible to update the STDOUT to display the new percent.
When I use rewind() or fseek() it just throws an error message.
See this code:
<?php
echo "1";
echo chr(8);
echo "2";
The output is only 2 since "chr(8)" is the char for "backspace".
So just print the amount of chars you need to go back and print the new percentage.
Printing "\r" works too on Linux and Windows but isn't going to cut it on a mac
Working example:
echo "Done: ";
$string = "";
for($i = 0; $i < 100; ++$i) {
echo str_repeat(chr(8), strlen($string));
$string = $i."%";
echo $string;
sleep(1);
}
Output \r and then flush to get back to the first column of the current line.
Writing to a console/terminal is surprisingly complex if you want to move backwards in the output raster or do things like add colours - and the behaviour will vary depending on the type of console/terminal you are using. A long time ago some people came up with the idea of building an abstract representation of a terminal and writing to that.
See this article for details of how to do that in PHP.
I have a dedicated server running Cent OS with a Parallel PLESK panel. I need to run a PHP script every second to update my database. These is no alternative way time-wise, it needs to be updated every second.
I can find my script using the URL http://www.somesite.com/phpfile.php?key=123.
Can the file be executed locally every second? Like phpfile.php?
Update:
It has been a few months since I added this question. I ended up using the following code:
#!/user/bin/php
<?php
$start = microtime(true);
set_time_limit(60);
for ($i = 0; $i < 59; ++$i) {
doMyThings();
time_sleep_until($start + $i + 1);
}
?>
My cronjob is set to every minute. I have been running this for some time now in a test environment, and it has worked great. It is really super fast, and I see no increase in CPU nor Memory usage.
You could actually do it in PHP. Write one program which will run for 59 seconds, doing your checks every second, and then terminates. Combine this with a cron job which runs that process every minute and hey presto.
One approach is this:
set_time_limit(60);
for ($i = 0; $i < 59; ++$i) {
doMyThings();
sleep(1);
}
The only thing you'd probably have to watch out for is the running time of your doMyThings() functions. Even if that's a fraction of a second, then over 60 iterations, that could add up to cause some problems. If you're running PHP >= 5.1 (or >= 5.3 on Windows) then you could use time_sleep_until()
$start = microtime(true);
set_time_limit(60);
for ($i = 0; $i < 59; ++$i) {
doMyThings();
time_sleep_until($start + $i + 1);
}
Have you thought about using "watch"?
watch -n 1 /path/to/phpfile.php
Just start it once and it will keep going. This way it is immune to PHP crashing (not that it happens, but you never know). You can even add this inittab to make it completely bullet-proof.
Why not run a cron to do this and in the php file loop 60 times which a short sleep. That is the way I have overcome this to run a php script 5 times a minute.
To set up your file to be run as a script add the path to the your PHP on the first line such as a perl script
#!/user/bin/php
<?php
while($i < 60) {
sleep(1);
//do stuff
$i++;
}
?>
This is simple upgraded version of nickf second solution witch allow to specify the desired interval in seconds beetween each executions in execution time.
$duration = 60; // Duration of the loop in seconds
$sleep = 5; // Sleep beetween each execution (with stuff execution)
for ($i = 0; $i < floor($duration / $sleep); ++$i) {
$start = microtime(true);
// Do you stuff here
time_sleep_until($start + $sleep);
}
I noticed that the OP edited the answer to give his solution. This solution did not work on my box (the path to PHP is incorrect and the PHP syntax is not correct)
This version worked (save as whatever.sh and chmod +X whatever.sh so it can execute)
#!/usr/bin/php
<?php
$start = microtime(true);
set_time_limit(60);
for ($i = 0; $i < 59; ++$i) {
echo $i;
time_sleep_until($start + $i + 1);
}
?>
You can run your infinite loop script with nohup command on your server which can work even you logout from system. Only restart or physically shutdown can destroy this process. Don't forget to add sleep (1) in your php script.
nohup php /path/to/you/script.php
Now in case you don't need to use the console while it's working, it'll write its output to nohup.out file in your working directory (use the pwd command to get it).