I have created a php script to import rss feed into the database. The feed which is huge (from year 2004 to 2010, approx 2 million records) has to be inserted into the database. I have been running the script in browser but the pace it is inserting (approx. 1 per second) i doubt it takes another 20-25 days to input this data even if i run it 24 hrs a day. I have tried it running on different browser windows at the same time and have finished only 70000 records in last two days. I am not sure how the server would react if i run 10-12 instances of it simultaneously.
A programmer at my client's end says that i could run it directly on the server through command line. Could anyone tell me how much difference it would make if i run it through command line? Also what is the command line syntax to run it? I am on apache, php/mysql. I tried out over the web for a similar answer but they seem quite confusing to me as i am not a system administrator or that good in linux although i have done tasks like svn repositories and installing some apache modules on server in the past so i hope i could manage this if someone tell me how to do it.
Difference in speed: Minimal. All you save on is blocking on NET I/O and connection (and the apache overhead which is negligible).
How to do it:
bash> php -f /path/to/my/php/script.php
You may only have the php5-mod package installed which is php for apache, you may have to install the actual command line interpreter, however a lot of distros install both. Personally I think you have an efficiency problem in the algorithm. Something taking days and days seems like it could be sped up by caching & worst-case performance analysis (Big-O notation).
Also, php vanilla isn't very fast, there's lots of ways to make it really fast, but if you're doing heavy computation, you should consider c/c++, C#/Mono (Maybe), possibly python (can be pre-compiled, may not actually be much faster).
But the exploration of these other outlets is highly recommended.
Only providing the filename to execute is sufficient:
php -f <YourScriptHere.php>
See the documentation for more command line options.
To run a php script in the command line just execute:
php yourscript.php
If you want to keep this process running in background do:
php yourscript.php &
You can then run several processes at the same time. To identify the instances of the script that are currently running execute:
ps aux | grep yourscript.php
However, if you think it takes too long, try to find out whether there's any bottleneck in your code and optimize it.
in linux:
php -f file.php
type
php --help
for other options
You may also need the -n option (no php.ini file) or options to specify where php-cli.ini or php.ini file can be found.
Related
I'm setting several cronjobs for PHP scripts so I want to know which option is better to CPU/RAM performance. I have a CentOS 7.3 server
I would definitely go with PHP CLI, less overhead.
I run a few crontasks myself and just sticking the following in your crontab (changed to use your files of course) makes it simple.
php -f /var/www/cron.php
You can run some tests to confirm it's better for you though - see which one runs quicker and uses less memory.
I have an XML database that I want to manage independently from users on my website. Looking into the matter it appears that I should write a daemon script to manage my database. That is all fine and dandy but, I feel like I'm opening a can of worms. I wanted to write my daemon script in PHP, so I looked into PCNTL. But I quickly learned that PCNTL is not suited for web servers. So now I am stumped. How can I get a daemon to run on my server? Do I need to learn another language? I only want to write my own scripts. But I feel lost. I would prefer to write my daemon in PHP as I am familiar with the language.
I have been researching everything from PCNTL, CLI, SO questions, numerous articles on daemon processes... etc
I am running PHP 5.6.32 (cli), windows 7, on Apache. XAMPP 5.6.32. Unix system.
EDIT: I also have windows setup to run PHP from command prompt.
There's nothing wrong in running a PHP daemon, however it's not the fastest thing, especially before the 7.0 version. You can proceed in two ways:
Using Cron Jobs, if you're under Unix systems crontab will be fine, in this way you can specify the interval within the system automatically executes the specified script and then exit.
The true daemon, firstly you need to change the max_execution_time in PHP.ini to 0 (infinite), then in your daemon call for first function set_time_limit(0);, remember to run it only once. However if there is some failure like a thrown error uncatched the script will exit and you need to open it again manually, and don't try...catch in a while loop because it will probably go into an endless loop. Execute the script with php -f daemon.php.
I have a laravel 4 web project that implements a laravel command.
When running in the development homestead vm, it runs to completion (about 40 seconds total time).
However when running it on the production server, it quits with a 'killed' output on the command line.
At first i thought it was the max_execution_time in cli php.ini, so I set it to 0 (for unlimited time).
How can I find out what is killing my command?
I run it on ssh terminal using the standard artisan invokation:
php artisan commandarea:commandname
Does laravel 4 have a command time limit somewhere?
The vps is a Ubuntu 4.10 machine with mysql, nginx and php-fpm
So, firstly, thank you everyone who has pointed me in the right direction regarding PHP and laravel memory usage tracking.
I have answered my own question hoping that it will benefit laravel devs in the future, as my solution was hard to find.
After typing 'dmesg' to show system messages. I found that the php script was being killed by Linux.
So, I added memory logging calls into my script before and after each of the key areas of my script:
Log::Info('Memory now at: ' . memory_get_peak_usage());
Then I ran the script while watching the log output and also the output of the 'top' command.
I found that even though my methods were ending and the variables were going out of scope, the memory was not being freed.
Things that I tried, that DIDNT make any difference in my case:
unset($varname) on variables after I have finished with them - hoping to get GC to kick off
adding gc_enable() at beginning of script and then adding gc_collect_cycle() calls after a significant number of vars are unset.
Disabling mysql transactions - thinking maybe that is memory intensive - it wasnt.
Now, the odd thing was that none of the above made any difference. My script was still using 150mb or ram by time it killed!
The solution that actually worked:
Now this is definitely a laravel specific solution.
But my scripts purpose is basically to parse a large xml feed and then insert thousands of rows into mysql using Elequent ORM.
It turns out that Laravel creates logging information and objects to help you see the query performance.
By turning this off with the following 'magic' call, I got my script down from 150mb to around 20mb!
This is the 'magic;' call:
DB::connection()->disableQueryLog();
I can tell you by the time I found this call, I was grasping at straws ;-(
A process may be killed for several reasons:
Out of Memory
There are two ways to trigger this error: Exceed the amount of memory allocated to PHP script in php.ini, or exceed the available system memory. Check the PHP error log and php.ini file to rule out the first possibility, and use dmesg output to check for the second possibility.
Exceeded the execution time-out limit
In your post you indicate that you disabled the timeout via the max_execution_time setting, but I have included it here for completeness. Be sure that the setting in php.ini is correct and (for those using a web server instead of a CLI script) restart the web server to ensure that the new configuration is active.
An error in the stack
If your script is error-free and not encountering either of the above errors, ensure that your system is running as expected. When using a web server, restart the web server software. Check the error logs for unexpected output, and stop or upgrade related daemons and needed.
Had this issue on a Laravel/Spark project. just wanted to share if others have this issue.
Try a refresh/restart of your dev server if running Vagrant or Ubuntu before more aggressive approaches.
I accidentally ran install of dependency packages on a Vagrant server. I also removed and replaced a mirrored folder repeatedly during install errors. My error was on Laravel/Spark 4.~. I was able to run migrations on other projects; I was getting 'killed' very quickly, 300ms timeframe, on a particular project for nearly all commands. Reading other users, I was dreading trying to find the issue or corruption. In my case, a quick Vagrant reload did the trick. killed issue was resolved.
I'm troubleshooting multiple huge php scripts. One of those scripts usually hangs for a very long time. Is their a way through which I can find out the line number by which the script hanged ?
Note that every single script is executed multiple times simultaneously, so I have numerous instances running.
The script is very big and this happens randomly so it's very exhausting to write debugging lines all over the script. If I do so the logs will eat up my disks pace so please, I need a solution that has nothing to do with logging.
Can some kind of a process explorer do that ?
You can install the xdebug extension and use profiling. Then view the profiler output files in WinCacheGrind (Windows) or KCacheGrind (Linux)
How can I test performance of a PHP app using Apache Benchmark?
My environment is Ubuntu Linux - are there packages I can install?
If you have Apache 2 installed, Apache Benchmark is already installed. See man ab on how to use it. In most cases its just something like
ab -n 1000 -c 10 http://localhost/path/to/app
Where -n is the number of all requests, that should be performed and -c is the number of requests, that should be performed in concurrency.
Note, that you don't test the performance of your php project this way, but test everything, that is affected, beginning with the webserver, PHP, your application, the database, your filesystem, and so on. This means, that if you got poor results, that can also be caused by low memory, or you have just many stuff running in the background, or such. Use a profiler to analyze the performance of a php application. A profiler is built-in within xdebug.