I have a large php web scraping script that logs the results onto a mysql database as it goes.The script generally runs for 5 to 10 minutes at a time.
The problem is that when this script is running other pages on the application will not load.
The script is on a dedicated server with plenty of RAM so I have tried increasing the allowed memory usage for MYSQL and PHP. Also increased the max allowed connections. None of this has helped.
Does anyone have any ideas about what else I can try?
Probably, problem in your session. Try to use session_write_close() before you start "big script".
well, there's a big difference between "slowing down" and "not load"!
try the following:
build a static html site and check if it is loaded well during the execution of the big script
build a php site that doesn't connect to the DB (just echo something) during the execution of the big script
build a small php site that connects to the DB (just select something from a table)
if 1 or 2 doesn't work well, your problem has something todo with the web server or server resources. if 3 doesn't work well, there could be resource issues with the mysql server.
if everything works well, check the scraping script. does it lock any table that is needed by the main application?
Related
I'm currently on XAMPP and Mac OSX and I have the following scenario and problem.
Let's pretend I have two VirtualHosts
a.dev and b.dev
a.dev has a cronjob that takes some time. Now, while doing that I want to develop further on other sites, but the cronjob on a.dev has blocked everything and I can't access anything on the page anymore. However, b.dev works without problems, so it seems that there is only a problem with the hosts.
Or could it be the database that makes problems? That it somehow locks tables? But then I wouldn't be able to access the phpmyadmin page of the database, right?
Your cron job is probably eating all the resources allocated for the php script on a.dev. You need to raise the limits in your php.ini or improve the script:
you delete the variables you don't need anymore to free the memory
if your script contains a loop, you can use sleep() after each loop to limit the CPU usage
I have a php script that inserts random data in mysql database. (approximative 1 milion rows)
Afer the script begins to run on my browser, after a few minutes (ex. 10 min or so) the script stops. (Chrome is giving me "No data recieved")
In my script I have set the followings:
ini_set('display_errors','On');
ini_set('error_reporting',E_ALL);
ini_set('max_execution_time' ,0);
ini_set('set_memory_limit', '10240MB');
For this project I am using the Laravel framework but I don't think it has something to do with it.
Have any ideas?
It's always a good approach to do such long running jobs via SSH (if you have SSH access to your server). Then you can execute any PHP-Script via php your-script.php. You just to keep two things in mind: you need to bootstrap your framework properly (if the script depends on it), and then there is also a timeout for SSH connections. You can work around this limitation by sending the running job in the background or use the screen command.
Found the problem. It was from the framework.
For best practices I recomend:
https://laracasts.com/lessons/eager-loading
(Our server is Linux based)
I'm an experienced PHP developer but first time i'll develop a bot which always running and fetch some datas.
I'll explain my application with a simple (and sample) scenario. I have about 2000 web site url and my application will visit this url's and record contents of web page's . This application will work 7 days 24 hours. It will start working again when it's finish 2000 web sites.
But i need some suggestions for my server. As you see, my application will be run infinity until i shut down server. I can do this infinity loop with this :
while(true)
{
APPLICATION CODES HERE
}
But i think this will be an evil for server :) Is it possible to doing something like this, on server side?
Also i think using cronjobs but it's not work for my scenario. Because my script start working again asap it's finish working. I have to "start again when you finish your work" , not "start every 30 minutes" . Because i don't know, maybe fetching all 2000 websites, will take more than 30 minutes or less than 30 minutes.
I hope i explained it very well.
Also i'm worried about memory usage. As you know garbage collector cleans memory after every PHP script stop. But as i said, my app won't stop for days (maybe weeks) . So garbage collector won't be triggered. I'm manually unsetting (unset() function) all used variables at end of script. Is it enough?
I need some suggestions from server administrators :)
PS. I'm developing it as console application, not a web application. I can execute it from command line.
Batch processing.. store all the sites in a csv or something, mark them after completion, then work on all the ones non-marked, then work on all the marked.. etc. Only do say 1 or 5 at a time, initiate batch script every minute from cron..
Don't even try to work on all of them at once.. any errors and you won't know what happened..
Could even store the jobs in a database, store processing stats etc.. allows for fine-tuning and better reporting.
You will probably hit time-limits trying to run infinite php scripts, even from the command line.. also your server admin will hate you. Will probably run into Memory limits if you don't release resources properly.. far too easily done with php.
Read: http://www.ibm.com/developerworks/opensource/library/os-php-batch/
Your script could just run through the list once and quit. That way, what ever resources php is holding can be freed.
Then have a shell script that calls the php script in an infinite loop.
As php is not designed for long running task, I am not sure if the garbage collection is up to the task. Quiting after every run will force it to release everything.
I am currently experiencing slowness with one of my servers. It is running an apache2 server with PHP and MySQL. The MySQL server is hosted on the same machine as the webserver itself.
Whenever I request a PHP file containing MySQL queries the page needs approximately 24 seconds to show up. While requesting the page the CPU usage of apache2 goes up to 11% (!) which is very much in comparison to what it used to be a week ago.
Non-PHP files or PHP files without MySQL queries are showing up immediately.
What could be causing the problems with scripts containing MySQL queries?
I was unable to find any useful information inside the apache error logs.
In mysql console
show full processlist; <-- to show what are the current SQL
To check where is the log file:-
show variables like '%log%'; <-- to show mysql variables
When doing query benchmark / testing, always remember to turn off query cache, using :-
set session query_cache_type=off;
database queries take time to run, and each query involves opening up at least one file. file access is slow.
you can speed up the requests by running the database in RAM instead of from the hard-drive, but the real answer is probably to cache as much as you can so you're doing as little database querying as possible.
You can check if the mysql database is greater then 2GB (or 4GB) because of some cms logging function and exceed a file size limit.
I'm currently running a Linux based VPS, with 768MB of Ram.
I have an application which collects details of domains and then connect to a service via cURL to retrieve details of the pagerank of these domains.
When I run a check on about 50 domains, it takes the remote page about 3 mins to load with all the results, before the script can parse the details and return it to my script. This causes a problem as nothing else seems to function until the script has finished executing, so users on the site will just get a timer / 'ball of death' while waiting for pages to load.
**(The remote page retrieves the domain details and updates the page by AJAX, but the curl request doesnt (rightfully) return the page until loading is complete.
Can anyone tell me if I'm doing anything obviously wrong, or if there is a better way of doing it. (There can be anything between 10 and 10,000 domains queued, so I need a process that can run in the background without affecting the rest of the site)
Thanks
A more sensible approach would be to "batch process" the domain data via the use of a cron triggered PHP cli script.
As such, once you'd inserted the relevant domains into a database table with a "processed" flag set as false, the background script would then:
Scan the database for domains that aren't marked as processed.
Carry out the CURL lookup, etc.
Update the database record accordingly and mark it as processed.
...
To ensure no overlap with an existing executing batch processing script, you should only invoke the php script every five minutes from cron and (within the PHP script itself) check how long the script has been running at the start of the "scan" stage and exit if its been running for four minutes or longer. (You might want to adjust these figures, but hopefully you can see where I'm going with this.)
By using this approach, you'll be able to leave the background script running indefinitely (as it's invoked via cron, it'll automatically start after reboots, etc.) and simply add domains to the database/review the results of processing, etc. via a separate web front end.
This isn't the ideal solution, but if you need to trigger this process based on a user request, you can add the following at the end of your script.
set_time_limit(0);
flush();
This will allow the PHP script to continue running, but it will return output to the user. But seriously, you should use batch processing. It will give you much more control over what's going on.
Firstly I'm sorry but Im an idiot! :)
I've loaded the site in another browser (FF) and it loads fine.
It seems Chrome puts some sort of lock on a domain when it's waiting for a server response, and I was testing the script manually through a browser.
Thanks for all your help and sorry for wasting your time.
CJ
While I agree with others that you should consider processing these tasks outside of your webserver, in a more controlled manner, I'll offer an explanation for the "server standstill".
If you're using native php sessions, php uses an exclusive locking scheme so only a single php process can deal with a given session id at a time. Having a long running php script which uses sessions can certainly cause this.
You can search for combinations of terms like:
php session concurrency lock session_write_close()
I'm sure its been discussed many times here. I'm too lazy to search for you. Maybe someone else will come along and make an answer with bulleted lists and pretty hyperlinks in exchange for stackoverflow reputation :) But not me :)
good luck.
I'm not sure how your code is structured but you could try using sleep(). That's what I use when batch processing.