I am trying to crawl every page on my site (ran by a cron) to update data. There are roughly 500 pages.
I have tried 2 options.
PHP Simple HTML DOM Parser
PHP get_headers
Using either of the above, each page roughly takes 1.402 seconds to load. In total this takes about 570 seconds.
Is there a more efficient way of doing this?
Request pages in parallel (i.e. concurrently). Then it won't matter how long each request takes, because many will fire at once.
There are many ways to achieve this, but here is one example:
curl www.website.com/page1 &
curl www.website.com/page2 &
curl www.website.com/page3 &
Use xargs or other tools to prevent flooding the server with too many concurrent connections. e.g. Bash script processing commands in parallel
It can be complicated to run commands in parallel inside a single PHP script. Easier to use the command line, if possible.
Related
I have a mailing function and 100k email ids. I want to call a function multiple times like 10 times and in each time it will process 10k emails. I want to call this function without waiting for response just call a function and another and another without getting response.
I tried pthread for multi threading but can't run it successfully.
I am using my sql database
You can use multiple PHP processes for that, just like you can use multiple threads, there isn't much of a difference for PHP, as PHP is shared nothing.
You probably want to wait for it to finish and to notice any errors, but don't want to wait for completion before launching another process. Amp is perfectly suited for such use cases. Its amphp/parallel package makes multi-processing easier than using PHP's native API.
You can find a usage example in the repository.
php name_of_script.php &>/dev/null &
this line will start your script in the background.
I suppose you do not want to control a critical process like sending mails from your browser? What if your connection breaks a few seconds?
If so: still use command line approach, but using exec
$command = '/usr/bin/php path/to/script.php & echo $!');
I have a webpage that when users go to it, multiple (10-20) Ajax requests are instantly made to a single PHP script, which depending on the parameters in the request, returns a different report with highly aggregated data.
The problem is that a lot of the reports require heavy SQL calls to get the necessary data, and in some cases, a report can take several seconds to load.
As a result, because one client is sending multiple requests to the same PHP script, you end up seeing the reports slowly load on the page one at a time. In other words, the generating of the reports is not done in parallel, and thus causes the page to take a while to fully load.
Is there any way to get around this in PHP and make it possible for all the requests from a single client to a single PHP script to be processed in parallel so that the page and all its reports can be loaded faster?
Thank you.
As far as I know, it is possible to do multi-threading in PHP.
Have a look at pthreads extension.
What you could do is make the report generation part/function of the script to be executed in parallel. This will make sure that each function is executed in a thread of its own and will retrieve your results much sooner. Also, set the maximum number of concurrent threads <= 10 so that it doesn't become a resource hog.
Here is a basic tutorial to get you started with pthreads.
And a few more examples which could be of help (Notably the SQLWorker example in your case)
Server setup
This is more of a server configuration issue and depends on how PHP is installed on your system: If you use php-fpm you have to increase the pm.max_children option. If you use PHP via (F)CGI you have to configure the webserver itself to use more children.
Database
You also have to make sure that your database server allows that many concurrent processes to run. It won’t do any good if you have enough PHP processes running but half of them have to wait for the database to notice them.
In MySQL, for example, the setting for that is max_connections.
Browser limitations
Another problem you’re facing is that browsers won’t do 10-20 parallel requests to the same hosts. It depends on the browser, but to my knowledge modern browsers will only open 2-6 connections to the same host (domain) simultaneously. So any more requests will just get queued, regardless of server configuration.
Alternatives
If you use MySQL, you could try to merge all your calls into one request and use parallel SQL queries using mysqli::poll().
If that’s not possible you could try calling child processes or forking within your PHP script.
Of course PHP can execute multiple requests in parallel, if it uses a Web Server like Apache or Nginx. PHP dev server is single threaded, but this should ony be used for dev anyway. If you are using php's file sessions however, access to the session is serialized. I.e. only one script can have the session file open at any time. Solution: Fetch information from the session at script start, then close the session.
I have a PHP script that needs to make three separate API calls, combine the results, and output it for the user.
The problem is that each API call takes about 5 seconds to execute. With 3 API calls at 5 seconds each, it take about 15 seconds to execute the script.
Is there a way that I can somehow start the three API calls simultaneously and once the last one finishes, combine the results? If that's possible I could potentially reduce the length of time from 15 to 5 seconds, dramatically improving my user's experience.
I researched asynchronous function calls in PHP, but there doesn't seem to be a lot of good options. I'm hoping that someone out there has been in a similar situation and found an elegant way to handle this.
PHP scripts by themselves are single threaded. There are ways to "fork" child processes in PHP using pcntl_fork function. But as far as I know, this only really works well with the CLI sapi. With any of the web server sapi's, either it's buggy or not supported.
If you have to initiate it from a web request, you could try using shell_exec to spawn a master PHP CLI process in the background (append & at the end), and then make use of pcntl_fork to divide up the work.
If you're just waiting on web requests, I would follow Dagon's suggestion of using the curl_multi functions. But forking can be useful if you have other intensive tasks, if used correctly.
http://docs.php.net/Thread
That's the entire answer, as short as it is: yes, in exactly the way you imagine it should ( I hope ).
I have some php code, that execute for a very long time.
I need to realise next scheme:
User enter on some page(page 1)
This page starts execution of my large PHP script in background .(Every change is writting to database)
We sent every N seconds query to database to get current status of execution.
I don't want to use exec command because 1000 users makes 1000 php processes. It's not way for me...
So you basically want a queue (possibly stored in a database) and a command line script ran by cron that process queued items.
Clarification: I'm not sure about what's unclear about my answer, but this complies with the two requirements imposed by the question:
The script cannot be aborted by the client
You share a single process between 1,000 clients
Use http requests to the local http server from within your script in combination with phps ignore_client_abort() function.
That way you keep the load inside the http servers worker processes, have a natural limit and queuing of requests comes for free.
You can use CLI to execute multiple PHP scripts
or
you can try Easy Parallel Processing in PHP
Greetings All!
I am having some troubles on how to execute thousands upon thousands of requests to a web service (eBay), I have a limit of 5 million calls per day, so there are no problems on that end.
However, I'm trying to figure out how to process 1,000 - 10,000 requests every minute to every 5 minutes.
Basically the flow is:
1) Get list of items from database (1,000 to 10,000 items)
2) Make a API POST request for each item
3) Accept return data, process data, update database
Obviously a single PHP instance running this in a loop would be impossible.
I am aware that PHP is not a multithreaded language.
I tried the CURL solution, basically:
1) Get list of items from database
2) Initialize multi curl session
3) For each item add a curl session for the request
4) execute the multi curl session
So you can imagine 1,000-10,000 GET requests occurring...
This was ok, around 100-200 requests where occurring in about a minute or two, however, only 100-200 of the 1,000 items actually processed, I am thinking that i'm hitting some sort of Apache or MySQL limit?
But this does add latency, its almost like performing a DoS attack on myself.
I'm wondering how you would handle this problem? What if you had to make 10,000 web service requests and 10,000 MySQL updates from the return data from the web service... And this needs to be done in at least 5 minutes.
I am using PHP and MySQL with the Zend Framework.
Thanks!
I've had to do something similar, but with Facebook, updating 300,000+ profiles every hour. As suggested by grossvogel, you need to use many processes to speed things up because the script is spending most of it's time waiting for a response.
You can do this with forking, if your PHP install has support for forking, or you can just execute another PHP script via the command line.
exec('nohup /path/to/script.php >> /tmp/logfile 2>&1 & echo $!'), $processId);
You can pass parameters (getopt) to the php script on the command line to tell it which "batch" to process. You can have the master script do a sleep/check cycle to see if the scripts are still running by checking for the process id's. I've tested up to 100 scripts running at once in this manner, at which point the CPU load can get quite high.
Combine multiple processes with multi-curl, and you should easily be able to do what you need.
My two suggestions are (a) do some benchmarking to find out where your real bottlenecks are and (b) use batching and cacheing wherever possible.
Mysqli allows multiple-statement queries, so you could definitely batch those database updates.
The http requests to the web service are more likely the culprit, though. Check the API you're using to see if you can get more info from a single call, maybe? To break up the work, maybe you want a single master script to shell out to a bunch of individual processes, each of which makes an api call and stores the results in a file or memcached. The master can periodically read the results and update the db. (Careful to rotate the data store for safe reading and writing by multiple processes.)
To understand your requirements better, you must implement your solution only in PHP? Or you can interface a PHP part with another part written in another language?
If you could not go for another language, try to perform this update maybe as php script that runs in the background and not through the apache.
You can follow Brent Baisley advice for a simple use case.
If you want to build a robuts solution, then you need to :
set up a representation of the actions in a table in database that will be your process queue;
set up a script that pop this queue and process your action;
set up a cron daemon that run this script every x.
This way you can have 1000 PHP scripts running, using your OS parallelism capabilities and not hanging when ebay is taking to to respond.
The real advantage of this system is that you can fully control the firepower you throw at your task by adjusting :
the number of request one PHP script does;
the order / number / type / priority of the action in the queue;
the number or scripts the cron daemon runs.
Thanks everyone for the awesome and quick answers!
The advice from Brent Baisley and e-satis works nicely, rather than executing the sub-processes using CURL like i did before, the forking takes a massive load off, it also nicely gets around the issues with max out my apache connection limit.
Thanks again!
It is true that PHP is not multithreaded, but it can certainly be setup with multiple processes.
I have created a system that resemebles the one that you are describing. It's running in a loop and is basically a background process. It uses up to 8 processes for batch processing and a single control process.
It is somewhat simplified because i do not have to have any communication between the processes. Everything resides in a database so each process is spawned with the full context taken from the database.
Here is a basic description of the system.
1. Start control process
2. Check database for new jobs
3. Spawn child process with the job data as a parameter
4. Keep a table of the child processes to be able to control the number of simultaneous processes.
Unfortunately it does not appear to be a widespread idea to use PHP for this type of application, and i really had to write wrappers for the low level functions.
The manual has a whole section on these functions, and it appears that there are methods for allowing IPC as well.
PCNTL has the functions to control forking/child processes, and Semaphore covers IPC.
The interesting part of this is that i'm able to fork off actual PHP code, not execute other programs.