I want to accomplish the following behavior in php:
1 - Script gets called with parameters
2- I Intiate a thread for a long running operation
3 - Script should return control to the caller
4- Thread executes till its finished
Is this behavior possible? What i am seeing now, is that the script wont return until the thread has finished executing, which makes sense as the execution of the thread would probably die if the script stops executing , but is there no way to stop blocking the client so they can go on about their business? Am i stuck using some exec() call to get this behavior? Is there a way to get this done with threading only? Id like to avoid using exec if possible..
So if someone calls my script from a browser, it should just return immidiatly, and the long running process should keep executing until its done.
Thanks
Daniel
Yes, its possible. Call your php script via AJAX, and and create multiple instances of the ajax function dynamically. See attached screenshot. When I compared results of running a single function versus 24 instances, my data was processed about 15x faster. I am trying to populate a MySQL table with about 30 million records, and each record involves calculating distance in miles from city center, based on lat/lng. So yes, its no walk in the park. As you can see, I am averaging about See this:
multi threads http://gaysugardaddyfinder.com/screen2.PNG
multi threads http://gaysugardaddyfinder.com/screen.png
This may be a glorious hack or what not - but it sure worked great for me.
My server is a Xeon 72 Core setup with 64 GB RAM.
Related
Here's what I'm trying to accomplish in high-level pseudocode:
query db for a list of names (~100)
for each name (using php) {
query a 3rd party site for xml based on the name
parse/trim the data received
update my db with this data
Wait 15 seconds (the 3rd party site has restrictions and I can only make 4 queries / minute)
}
So this was running fine. The whole script took ~25 minutes (99% of the time was spent waiting 15 seconds after every iteration). My web host then made a change so that scripts will timeout after 70 seconds (understandable). This completely breaks my script.
I assume I need to use cronjobs or command line to accomplish this. I only understand the basic us of cronjobs. Any high level advice on how to split up this work in a cronjob? I am not sure how a cronjob could parse through a dynamic list.
cron itself has no idea of your list and what is done already, but you can use two kinds of cron-jobs.
The first cron-job - that runs for example once a day - could add your 100 items to a job queue.
The second cron-job - that runs for example once every minute in a certain period - can check if there are items in the queue, execute one (or a few) and remove it from the queue.
Note that both cron-jobs are just triggers to start a php script in this case and you have two different scripts, one to set the queue and one to process part of a queue so almost everything is still done in php.
In short, there is not much that is different. Instead of executing the script via modphp or fcgi, you are going to execute it via command line php /path/to/script.php.
Because this is a different environment than http, some things obviously don't work. Sessions, cookies, get and post variables. Output gets send to stdout instead of the browser.
You can pass arguments to your script by using $argv.
i have a big script written in php, which should import a lot of informations in a prestashop installation, using webservices, this script is written in "sections" I mean, there is a function that import the categories, another one that import products, then manufacturers, and so on, there are about 7 - 10 functions called in the main script. Basically I assume that this script must run for about an hour, passing from a function to the next one and so on since it arrives at the last function, then return some values and stops until the next night.
i would like to understand if it could be better :
1) impose a time limit of 30 minutes everytime i enter a new function (this will prevent the timeout)
2) make a chain of pages, each one with a single function call (and of course the time limit)
or any other idea... i would like to :
know if a function has been called (maybe using a global variable?)
be sure that the server will execute the function in order (so the pages chain)...
i hope to have beeen clear, otherwise i'll update the question.
edits:
the script is executed by another server that will call a page, the other server is "unkown" from me, so I simply know only that this page is called (they could also call the function by going on the page) but anyway i have no controll on it.
For any long running scripts, I would run it through the commandline, probably with a cronjob to kick it off. If it's triggered from the outside, I would create a job queue (for example in the database) where you insert a new row to signify that it should run, along with any variable input params. Then the background job would run - say - every 5 minutes, check if there's a new job in the queue. If there's not, just exit. If there is, mark that it has begun work and start processing. When done, mark that it's done.
1 hour of work is a looooooooong time though. Nothing you can do to optimise that?
You can increase the time limit for execution of a script as much as you want using :
set_time_limit(seconds);
And also for long running scripts you need a more memory. you can increase the memory limit using :
ini_set('memory_limit','20M');
And second other thing you have to make sure is that you are running your script on a dedicated server because if you are using a shared server you server will kill automatically long running scripts.
I suspect this question will seem too... silly, but I'm trying to get my head around a nice solution and I'm kinda stuck.
So, here's my situation :
I'm using Ajax to perform a series of tasks. Actually, I'm queuing (or parallely at times, doesn't really matter to me) 1 or more requests.
Show the progress as a percentage (1 out of X tasks performed)
When finished, show the final result.
What I'm trying to do :
Instead of having 3-4 different tasks running (= 3-4 different PHP scripts called asynchronously via Ajax), I would like to have just 1 (= 1 script) - in other words, combine the X scripts into one. (That's easy).
Issues I'm facing :
How could I still report the percentage complete (1 out of X tasks)?
Any ideas?
I would have updated a key in Memcached each time a task is complete. And then you would let your ajax-call simply get the value from your memcached key.
http://php.net/manual/en/book.memcached.php
If you have the request dropped into a database by your original ajax, then kick off the script, you will still have time (assuming the request takes some time to complete) for subsequent tasks to be dropped into the same database to be picked up by the script that is still running.
As for reporting it, perhaps run a quick jquery to see how many items are in the queue? Alternately have the task.php file update the database (possibly another table even) to say how many jobs it has completed and how many are currently still in the queue.
If you don't need to send much data to the PHP script, you can use a "long poll" approach. With this, you don't use AJAX but insert a script tag like this:
<script src="my_php_script?task1=x¶m_t1_1=42&task2=y"></script>
The PHP file can then send back a JavaScript command like
updatePercent(12);
after each task is done. The commands should be executed by the browser whenever they come in. Be sure to call flush after every task.
Looking into Comet may give you other ideas on how to handle the Client-Server connection.
You can manage queue - before sending AJAX request - put the task in queue (could be object or array). Run AJAX asynchronously with complete function which will remove a job from queue when it's done.
You can update progress together with removing job from queue or handle it separately using setTimeout() which will check how many task there are in queue and how many were put in it in total: % = (submitted_tasks - items_in_queue) / submitted_tasks * 100
I'm looking for a way to run a php script multiple times from a browser. Here's the scenario:
I'm building a mySQL table from a series of large files ranging anywhere from 100 megs to 2 gigs. On average, there will be around 150,000 records in the table.
I'm doing so right now by having a javascript function that does an AJAX call to the PHP script. On success, the function sets a timeout to run itself and trigger the AJAX call to run the second hundred.
My thinking behind this was to give the function a second to close out before it runs itself again.
This isn't working so well. The whole function itself works, but performance-wise it is quite slow.
When I wasn't doing 100 records at a time and not wasn't using javascript, just PHP, I could get about 15,000 records into the table before it would time out. Right now it takes about 10 minutes for it to do the same number of records.
I know that the continuous running javascript is bleeding memory and performance like crazy and was just wondering if anyone had any ideas on how to accomplish running a PHP script over and over from a browser. Crons are not an option at this point.
Its called (async) work/job queues, seems you need to explore Gearman
Couldn't you just have the PHP script itself repeat the function multiple times? If the problem is that the function sometimes fails or times out, could you could catch the exception within your script? Or do you have an unavoidable and totally fatal error that really necessitates using an external minder?
I ran into a similar situation... my solution was to use an ajax queue. Essentially you feed a series of ajax calls into a queue which runs them sequentially, starting the next after the previous has returned from the server as successful.
Setting a timeout can run into a situation where the next ajax call is made before the server completed the last. This is the likely cause of your performance issue. I don't really like javascript timeouts myself just for the resource overuse alone.
Google "Ajax Queue" for code that you find useful, or I can post mine, which is jQuery.
configure a cronjob to run your script every minute
I have a little more than hundred php scripts running on my server right now. Each one of them run loops and insert data into my db. I did that in order to learn killing processes in mysql. So to kill them, I coded a php file that loops through the processlist and kill them one by one. The problem is that this script is not executed. It keeps loading in my browser (no errors...). Also do note that I can't manually launch a show processlist in mysql, as mysql is totally overloaded at the moment, and nothing is responding. So what I guess is that my 'killing process' script is the last one on the queue and will only be executed at the end. So my question is to know if there is a way to force a process in mysql and put it at priority number one. Thank you in advance for your replies. Cheers. Marc
This is how I am killing the processes:
$qry = mysql_query("SHOW FULL PROCESSLIST");
while ($row=mysql_fetch_array($qry)) {
$process_id=$row["Id"];
$sql="KILL $process_id";
mysql_query($sql);
}
I'm not sure if this will actually affect MySQL, but on Unix/Linux, you could try calling proc_nice() near the top of your script with a negative increment (like -20). It basically does the same thing as the nice command.
From the Wikipedia page on nice:
"... nice is used to invoke a utility or shell script with a particular priority, thus giving the process more or less CPU time than other processes. A niceness of −20 is the highest priority and 19 or 20 is the lowest priority. The default niceness for processes is inherited from its parent process, usually 0."