I'm looking for a way to run a php script multiple times from a browser. Here's the scenario:
I'm building a mySQL table from a series of large files ranging anywhere from 100 megs to 2 gigs. On average, there will be around 150,000 records in the table.
I'm doing so right now by having a javascript function that does an AJAX call to the PHP script. On success, the function sets a timeout to run itself and trigger the AJAX call to run the second hundred.
My thinking behind this was to give the function a second to close out before it runs itself again.
This isn't working so well. The whole function itself works, but performance-wise it is quite slow.
When I wasn't doing 100 records at a time and not wasn't using javascript, just PHP, I could get about 15,000 records into the table before it would time out. Right now it takes about 10 minutes for it to do the same number of records.
I know that the continuous running javascript is bleeding memory and performance like crazy and was just wondering if anyone had any ideas on how to accomplish running a PHP script over and over from a browser. Crons are not an option at this point.
Its called (async) work/job queues, seems you need to explore Gearman
Couldn't you just have the PHP script itself repeat the function multiple times? If the problem is that the function sometimes fails or times out, could you could catch the exception within your script? Or do you have an unavoidable and totally fatal error that really necessitates using an external minder?
I ran into a similar situation... my solution was to use an ajax queue. Essentially you feed a series of ajax calls into a queue which runs them sequentially, starting the next after the previous has returned from the server as successful.
Setting a timeout can run into a situation where the next ajax call is made before the server completed the last. This is the likely cause of your performance issue. I don't really like javascript timeouts myself just for the resource overuse alone.
Google "Ajax Queue" for code that you find useful, or I can post mine, which is jQuery.
configure a cronjob to run your script every minute
Related
I'm building a PHP application which has a database containing approximately 140 URL's.
The goal is to download a copy of the contents of these web pages.
I've already written code which reads the URL's from my database then uses curl to grab a copy of the page. It then gets everything between <body> </body>, and writes it to a file. It also takes into account redirects, e.g. if I go to a URL and the response code is 302, it will follow the appropriate link. So far so good.
This all works ok for a number of URL's (maybe 20 or so) but then my script times out due to the max_execution_time being set to 30 seconds. I don't want to override or increase this, as I feel that's a poor solution.
I've thought of 2 work arounds but would like to know if these are a good/bad approach, or if there are better ways.
The first approach is to use a LIMIT on the database query such that it splits the task up into 20 rows at a time (i.e. run the script 7 separate times, if there were 140 rows). I understand from this approach it still needs to call the script, download.php, 7 separate times so would need to pass in the LIMIT figures.
The second is to have a script where I pass in the ID of each individual database record I want the URL for (e.g. download.php?id=2) and then do multiple Ajax requests to them (download.php?id=2, download.php?id=3, download.php?id=4 etc). Based on $_GET['id'] it could do a query to find the URL in the database etc. In theory I'd be doing 140 separate requests as it's a 1 request per URL set up.
I've read some other posts which have pointed to queueing systems, but these are beyond my knowledge. If this is the best way then is there a particular system which is worth taking a look at?
Any help would be appreciated.
Edit: There are 140 URL's at the moment, and this is likely to increase over time. So I'm looking for a solution that will scale without hitting any timeout limits.
I dont agree with your logic , if the script is running OK and it needs more time to finish, just give it more time it is not a poor solution.What you are suggesting makes things more complicated and will not scale well if your urls increase.
I would suggest moving your script to the command line where there is no time limit and not using the browser to execute it.
When you have an unknown list wich will take an unknown amount of time asynchronous calls are the way to go.
Split your script into a single page download (like you proposed, download.php?id=X).
From the "main" script get the list from the database, iterate over it and send an ajax call to the script for each one. As all the calls will be fired all at once, check for your bandwidth and CPU time. You could break it into "X active task" using the success callback.
You can either set the download.php file to return success data or to save it to a database with the id of the website and the result of the call. I recommend the later because you can then just leave the main script and grab the results at a later time.
You can't increase the time limit indefinitively and can't wait indefinitively time to complete the request, so you need a "fire and forget" and that's what asynchronous call does best.
As #apokryfos pointed out, depending on the timing of this sort of "backups" you could fit this into a task scheduler (like chron). If you call it "on demand", put it in a gui, if you call it "every x time" put a chron task pointing the main script, it will do the same.
What you are describing sounds like a job for the console. The browser is for the users to see, but your task is something that the programmer will run, so use the console. Or schedule the file to run with a cron-job or anything similar that is handled by the developer.
Execute all the requests simultaneously using stream_socket_client(). Save all the socket IDs in an array
Then loop through the array of IDs with stream_select() to read the responses.
It's almost like multi-tasking within PHP.
I have an html which run a php several times (about 20).
Something like
for(i=0;i<20; i++){
$.get('script.php?id='+i,function(){...});
}
Every time the script runs, it get some content through different websites (every time a different one), so each script takes from 1 to 10 seconds to complete and give a response.
I want to run all the script simultaneously to be faster, but 2 things makes it slow: the first is on the html page, it seems that ajax requested are queued after first five, at least chrome developer said me that... (this can be fixed easily I think, I've not bothered yet to find a solution); the second thing is on php side: even if the first 5 scripts are triggered together, they are run sequentially, not even in order. I 've put some
echo microtime(true);
around the script to get wherever the script is slow and what I found out (a bit surprised) is that the time on the beginning of the script (the one which should be run at almost the same time on all script) is different: the difference is very consistent, also 10 seconds, like if the second script wait the first to end before begin. How can I have all the script be running together at the same time? Thank you.
I very frankly advise that you should not attempt anything "multi-threaded" here, "nor particularly 'fancy.'"
Design your application to have some kind of queue (it can be a simple array) of "things that need to be done." Then, issue "some n" number of initial AJAX requests: certainly no more than 3 to 5.
Now, wait for notification that each request has succeeded or failed. After somehow recording the status of the request, de-queue another request and start it.
In this way, "n requests, but no more than n requests," will be active at any one time, and yes, they will each take a different amount of time, "but nobody cares."
"JavaScript is not multi-threaded," and we have no need for it. Everything is done by events, which are handled one at a time, and that happens to work beautifully.
Your program runs until the queue is empty and all of the outstanding AJAX requests have been completed (or, have failed).
There's absolutely no advantage in "getting too fancy" when the requests that you are performing might each take several seconds to complete. There's also no particular advantage in trying to queue-up too many TCP/IP network requests. Predictable Simplicity, in this case, will (IMHO) rule the day.
I want to accomplish the following behavior in php:
1 - Script gets called with parameters
2- I Intiate a thread for a long running operation
3 - Script should return control to the caller
4- Thread executes till its finished
Is this behavior possible? What i am seeing now, is that the script wont return until the thread has finished executing, which makes sense as the execution of the thread would probably die if the script stops executing , but is there no way to stop blocking the client so they can go on about their business? Am i stuck using some exec() call to get this behavior? Is there a way to get this done with threading only? Id like to avoid using exec if possible..
So if someone calls my script from a browser, it should just return immidiatly, and the long running process should keep executing until its done.
Thanks
Daniel
Yes, its possible. Call your php script via AJAX, and and create multiple instances of the ajax function dynamically. See attached screenshot. When I compared results of running a single function versus 24 instances, my data was processed about 15x faster. I am trying to populate a MySQL table with about 30 million records, and each record involves calculating distance in miles from city center, based on lat/lng. So yes, its no walk in the park. As you can see, I am averaging about See this:
multi threads http://gaysugardaddyfinder.com/screen2.PNG
multi threads http://gaysugardaddyfinder.com/screen.png
This may be a glorious hack or what not - but it sure worked great for me.
My server is a Xeon 72 Core setup with 64 GB RAM.
I suspect this question will seem too... silly, but I'm trying to get my head around a nice solution and I'm kinda stuck.
So, here's my situation :
I'm using Ajax to perform a series of tasks. Actually, I'm queuing (or parallely at times, doesn't really matter to me) 1 or more requests.
Show the progress as a percentage (1 out of X tasks performed)
When finished, show the final result.
What I'm trying to do :
Instead of having 3-4 different tasks running (= 3-4 different PHP scripts called asynchronously via Ajax), I would like to have just 1 (= 1 script) - in other words, combine the X scripts into one. (That's easy).
Issues I'm facing :
How could I still report the percentage complete (1 out of X tasks)?
Any ideas?
I would have updated a key in Memcached each time a task is complete. And then you would let your ajax-call simply get the value from your memcached key.
http://php.net/manual/en/book.memcached.php
If you have the request dropped into a database by your original ajax, then kick off the script, you will still have time (assuming the request takes some time to complete) for subsequent tasks to be dropped into the same database to be picked up by the script that is still running.
As for reporting it, perhaps run a quick jquery to see how many items are in the queue? Alternately have the task.php file update the database (possibly another table even) to say how many jobs it has completed and how many are currently still in the queue.
If you don't need to send much data to the PHP script, you can use a "long poll" approach. With this, you don't use AJAX but insert a script tag like this:
<script src="my_php_script?task1=x¶m_t1_1=42&task2=y"></script>
The PHP file can then send back a JavaScript command like
updatePercent(12);
after each task is done. The commands should be executed by the browser whenever they come in. Be sure to call flush after every task.
Looking into Comet may give you other ideas on how to handle the Client-Server connection.
You can manage queue - before sending AJAX request - put the task in queue (could be object or array). Run AJAX asynchronously with complete function which will remove a job from queue when it's done.
You can update progress together with removing job from queue or handle it separately using setTimeout() which will check how many task there are in queue and how many were put in it in total: % = (submitted_tasks - items_in_queue) / submitted_tasks * 100
I have daemon script written in Perl that checks a database tables for rows, pulls them in one by one, sends the contents via HTTP post to another service, then logs the result and repeats (only a single child). When there are rows present, the first one is posted and logged immediately, but every subsequent one is delayed for around 20 seconds. There are no sleep()'s running, and I can't find any other obvious delays. Any ideas?
Without code nobody can help you. You should reduce your code in a minimum testcase that represent your error and post it here. Probablby if you this often you find the error yourself.
And probably even without "sleep" your process can hang if you don't did asynchronous programming and call something that just takes a long of time to execute.
You can find such code that hangs if you execute your program in a profiler like Devel::NYTProf