I have an old application that uses files instead of a DB to store user data in an xml. All totally legacy. Until the migration to a DB has finished, I need support the current state.
The problem is that sometimes (unreproducible) it occurred, that the file contents were not written completely. This caused the xml files to not be complete and therefore to be invalid.
My assumption is, that the server can't handle the amount of parallel file writes.
Note, that parallel does not mean many scripts write one file, but each script that writes the user content writes for one user per call. There are up to 100 calls to the script in peak times.
The script uses the standard fwrite functionality to write an incoming xml string to the file. No appending, just replacing the whole content, which works in the tests. The problem only occurs when a greater amount of users use the application.
Is there a way to queue the file writes or delay the script calls?
Please comment, if the description is insufficient.
Related
What I'm running here is a graphical file manager, akin to OneDrive or OpenCloud or something like that. Files, Folders, Accounts, and the main server settings are all stored in the database as JSON-encoded objects (yes, I did get rid of columns in favor of json). The problem is that is multiple requests use the same object at once, it'll often save back incorrect data because the requests obviously can't communicate changes to each other.
For example, when someone starts a download, it loads the account object of the owner of that file, increments its bandwidth counter, and then encodes/saves it back to the DB at the end of the download. But say if I have 3 downloads of the same file at once, they'll all load the same account object, change the data as they see fit, and save back their data without regards to the others that overlap. In this case, the 3 downloads would show as 1.
Besides that downloads and bandwidth are being uncounted, I'm also having a problem where I'm trying to create a maintenance function that loads the server object and doesn't save it back for potentially several minutes - this obviously won't work while downloads are happening and manipulating the server object all the meanwhile, because it'll all just be overwritten with old data when the maintenance function finishes.
Basically it's a threading issue. I've looked into PHP APC in the hope I could make objects persist globally between threads but that doesn't work since it just serializes/deserialized data for each request rather than actually having each request point to an object in memory.
I have absolutely no idea how to fix this without completely designing a new system that's totally different.... which sucks.
Any ideas on how I should go about this would be awesome.
Thanks!
It's not a threading issue. Your database doesn't conform to neither of the standards of building databases, including even the first normal form: every cell must contain only one value. When you're storing JSON data in DB, you cannot write an SQL request to make that transaction atomic. So, yes, you need to put that code in a trash bin.
In case you really need to get that code working, you can use some mutexes to synchronize running PHP scripts. The most common implementation in PHP is file mutex.
You can try to use flock , I guess you already have a user id before getting JSON from DB.
$lockfile = "/tmp/userlocks/$userid.txt";
$fp = fopen($lockfile, "w+");
if (flock($fp, LOCK_EX)) {
//Do your JSON update
flock($fp, LOCK_UN); //unlock
}else{
// lock exist
}
What you need to figure out is what to do when there is a lock, maybe wait for 0.5 secs and try to obtain lock again , or send a message "Only one simultaneous download allowed " or ....
On my webservice users are allowed to create forms and let theire friends or co-workers create data with them. The collected data can be downloaded as a zip file stream. Sometimes users have huge amounts of data (up to 2gb) and the server simply kills the php process for obvious reasons. Is it somehow possible to create such a file on client side without flash,java (btw java doesn't work anyway for most of my users) etc. ?
Increase your script timeout and memory usage.
use set_time_limit function Docs
And use ini_set for memory_limit parameter.
And one more solution is to give the clients file parts. i.e. give them a limit for downloading the number of records. i.e. 1-1000, 1001-2000 etc
If you have control over the web server process I suggest you explore x-send-file as a solution to this.
See this SO question
In essence it will end the php process and send the file via the http server. This way time limits aren't an issue and you don't have a php instance hanging around.
Create a worker shell that keeps running in the background in a loop and checks for new data. If it finds new, unprocessed data have it prepare the download in the background. When the data is ready for download flag it as "ready" and inform the user (by email, polling via ajax for an status update, however you like) that his data was processed and is ready for download.
You can use nice to limit the CPU power used for that shell to avoid that it consumes all the available processing power and your site becomes slow.
That's exactly how I handle audio and video processing in one of my projects and it works fine.
So here's the lowdown:
The client i'm developing for is on HostGator, which has limited their max_execution_time to 30 seconds and it cannot be overridden (I've tried and confirmed it cannot be via their support and wiki)
What I'm have the code doing is take an uploaded file and...
loop though the xml
get all feed download links within the file
download each xml file
individually loop though each xml array of each file and insert the information of each item into the database based on where they are from (i.e. the filename)
Now is there any way I can queue this somehow or split the workload into multiple files possibly? I know the code works flawlessly and checks to see if each item exists before inserting it but I'm stuck getting around the execution_limit.
Any suggestions are appreciated, let me know if you have any questions!
The timelimit is in effect only when executing PHP scripts through a webserver, if you execute the script from CLI or as a background process, it should work fine.
Note that executing an external script is somewhat dangerous if you are not careful enough, but it's a valid option.
Check the following resources:
Process Control Extensions
And specifically:
pcntl-exec
pcntl-fork
Did you know you can trick the max_execution_time by registering a shutdown handler? Within that code you can run for another 30 seconds ;-)
Okay, now for something more useful.
You can add a small queue table in your database to keep track of where you are in case the script dies mid-way.
After getting all the download links, you add those to the table
Then you download one file and process it; when you're done, you check them off (delete from) from the queue
Upon each run you check if there's still work left in the queue
For this to work you need to request that URL a few times; perhaps use JavaScript to keep reloading until the work is done?
I am in such a situation. My approach is similar to Jack's
accept that execution time limit will simply be there
design the application to cope with sudden exit (look into register_shutdown_function)
identify all time-demanding parts of the process
continuously save progress of the process
modify your components so that they are able to start from arbitrary point, e.g. a position in a XML file or continue downloading your to-be-fetched list of XML links
For the task I made two modules, Import for the actual processing; TaskManagement for dealing with these tasks.
For invoking TaskManager I use CRON, now this depends on what webhosting offers you, if it's enough. There's also a WebCron.
Jack's JavaScript method's advantage is that it only adds requests if needed. If there are no tasks to be executed, the script runtime will be very short and perhaps overstated*, but still. The downsides are it requires user to wait the whole time, not to close the tab/browser, JS support etc.
*) Likely much less demanding than 1 click of 1 user in such moment
Then of course look into performance improvements, caching, skipping what's not needed/hasn't changed etc.
I don’t know much PHP and I want to see how/if the following algorithm can be implemented in PHP:
I am sending a string to a PHP script via HTTP GET. When the script receives the data, I PROCESS it and write the result to a txt file. The file already exists. I only update some lines. My problem is: what happens if the server fails while my data is processed? How can I minimize the damage in case of server/script failure?
The processing of data may take up to one second. So I think it is a high risk that the server will breakdown during it. Therefore, I am thinking in breaking it in two parts:
One script (let’s call it RECEIVER) that receives the data from HTTP GET and store it to a file (called Jobs.txt). It should finish really fast as it has to write only 20-50 chars.
A second script (let’s call it PROCESSOR) that checks this file every 2-3 seconds to see if new entries were added. If it finds new entries, it processes the data, save it and finally deletes the entry from the Jobs file. If the server fails, on resume, maybe I can start my PROCESSOR and start the work from where it was interrupted.
How it sounds?
Problems: What happens if two users are sending GET commands at the same time to the RECEIVER? It will be a conflict on who will write to the file. Also, the PROCESSOR may conflict over that file as it also wants to write to it. How can I fix this?
to send some data to PHP use just URL
http://www.mydomain.com/myscript.php?getparam1=something&getparam2=something_else
to get it from PHP script (in this example myscript.php)
$first_parameter = $_GET['getparam1'];
$second_parameter = $_GET['getparam2'];
// or use $_REQUEST instead of $_GET
or
$get_array = $_GET;
print_r($get_array);
or
$get_array = explode('; ', $_SERVER['QUERY_STRING']);
to write files to text file use:
error: This is XXI century!
Consider using database rather then text file.
As Michael J.V. suggested, using a DB instead of writing to a file will solve some of the problems: "you won't get half-written data and end up with wingdings in your dataset". ACID standards should be used while programming the script: "as you can see, what you need conforms to the ACID standards".
I am working in a tool in PHP that processes a lot of data and takes a while to finish. I would like to keep the user updated with what is going on and the current task processed.
What is in your opinion the best way to do it? I've got some ideas but can't decide for the most effective one:
The old way: execute a small part of the script and display a page to the user with a Meta Redirect or a JavaScript timer to send a request to continue the script (like /script.php?step=2).
Sending AJAX requests constantly to read a server file that PHP keeps updating through fwrite().
Same as above but PHP updates a field in the database instead of saving a file.
Does any of those sound good? Any ideas?
Thanks!
Rather than writing to a static file you fetch with AJAX or to an extra database field, why not have another PHP script that simply returns a completion percentage for the specified task. Your page can then update the progress via a very lightweight AJAX request to said PHP script.
As for implementing this "progress" script, I could offer more advice if I had more insight as to what you mean by "processes a lot of data". If you are writing to a file, your "progress" script could simply check the file size and return the percentage complete. For more complex tasks, you might assign benchmarks to particular processes and return an estimated percentage complete based on which process has completed last or is currently running.
UPDATE
This is one suggested method to "check the progress" of an active script which is simply waiting for a response from a request. I have a data mining application that I use a similar method for.
In your script that makes the request you're waiting for (the script you want to check the progress of), you can store (either in a file or a database, I use a database as I have hundreds of processes running at any time which all need to track their progress, and I have another script that allows me to monitor progress of these processes) a progress variable for the process. When the process begins, set this to 1. You can easily select an arbitrary number of 'checkpoints' the script will pass and calculate the percentage given the current checkpoint. For a large request, however, you might be more interested in knowing the approximate percent the request has completed. One possible solution would be to know the size of the returned content and set your status variable according to the percentage received at any moment. I.e. if you receive the request data in a loop, each iteration you could update the status. Or if you are downloading to a flat file you could poll the size of the file. This could be done less accurately with time (rather than file size) if you know the approximate time the request should take to complete and simply compare against the script's current execution time. Obviously neither of these are perfect solutions, but I hope they'll give you some insight into your options.
I suggest using the AJAX method, but not using a file or a database. You could probably use session values or something like that, that way you don't have to create a connection or open a file to do anything.
In the past, I've just written messages out to the page and used flush() to flush the output buffer. Very simple, but it may not work correctly on every web server or with every web browser (as they may do their own internal buffering).
Personally, I like your second option the best. Should be reliable and fairly simple to implement.
I like option 2 - using AJAX to read a status file that PHP writes to periodically. This opens up a lot of different presentation options. If you write a JSON object to the file, you can easily parse it and display things like a progress bar, status messages, etc...
A 'dirty' but quick-and-easy approach is to just echo out the status as the script runs along. So long as you don't have output buffering on, the browser will render the HTML as it receives it from the server (I know WordPress uses this technique for it's auto-upgrade).
But yes, a 'better' approach would be AJAX, though I wouldn't say there's anything wrong with 'breaking it up' use redirects.
Why not incorporate 1 & 2, where AJAX sends a request to script.php?step=1, checks response, writes to the browser, then goes back for more at script.php?step=2 and so on?
if you can do away with IE then use server sent events. its the ideal solution.