I am using PHP and AJAX requests to get the output of a program that is always running and print it on a webpage at 5 second intervals. Sometimes this log file can get up to 2mb in size. It doesn't seem practical to me for the AJAX request to fetch the whole contents of this file every 5 seconds if the user has already gotten the full contents at least once. The request just needs to get whatever contents the user hasn't gotten in a previous request.
Problem is, I have no clue on where to begin to find what contents the user hasn't received. Any hints, tips, or suggestions?
Edit: The output from the program starts off with a time (HH:MM:SS AM/PM), everything after has no pattern. The log file may span over days, so there might not be just one "02:00:00 PM" in the file, for example. I didn't write the program that is being logged, so there isn't a way for me to modify the format in which it prints it's output.
I think using a head request might get you started along the right path.
check this thread:
HTTP HEAD Request in Javascript/Ajax?
if you're using jquery, it's a simple type change in the ajax call:
$.ajax({url: "some url", type: "HEAD".....});
personally, I would check the file size & date modified against the previous response, and fetch the new data if it has been updated. I'm not sure if you can fetch only parts of a file via ajax, but I'm sure this can be accomplished via php pretty easily, possibly this thread may help:
How to read only 5 last line of the text file in PHP?
It depends how your program is made and how does it print your data, but you can use timestamps to reduce the amount of data. If you have some kind of IDs, you should probably use them insteam of timestamps.
Related
I have about 14,000 lines in a CSV file.
Each line is formatted like this:
Firstname Lastname,UUID
Firstname Lastname,UUID
I want to Post/CURL to a url like:
http://some.url/directory/file.php?name=Firstname%20Lastname&uuid=UUID
I have tried doing this a few ways, but I always run into an issue - what's a good way to do this, and I don't want to attack the URL by sending 14,000 requests at once.
I do not need any data back from each posting, I just need to send the data over to the PHP file so it can be analyzed.
NOTE: Each post means a mysql query will be analyzing the data, so I don't want to flood the server with 14,000 mysql requests at once either - more reason why I might want to throttle this.
I just have to do this one time ever.
Would something like pull data from csv file and submit each row to url with curl be a good place to start?
I have a file (usernames.txt) that has every username of my website members, when they submit their usernames, it will be saved in "usernames.txt", the problem is that there are a lot of users submitting there usernames every day, I want a php code or something that will delete the first (at the top) username every 30 seconds automatically (Even though more than 20 usernames are submitted every minute, the script I need will make the "usernames.txt" file smaller and that will make my server a bit faster. :)
It would be really great if someone has or knows the script I am talking about. :)
Thanks
I would also really, really encourage you to look at http://php.net/manual/en/refs.database.php as #elclanrs suggested. Really encourage you.
Failing that, and a very far second place suggestion, I'd even recommend trying to use serialize and unserialize, so you can read the data in and out of the file quicker. You can then read the data in as an array/object using unserialize, manipulate it, and put it back in the file using serialize.
Further failing that, you would have to read the file into memory (file_get_contents), loop through the lines, and remove the first line, and then write the entire file back out. Alternativly, read it in one line at a time (fopen), skip the first line, write the rest of the file back to a temporary file (fputs), and then swap the files around (rename).
Lastly, assuming you're on a *nix system, you could use something like:
exec("sed '1d' {$my_file_name}", $result);
file_put_contents($my_file_name, $result);
That all being said, you should really look at using a DB. If you don't want a standalone database, you can use sqlite, which will write the database to a local file:
$dbhandle = sqlite_open('db/test.db', 0666, $error);
I want to create a very simple site, but unfortunately my php skills are weak. Basically, when a user shows up, I want to have a page with text and a blinking cursor (I can probably figure the cursor part out myself, but feel free to suggest). When a user types, I want it to show the text as they type, and when they hit enter (or click something/whatever), the text just typed will be sent to a database and then the page will update with that new text, for anybody else to see. The cursor will then be blinking on the next line down. So basically it's like a really simple wiki, where anyone can add anything, but nobody can ever remove what has been typed before. No logging in or anything. Can someone suggest the best way to go about this? I assume it will require a php call to the database to display the initial page, then another php request to send data, then another php request to display the new page. I just don't know the details. Thanks so much!
Bonus question 1: How can the page be updated dynamically, so if A sends text while B is typing, B sees the text A sent on B's page immediately?
Bonus question 2: What sorts of issues might arise if this database grows extremely large (say, millions of words), and how might I address these up front? If necessary, I could show only a small chunk of the (text-only) database on any given page, then have pagination.
If you only have one page, you don't need a database. All you need to do is save a text file on the server (use fopen() and related functions) that only gets appended to. If you have multiple pages, then a simple id (INTEGER), filetext (LARGEBLOB). (Note largeblob has a limit of 2^32 bytes).
For the user's browser part, you'll need to use Javascript and AJAX to inform the server of any updates. Just get in touch with a PHP script that (1) accepts the input and (2) appends it to a file.
Bonus question 1: How can the page be updated dynamically, so if A sends text while B is typing, B sees the text A sent on B's page immediately?
Also use the AJAX call to fetch new content (e.g. if you assign line numbers, then the browser just tells the script the last line it read, and the script returns all new lines past that point).
I assume it will require a php call to the database to display the initial page, then another php request to send data, then another php request to display the new page.
Pretty much. But only send the last 50 lines or so of the file when the browser visits it. You don't want to crash the browser.
Bonus question 2: What sorts of issues might arise if this database grows extremely large (say, millions of words), and how might I address these up front? If necessary, I could show only a small chunk of the (text-only) database on any given page, then have pagination.
Think in terms of bytes, not words, and you'll likely run into performance issues. You could cap file sizes or split up the storage into multiple files at a certain size so you don't have to scan pass content that will rarely be fetched.
Real world problem: I'm generating a page dinamically. This page is an xml which is retrieved by the user (curl, file_get_contents or whatever can by made server side scripting).
Once the user make the request, he start waiting and I start retrieving a large set of data from the db and building an xml with them (using the php dom objects). Once I've done I fire the "print $document->saveXML()". It takes about 8 minutes to create this 40 megabytes document. Then as it is ready I serve the page/document. Now I have a user who has a 60 seconds connection timeout: he said I need to send the first octet each 60 seconds. How can I achieve such a thing?
Since it's useless to post a 23987452 lines code cause nobody is gonna read them, I'll explain the script which serves this page as real-very-pseudo-pseudo-code:
grab all the data from the db: an enormous set of rows
create a domdocument element
loop through each row and add a node element to the domdocument to contain a piece of data
call the $dom->saveXML() to get the document as a string
print the string so the user retrieve an xml document
1) I can't send real data since it is an xml document and it has to begin with "<?xml..." to not mess up the parser.`
2) The user can't deal with firewall/serverconfig
3) I can't deal with "buy a more powerful server"
4) I tried using an ob_start() at the top of the script and then at the beginning of each loop a "header("Transfer-Encoding: chunked"); ob_flush(); "
but nothing: nothing comes before the 8 minutes.
Help me guys!!
I would
Generate a random value
Start the XML generating script as a background process (see e.g. here)
Make the generating script write the XML into a file with the random value as the name when the script is done
Frequently poll for the existence of that empty file, e.g. using Ajax requests every 10 seconds, until it's there. Then fetch the XML from the file.
You send padding and still have it be valid XML. Trivial examples include whitespace in a lot of places, or comments. Once you've sent the XML declaration, you could start a comment, and keep sending padding:
<?xml version="1.0">
<!-- this comment to prevent timeouts:
30
60
90
⋮
or whatever, the exact data doesn't matter of course.
That's the easy solution. The better solution is to make that generation run in the background, and e.g., use AJAX to poll the server every 10s to check if its done. Or to implement an alternate notification method (e.g., email a URL when the the document is ready).
If this isn't a browser accessing, you may want a trivially simple API: Have one request to start generating the document, and another to fetch it. The one to fetch it may return "not ready yet" as e.g., a HTTP status code 500, 503, or 504. Then the script requesting should retry later. (For example, with curl, the --retry option will do this).
I've got a script in php that continually grows an array as it's results are updated. It executes for a very long time on purpose as it needs to filter a few million strings.
As it loops through results it prints out strings and fills up the page until the scroll bar is super tiny. Instead of printing out the strings, I want to just show the number of successful results dynamically as the php script continues. I did echo(count($array)); and found the number at 1,232,907... 1,233,192 ... 1,234,874 and so forth printed out on many lines.
So, how do I display this increasing php variable as a single growing number on my webpage with Javascript?
Have your PHP script store that number somewhere, then use AJAX to retrieve it every so often.
You need to find a way to interface with the process, to get the current state out of it. Your script needs to export the status periodically, e.g. by writing it to a database.
The easiest way is to write the status to a text file every so often and poll this text file periodically using AJAX.
You can use the Forever Frame technique. Basically, you have a main page containing an iframe. The iframe loads gradually, intermittently adding an additional script tag. Each script tag modifies the content of the parent page.
There is a complete guide available.
That said, there are many good reasons to consider doing more pre-computation (e.g. in a cron job) to avoid doing the actual work during the request.
This isn't what you're looking for (I'm as interested in an answer to this..), but a solution that I've found works is to keep track of the count server-side, and only print every 1000/5000/whatever number works best, rather than one-by-one.
I'd suggest that you have a PHP script that returns the value in JSON format. Then in another php page you can do an AJAX call to the page and fetch the JSON value and display it. Your AJAX call can be programmed to run perhaps every 5 seconds or so depending on how fast your numbers output. Iframe though easier, is a bit outdated.