I've got a web server that accepts a file from the user. It is then processed and the user gets a chance to modify it. The file format is not comfortable for the user to operate with, so my PHP code parses the file and creates a form that would let the user change the file contents. The form is then submitted and a new, modified file is created. In the worst case scenario, the file provided can have up to 160 input fields per row and up to 500 rows which results in 80k of input fields. Even though this sounds horrible, it has to be like that (So all 500 lines have to appear on the same page). The problem arises because of PHP's max_input_vars which is set to 1000. There is no way I can modify it. Therefore I would like to know if there is a way to bypass that restriction or if there is any other, better way of implementing this.
Related
I am working on a website in which it would be useful to to allow a user the option of downloading the content of a file, even when it's going to be updated by another user at the same time or later.
My problem is that the solution I've tried so far allows downloading, but will disrupt any later updating of the file. I don't think I can represent the code relating to the updating concisely (it is spread over multiple files), except that it's through AJAXing the data (which I'm not sure why it would cause this problem). In case it's relevant, this is a file which gets updated multiple times.
When I use fireftp I can download the file without disrupting this process, which makes me optimistic there's a PhP solution. I am currently downloading the data by Ajaxing the file contents to the page the "downloading user" is on. The code for this (within php) is:
$file_contents = file_get_contents ($_POST['file'])); //file address comes through Ajax POST request.
echo ($file_contents); //to access the content client side
Is there another way to access the text/content within a file without any unintended consequences on other server processing of it?
I'm looking for a way to send a user a regular file (mp3s or pictures), and keeping count of how many times this file was accessed without going through an HTML/PHP page.
For example, the user will point his browser to bla.com/file.mp3 and start downloading it, while a server-side script will do something like saving data to a database.
Any idea where should I get started?
Thanks!
You will need to go through a php script, what you could do is rewrite the extensions you want to track, preferably at the folder level, to a php script which then does the calculations you need and serves the file to the user.
For Example:
If you want to track the /downloads/ folder you would create a rewrite on your webserver to rewrite all or just specific extensions to a php file we'll call proxy.php for this example.
An example uri would be proxy.php?file=file.mp3 the proxy.php script sanitizes the file parameter, checks if the user has permission to download if applicable, checks if the file exists, serves the file to the client and perform any operations needed on the backend like database updates etc..
Do you mean that you don't want your users to be presented with a specific page and interrupt their flow? If you do, you can still use a PHP page using the following steps. (I'm not up to date with PHP so it'll be pseudo-code, but you'll get the idea)
Provide links to your file as (for example) http://example.com/trackedDownloader.php?id=someUniqueIdentifer
In the tracedDownloader.php file, determine the real location on the server that relates to the unique id (e.g. 12345 could map to /uploadedFiles/AnExample.mp3)
Set an appropriate content type header in your output.
Log the request to your database.
Return the contents of the file directly as page output.
You would need to scan log files. Regardless you most likely would want to store counters in a database?
There is a great solution in serving static files using PHP:
https://tn123.org/mod_xsendfile/
Let me describe what I've made ar first:
I have to import large ammount of data from different xml's to my database and because it last a lot I had to put a progress bar and I did it like this: I split the whole import into tiny little AJAX requests and I import little data at a time (when an ajax request completes the progress bar increases a bit). This whole idea is great but the data just keeps getting bigger and bigger and I can't optimize the code anymore (it's as optimized as it gets).
The problem is that everytime I do a AJAX call I lose a lot of time with things specific to the framework (model initializations and stuff), with the browser handling the url and so on. So I was wondering if I could use the flush function from php.
But I've been reading that the flush function doesn't work great on all browsers (which is weird cause it's a server-side function). If I would use the flush function I would just write <script>increase_progressbar</script> or whatever I want and I could do it.
So, any opinions on the flush function? I've been testing it on little scripts but I want to know if someone really used it with big scripts. Also, I can listen to any other suggestion of doing what I want to do :)
I wont give you direct advise, but I will tell you how I did it in one of my projects. In my case I need to upload an Excel files and then parse them. The data exceeding 3000 rows and I had to check all columns of each row for some data. When I parse it directly after the upload, the parser often crashes somewhere and it was really not safe.
So how I did it? The upload process has been split in 2 parts:
Upload physically the file (regular upload field and submit). When the button is clicked some CSS and JS "magic" hide the form and one nice loading bar appears on the screen. When the upload has been done the page just refreshes and the form appear again for the next file
Start parsing the data on the background using php-cli as #Dragon suggest with exec().
In the database I had a table which stores information about the files and there is a boolean field called "parsed". When the parser finishes the job, the last task is to update that field to true.
So here is the whole process from user point of view:
select a file and upload it.
wait until the file has been uploaded on the server. Till then a message and loading bar appear indicating that something is working. The upload form has been hidden with CSS and JS, so preventing user to upload another file.
When it's over the page has been refreshed (because I did normal _POST submit) the form appear on the screen again as well as a list of recently uploaded files (this I've stored this in the session).
In each of the nodes of that list I had an indicator (an icon). In the first time it's a spinner (ajax spinning wheel).
On a regular basis (30 sec or 1 min) I've checked the file table through Ajax call and reading the parsed field. If the background process has been over, the field was set to true and with some JS and CSS I've changed the icon to "Done". Otherwise the spinner remain.
In my project I doesn't have requirement to show extra details about the imports, but you can always go wild with other extra data.
Hope this help you with your project.
I am wondering what the best workflow would be to handle this process.
Basic steps are.
The user selects a csv file and uploads it.
The csv file is then checked against a set of rules.
If the csv file is invalid
The user is shown the rows that are invalid.
The user is given the choice to terminate the upload or, strip the invalid lines.
If the csv file is valid, or strip is clicked
The user is shown a screen to choose the filename.
If the filename is already taken the user is given the choice to
a) rename the file (to a name of their choosing)
b) replace existing file.
c) rename the file to filename_1 etc
When the name is chosen a table is created in the database called (csv_filename);
Then data from the csv is entered into the table.
The file is deleted.
The user is taken to a page showing the file data (from the table)
My issue is,
This is all run through ajax.
How do I handle reporting what file we are dealing with?
I dont want to pass back the filename in an ajax response as that is too easy to tamper with.
I dont want to create a table to hold the filepath and pass back an id, as it seems to be a waste to have a table for just this.
There are some issues with this, when the file is uploaded in the first step. Its done, its got a file name, and it can't be terminated because its already there. It has to be, or how will you analyze it?
From the users perspective you can make it look like thats how its working, which maybe is what you meant.
Anyways, to report what file you're dealing with, store it in a session variable.
I'm creating an online game in PHP where users can create playable characters. Each character can have a user-uploaded portrait. A player can simultaneously have multiple characters, and the pictures for them can be changed anytime. Naturally, the pictures have to be resized and re-compressed to avoid huge files. Here's my problem:
When the player changes his data (among it the picture), and then hits "save", server side validation kicks in. It checks for things like non-unique character names, empty mandatory fields, etc. If any errors are found, they are displayed. In this case the form should be pre-populated with the data the player entered, so he only has to change the faulty bit, not re-type everything. But how do you save the picture in such a "temporary" state?
I cannot pre-populate the file upload field, the browsers don't allow that. If I save it in a temporary file, the picture then has to be cleaned up at some point, because the player can simply close his browser and abort the whole process. When should that be? And what file name should I choose for the temporary file? If the player opens the same character to edit in two browser tabs, they should not conflict (each of them should have their own copy).
How would you solve this problem?
I'd save the file to a temporary location and store both a unique identifier as well as the current timestamp in the filename. Then put the filename in the user's session. When a user has successfully created or updated their account, you save the image file to its permanent location and remove the temporary file. You can run a cron process to scan the temporary directory and check the timestamps, deleting any files older than your expiration (an hour perhaps).
If you're unable to run a cron job, you could always just launch the directory clean-up each time you have a successful create/update validation. This might be a bit more inefficient (extra directory reads and possibly file operations for every successful submission) but unless you're dealing with a lot of traffic, you probably won't even notice.
Create a table to hold references to images.
When a file is uploaded, if it's a valid image, do all your resizing, etc, and create a record in the table that points at the file.
Pass the id of the reference record around with the form data. Display the image when you redisplay the form, so the user knows they don't have to re-upload.
When you finally accept the new character object, set avatar_id or whatever.
Run a regular cron-job to cull orphaned image records (deleting the files on disk as well).
You could always populate a disabled text box to hold the name of the picture - it won't populate the browse input field, but is better than nothing. For editing, to avoid conflicts you could create some a "modifing" column for each user's characters, and on a character editing request change the value to true. When the user saves the character, set it back to false. For each edit request, grant it only when the "modifing" is false.
I'd recommend either updating the image immediately, regardless of error in the form, or separating the image updating to a separate form. That way you'll get rid of two problems without complex state machines and cleaning up.