i have a client requirement with resume-able file import.i will tell the use case file imported is csv that corresponds to 3 mysql-table datas. during upload if any kind of warnings occure like name too long,description too long it should stop execution and show the warning to client.client have 2 options ignore and re upload if he ignores then file upload will resume from the point where error happens and other option is simple just re upload the file.how should i continue the script execution after showing error msg ? how should i track no of csv lines get completed?
now lets explain my plan. i an using jquery ajax in first place.after saving each line of code save the no of lines executed in a session variable.if the user ignore the warning. then i will reupload csv and wait until the line no matches the one in session variable.is this the elegant way of doing things? reason why i put this question is each time user ignore the error msg according to my logic he/she needs to wait until the line stored in session reached. pls explain with an example if their is a better solution to this problem. thanks in advance.
A possible workflow
Analyze each line of the CSV file and load it into an empty table specially created for this purpose. The table should have the columns to store only the columns you import from the CSV file. It can also contain additional columns to store data that is computed during the import; f.e. if the processing and validation of the input data requires lookups
into the data that already exists in the database, the values found should be also stored in the import table to avoid running the same queries again.
Ignore the lines from the CSV file that cannot be imported but keep track of their information (content, failure reason etc). You'll need the info to display it to the user.
If there are no errors in the CSV file then move the (validated) data from the special table into the destination tables and call it a success.
If there are errors show them to the user.
If the user decides to reload the CSV then start over with step 1.
If the user decides to ignore the errors then resume with step 3.
Advantages
There is no need to upload the CSV file again.
There is no need to keep track of the progress.
Disadvantages
A lot of work is done then discard if the file is large, the error is at the beginning of CSV file and the user decides to upload it again. This can be improved in the algorithm by using an extra database table that contains the raw lines of the CSV file and an extra step (step 0): load the CSV file into it then use this table instead of the CSV file as input data for step 1. When a line is valid remove it from the CSV table; when a line is not valid then skip to step 4. When the user decides to ignore the error then resume with step 1.
Related
I am currently working on a web page that should do the following. As a little background, we are working on uploading a bunch of data that has a decent level of user-errors in the cell formatting to a database.
The data is stored in Excel spreadsheets. The spreadsheets are formatted in the same way, (99.9% of the time) but occasionally there are some wonky values within the cells themselves. Here is my goal: I would like the user to be able to
1) Provide the excel file to the page
2) Parse the file with PHP, then send the newly extracted data to a table on the page with editable cells
3) The user can see what came out of their file, then correct it for any slight errors before choosing to send the extracted data to the database.
I can do steps 2 and 3 using PHPExcel with a test file stored on the server, but I am having trouble getting from step 1 to step 2. My vision is to have the user specify the Excel spreadsheet and do the parsing without leaving the current page and losing the data in the fields. I would really like to keep the page from changing, as there are additional text fields on the page and I actually have to parse TWO spreadsheets before the data is taken from all of the fields and sent off to the database.
I guess I'm wondering if there's a way to use a client-provided file in PHP without doing a POST to upload. OR if there's a way to upload the file and have it parsed without leaving the page. I'm new to web development, so excuse me if those last few sentences made absolutely no sense.
Any thoughts?
PHP cannot do a single thing to the file until it is uploaded to the server. You could put a file upload form in an iframe and do some AJAX-y thing as described here.
As for handling PHP file uploads, the docs have a whole section.
When I do an import of data from a file, such as csv or text file, I would like to display on a summary page （table view) about what is going to be imported to the database, and user can select or deselect what will be imported into the database. I would like to find out what is the best way of temporary storing these data to be displayed onto the summary page?
To give a clearer understanding about what i am talking about, for example:
I have a csv file that holds a list of products.
I would do an importation to read the csv file and display onto a summary page with
table view and checkboxes to select/deselect some of the data.
After the selection, I would click the confirm button to store those checked data
So between the process of getting the csv data to the summary page for display, where should I store the data, in the database as a tbl_temp data and clean off when done, or just read directly from the csv file?
Thanks.
I would think an array would do the trick unless you think your environment is really unstable or, for user interface reasons, then just store to a temp file. Temp db table seems a bit much.
I would just display it in the browser, and stored? It's already stored, the uploaded CSV file!
Unless you don't run into an actual problem with that approach, why care?
Just use HTML to display the data read from the file in a web browser. Use standard HTML Form and form controls. Use PHP scripting logic to control what happens when the form is submitted. You could have the submit button hand off to a service or some other php file that could handle writing it to the db.
Why bother writing to a temp table, to just read back out and display in a form to be then written back to a perm. table.
If this were a process that you would be performing a lot, i might give it a little more consideration.
I am wondering what the best workflow would be to handle this process.
Basic steps are.
The user selects a csv file and uploads it.
The csv file is then checked against a set of rules.
If the csv file is invalid
The user is shown the rows that are invalid.
The user is given the choice to terminate the upload or, strip the invalid lines.
If the csv file is valid, or strip is clicked
The user is shown a screen to choose the filename.
If the filename is already taken the user is given the choice to
a) rename the file (to a name of their choosing)
b) replace existing file.
c) rename the file to filename_1 etc
When the name is chosen a table is created in the database called (csv_filename);
Then data from the csv is entered into the table.
The file is deleted.
The user is taken to a page showing the file data (from the table)
My issue is,
This is all run through ajax.
How do I handle reporting what file we are dealing with?
I dont want to pass back the filename in an ajax response as that is too easy to tamper with.
I dont want to create a table to hold the filepath and pass back an id, as it seems to be a waste to have a table for just this.
There are some issues with this, when the file is uploaded in the first step. Its done, its got a file name, and it can't be terminated because its already there. It has to be, or how will you analyze it?
From the users perspective you can make it look like thats how its working, which maybe is what you meant.
Anyways, to report what file you're dealing with, store it in a session variable.
I have a requirement in which, i need to upload an excel file into SQL Server database. It is working well till this point.
But sometimes i am getting excel file in such a way, that the first 2 rows and columns are empty rows. That is not even fixed every time. so the format of the table after upload is not as expected. It is assigning F1, F2 as column names for the table in SQL Server.
It varies from file to file. I want to program it in such away so that the user can enter the row number and the column number from where the actual data is starting. So that while upload it should from line 3 and column 2.
I don't know how to specify that row and column while uploading.
Please help me to solve the same.
You could use http://phpexcel.codeplex.com/
You can use this to check if and where the data you expect is located in the excel.
If not you could try to find the first column/row with data and work from there.
First of all you can ask your user before starting upload (in form with input type="file") and after user uploads file and you show him an parsed excel data (by your script)..
To more concrete answer you should at least say, by what do you parse excel files and what excel format do you use...
I'm creating an online game in PHP where users can create playable characters. Each character can have a user-uploaded portrait. A player can simultaneously have multiple characters, and the pictures for them can be changed anytime. Naturally, the pictures have to be resized and re-compressed to avoid huge files. Here's my problem:
When the player changes his data (among it the picture), and then hits "save", server side validation kicks in. It checks for things like non-unique character names, empty mandatory fields, etc. If any errors are found, they are displayed. In this case the form should be pre-populated with the data the player entered, so he only has to change the faulty bit, not re-type everything. But how do you save the picture in such a "temporary" state?
I cannot pre-populate the file upload field, the browsers don't allow that. If I save it in a temporary file, the picture then has to be cleaned up at some point, because the player can simply close his browser and abort the whole process. When should that be? And what file name should I choose for the temporary file? If the player opens the same character to edit in two browser tabs, they should not conflict (each of them should have their own copy).
How would you solve this problem?
I'd save the file to a temporary location and store both a unique identifier as well as the current timestamp in the filename. Then put the filename in the user's session. When a user has successfully created or updated their account, you save the image file to its permanent location and remove the temporary file. You can run a cron process to scan the temporary directory and check the timestamps, deleting any files older than your expiration (an hour perhaps).
If you're unable to run a cron job, you could always just launch the directory clean-up each time you have a successful create/update validation. This might be a bit more inefficient (extra directory reads and possibly file operations for every successful submission) but unless you're dealing with a lot of traffic, you probably won't even notice.
Create a table to hold references to images.
When a file is uploaded, if it's a valid image, do all your resizing, etc, and create a record in the table that points at the file.
Pass the id of the reference record around with the form data. Display the image when you redisplay the form, so the user knows they don't have to re-upload.
When you finally accept the new character object, set avatar_id or whatever.
Run a regular cron-job to cull orphaned image records (deleting the files on disk as well).
You could always populate a disabled text box to hold the name of the picture - it won't populate the browse input field, but is better than nothing. For editing, to avoid conflicts you could create some a "modifing" column for each user's characters, and on a character editing request change the value to true. When the user saves the character, set it back to false. For each edit request, grant it only when the "modifing" is false.
I'd recommend either updating the image immediately, regardless of error in the form, or separating the image updating to a separate form. That way you'll get rid of two problems without complex state machines and cleaning up.