I have a CSV file which is more than 5 MB. As an example like this,
id,song,album,track
200,"Best of 10","Best of 10","No where"
230,"Best of 10","Best of 10","Love"
4340,"Best of 10","Best of 10","Al Road"
I have plenty of records. I need to get a one record at a time. Assume I need to get the detail of the id 8999. My main question is Is there any method to get exact data record from CSV just like we query in mysql table.
Other than that I have following solutions in my mind to perform the task. What could be the best way.
Read the CSV and load in to array. Then search trough it and get data. (I have to get all records. The order is different that's why I face this issue. Other wise I could get record by record.)
Export CSV to MYSQL database and then query from that table
Read the CSV all the time without loading it to array
If I can search in CSV file in quick way it will be great. Appreciate your suggestions.
Thank you.
I'm not aware if there are any libraries that allow you to directly query a CSV file. If there are, that would be your best bet. If not, here's another solution.
If you want to get details of id 8999, it would be extremely memory inefficient to load in the entire file.
You could do something like this
Read in a line using fgets, explode on comma and check 0th element. If it is not the ID you want, discard and repeat.
Not familiar with PHP, but if this query is frequent, consider keeping the data in the memory and index them by id using a mapping data structure. (should also be Array in PHP)
Related
I need help parsing the following a CSV file in PHP, so I can insert the contents into a database.
I know I use file_get_contents() but after that I feel a bit lost.
What I'd like to store.
Collection1 - events.text & date
Collection2 - position & name.text & total
I'm not sure how best structure the data to insert into a database table.
"**collection1**"
"events.href","**events.text**","**date**","index","url"
"tur.com/events/classic.html","John Deere Classic","Thursday Jul 9
- Sunday Jul 12, 2015","1","tur.com/r.html"
"collection2"
"**position**","name.href","**name.text**","**total**","index","url"
"--","javascript:void(0);","Scott","--","2","tur.com/r.html"
"--","javascript:void(0);","Billy","--","3","tur.com/r.html"
"--","javascript:void(0);","Jon","--","4","tur.com/r.html"
"--","javascript:void(0);","Bill","--","5","tur.com/r.html"
"--","javascript:void(0);","Tim","--","6","tur.com/r.html"
"--","javascript:void(0);","Carlos","--","7","tur.com/r.html"
"--","javascript:void(0);","Robert","--","8","tur.com/r.html"
"--","javascript:void(0);","Rod","--","9","tur.com/r.html"
As per your previous question, I think this needs to be broken down into sections. As it stands it is rather too broad to answer.
Read the information using file_get_contents(). Make sure this works first, by echoing it to the console. (It sounded from your other question that you felt this would not work if the URL does not have a .csv suffix. It should work regardless of the file extension - try it. If it fails it may be dependent on cookies or JavaScript or some other problem).
Design and create your table structure in MySQL. It seems like you have two tables. They should both have a primary key. Are they related in some fashion? If so, perhaps one has a foreign key to the other one?
Explode your text file on the new line character and loop across the resulting array of lines.
If your CSV data has a title row in the first row position, delete that from your array.
For each line, read the elements of interest using PHP's build-in CSV parsing functions, and store them in variables.
Pass these variables to a custom function that saves the data.
For each save, you'll need to do an INSERT. I recommend using PDO here. Make sure you bind your parameters.
Where you get stuck on a specific problem, you can ask a new and focussed question. At present, the task is to break things down into discrete and researchable pieces.
One trick worth remembering is this shortcut to the PHP manual. If you do not know how fgetcsv works, for example, type php.net/fgetcsv into your browser address bar, and the PHP site will find the function for you. The documentation is excellent.
I am wondering what is the best way to tackle the following problem. I need to provide a client with the option to download various ranges of records from a database in CSV format. I was thinking of doing the following.
Provide user with a webform to determine what rows to pull from the DB.
Pull the appropriate records and create a large string from the return data.
Post the string to the script which creates the CSV for downloading.
Before I attempt the above I was wondering if there is a better way to do this?
Thanks ^_^
You can use MySQL's SELECT ... INTO OUTFILE to immediately obtain your results in a CSV. You then need only read the contents of that CSV file back to the user.
This is much safer than attempting to construct a CSV yourself (where you must consider how to properly escape field delimiters, etc); however, if you wanted to do it that way, I'd recommend using fputcsv(STDOUT, $results) to avoid common pitfalls.
i need some help with a project of mine. It is about a dvd database. In the moment i am planning to implement a csv data function to import dvds with all information from a file.
I will do this in three steps.
Step 1
- show data i want to import, building array
- import data, building session arrays
Step 2
- edit informations
Step 3
- showing result before update
- update data
so far it works but i have a problem with large files. the csv data has 20 columns (title, genre, plot etc.) and for each line in the csv there are some arrays i create to use it in the next steps.
When i have more about 500 lines the browser often collapse while importing. I get no response.
Anyway now i trying to do this as an ajax call process. The advantage is, that i can define how many procedures the system handle each call and the user can see that the system is still working, like an statusbar when down/uploading a file.
In the moment i try to find some usefull example illustrating how i can do this, but i could not find something useful till now.
Maybe you have some tipps or an example how this could work, saying processing 20 lines each call, building the array.
After i would like to use the same function to build the session arrays using in the next step and so on.
Some information:
i use fgetcsv() to read the rows from the file. i go through the rows and each column i have different querys like is the item id unique, the title exist, description exist etc.
So if one of these data is not entered i get an error which row and column the error occures.
I´d appreciate any help from you
use 'LOAD DATA INFILE' syntax. ive used it on files upwards of 500mb with 3mil rows and it takes seconds, not minutes.
http://dev.mysql.com/doc/refman/5.0/en/load-data.html
While this is not the direct answer you were looking for
500 lines shouldnt take too long to process, so.. heres another thought for you.
Create a temporary table with the right structure of fields
you can then extract from it using select statements the various unique entries for the plot, genre etc rather than making a bunch of arrays along the way
mysql import would be very fast of your data
You can then edit it as required, and finally insert into your final table the data you have from your temporary but now validated table.
In terms of doing it with ajax, you would have to do a repeating timed event to refresh the status, the problem is rather than 20 lines, it would need to be a specific time period, as your browser has no way to know, assuming the csv is uploaded and you can process it in 20 line chunks.
If you enter the csv in a big big textbox, you could work on by taking the first 20 lines, passing it the remainder to the next page etc, would strike me as potential mess.
So, while I know ive not answered your question directly, I hope I gave you food for thought as to alternative and possibly more practical alternatives
I am new to php and am asking for some coding help. I have little experience with php and have gone to the php.net site and read couple books to get some ideas on how to perform this task.
There seems to be many functions and I am confused on what would be the best fit. (i.e. fgetcsv, explode(), regex??) for extracting data in the file. THen I would need assistance printing/display this information in orderly fashion.
Here is what I need to do:
import, readin txt file that is
delimited (see sample)
The attributes are not always ordered and some records will have missing attributes.
Dynamically create a web table (html)
to present this data
Sample records:
attribute1=value;attribute2=value;attribute3=value;attribute4=value;
attribute1=value;attribute2=value;attribute4=value;
attribute1=value;attribute2=value;attribute3=value;
How do I go about this? What would be best practice for this? From my research it seems I would create an array? multidimensional? Thank you for your time and insight and i hope my question is clear.
Seems like homework, if so best to tag it as such.
You will want to look into file(), foreach() and explode() given that it is delimited by ;
The number of attributes should not matter if they are missing, but all depends on how you setup the display data. Given that they are missing though, you will need know what is the largest amount of attributes to setup the table correctly and not cause issues.
Best of luck!
i would first use the file() method, which will give you an array with each line as an element. Then a couple of explodes and loops to get through it all,first exploding on ';', then loop through each of these and explode on '='.
Alright, so I've just (almost) finished my first little php script. Basically all it does is handling forms. It writes whatever the user put in to the fields of the form to a text file, then I include that text file in the index of the little page I have set up.
So, currently it writes to the beginning of the text file (but doesn't overwrite previous data). But, several users wants this list to be alphabetically sorted instead. So what I want to do is make whatever they submit fall into the list in alphabetical order.
The thing here is also that all I use in the text file are divs. The list is basically 'divided' into 3 parts. 'Title', 'Link', and 'Poster'. All I have done is positioned them with css.
So, can I sort this list (the titles, in this case) alphabetically and still have the 'link' and 'poster' information assigned the way they already are, but just with the titles sorted?
It don't use databases at all on my site, so there is no database handling at all used in this script (mainly because I'm not experienced at all in this).
I would also suggest storing the data in the file as either XML or JSON. PHP can sort the records easily and the sorting will be preserved in the file when the data is read back in.
For example
file_put_contents("/tmp/datafile",json_encode($recordset));
and when reading the file back in
$recordset = json_decode(file_get_contents("/tmp/datafile"));
edit
but seriously if you have customers and are charging them for your experience and time, use a database, there are many out there (a few mentioned already)
MySQL
sqlite
PostgreSQL
Oracle
MSSQL
If you really don't want to use a database, even a one-file database (such as sqlite), you can group the three fields using a separator such as _ and sort this single-field list
If this is a small project for some friends or something and the file shouldn't ever have more than maybe a few hundred lines, the functions you're looking for are "file" and "usort".
If you make sure each row is on its' own line, you can load the lines into an array with "file". You can sort the array using a function to compare items with the "usort" function.