I have a csv file which contains 2000 data in one single column. The data format is like following:
name 100/1
name 100/2
name 105/6
...
So the general format is 'text integer_one/integer_two' Now I want it to store in a mysql database table with 2 columns. column_1 should have the data integer_one, and column_2 should have the data integer_two
How can I do this using php to push the data into mysql?
First start by extracting your values. If all your data is formated like in your example, you can use a simple explode.
$lines = explode(",", $data); // $data should be your file extracted into a string
foreach ($lines as $line) {
$temp = explode(" ", $line);
$name = $temp[0];
$temp_2 = explode("/", $temp[1]);
$integer_1 = $temp_2[0];
$integer_2 = $temp_2[1];
$query = "INSERT INTO table_two_columns SET name = '{$name}', integer_1 = '{$integer_1}', integer_2 = '{$integer_2}'";
}
Well that's my take on your problem.
Use ph function fgetcsv function to read a csv file (example is give in this page) then take each in loop and parse it using function explode() and create query from it and save in database
Take a look at preg_match, you should be able to get these values using a fairly trivial regular expression, like:
preg_match(/\w+\s*(\d+)\\\/(\d+)/, $row, $matches);
var_dump($matches);
To pull the data from a file take a look at fgets there are example of reading the file line-by-line to get the $row variable in the code above.
You can then write each line to the database using an INSERT statement.
Some things to consider:
How large is the file?
How often do you need to insert this data - is it a one-off?
These things will influence how you build your code; there are multiple ways to bulk insert data, and if this is just a one-off run then I would suggest maniuplating the text data using sed or similar, and then bulk inserting using the native methods of your database of choice (for mysql see LOAD DATA INFILE, or using the \. option in the client)
If the file is large and you need to insert things regularly, look at using prepared statements, and/or the multiple insert syntax:
INSERT INTO mytable (name, number, anothernumber) VALUES('paul', 1, 2),('bob',2,1)
have a look at fopen() and fread() or fgetcsv() php functions to handle the files, with them you should be able to loop through the lines of the cvs file, If you have control over your server configuration, I would suggest you to have a look into SplFileObject instead, because you can handle the files more efficient/elegantly, and object oriented. but require you to enable SPL. This last one is my recommendation, as you could read the file like this:
function random_name(){
$file_name = "names.csv";
if(is_readable($file_name)){
$file = new SplFileObject($file_name);
$file->seek($file->getSize());
$linesTotal = $file->key();
$file->seek(rand(1,$linesTotal-1));
$line = $file->fgetcsv();
return $line[4];
}
}
This was a sample script of mine to get a random name from a 1000 name cvs file.
As you loop and get the file information, you can then use explode(), this function separates your string by the separator you define, in your case it would be:
$array = explode("/", $myLine);
And you would access like this:
$array[0] or $array[1] to get the value
Related
I am writing a script in PHP to directly import data from a JSON file directly into a MySQL database. My sample data at the moment is pretty small. I have a few questions about it. I would like to do this without external packages. If you know or have a code sample I can look at that would be very helpful.
How can I make it work with very large data sets? Break it up in
chunks? Import it line by line? Use fopen?
If during it's run the script get interrupted for some reason, is
there a way to make sure it continues where it left of? How would you do this?
If I want to remove duplicates with the same emails should I do it
during the import or maybe after with a SQL query?
Would this work with XML and CSV as well?
DATA
[
{
"id":1,
"name":"Mike",
"email":"mike#businessweek.com"
},
{
"id":2,
"name":"Nick",
"email":"nick#businessweek.com"
}
]
$jsondata = file_get_contents('data.json');
$data = json_decode($jsondata, true);
foreach ($data as $row) {
$query = 'INSERT INTO `member` (`name`, `email`) VALUES (?, ?)';
$statement = $pdo->prepare($sql);
$statment->execute([$row['name'], $row['email']]);
}
This will probably be closed, but to give you some information:
The only way to break it up is if you had individual JSON objects per line, or individual JSON objects, or grab every X lines (5 in your example). Then you could just fopen and read it line by line. You could get from { to } (not recommended).
The only way that I can think of is to use transactions, and/or track the number of rows total and decrement each insert in a separate DB table.
If you import the entire file you can do it then, maybe with array_column to reindex on email which will remove duplicates. If not then it would have to be after with SQL.
Yes, you would do it the same way, just parsing the data is different. CSV would be fgetcsv or str_getcsv and XML would be Simple XML or something similar. After the data is in an array of arrays it will be the same inserting.
I would like to read a file, generally a text file, each record is starting with a with a specific code (filed name) in the line and ended by another specific code for a complete record. Each specific code is delimited by character ^ as its value in php into dump into sql database.
text file e.g.
001^UK2000009
008^S54/01/R/M/X,
009^Male
110^text1
200^text2
001^UK2000008
008^S54/012/R/M/X
009^Female
110^text1a
200^text2a
and so on...
This is similar to php constructor File_MARC
thanks in advance
First you have to read a file with file methods in php and than you can get a specific column name and it's value by below way
First read a single line from a file and than use a explode method to break that line into different elements with space delimitation.
$columns = explode(' ', $line_variable);
After generating columns I can see that each key values are delimited by ^ (cap) symbol so for that also we can use the explode method.
$newColumn =[];
foreach($columns as $column){
$splited = explode('^', $column);
$newColumn[][$splited[0]] = $splited[1];
}
print_r($newColumn);
This is just to give you an idea that how you can achieve your task but rest is completely dependent on you.
I have a CSV file with three columns:
Mary,150203_15:29:12,150203_16:10:12
John,150203_15:29:17,pending
Peter,150203_15:29:35,150203_15:49:35
Olga,150203_15:30:43,pending
...
..
.
Mary,150204_15:42:14,pending
Peter,150204_20:42:14,pending
Because there are many entries on that file all I want to do is
find the latest entry according to the Username and change the last value (from pending to date()).
In the example above lets say I want to change the latest Mary entry in the 3rd column from pending to date. Then replace the updated CSV file with the current one.
Any ideas on how to approach that?
Thank you
You can work with the file as one huge string and do a string replacement.
$data = file_get_contents('your_file.csv');
$data = str_replace('Joe,150203_16:21:43,pending','Joe,15203_16:21:43,15204_15:23:43',$data);
file_put_contents('your_file.csv', $data);
The comments below raise a concern of finding out what the latest date is for a name. That is also rather simple to do. Assuming you've loaded $data in, as above...
$matches = array(); // Just to point out that this is an array
preg_match_all("/Joe,(.*),/", $data, $matches);
Now, $matches[1] contains all the dates for Joe. Did you ONLY want the ones that are pending? No problem...
preg_match_all("/Joe,(.*),pending/", $data, $matches);
Now, $matches[1] only contains the pending dates. Which is the most recent?
rsort($mathes[1]);
Now, $matches[1][0] is the most recent date. So, you can do:
$data = str_replace('Joe,'.$matches[1][0].',pending','Joe,'.$matches[1][0].',15204_15:23:43',$data);
Is this the absolute most efficient way to do this? No. Is it impossibly hard? No. You should look into using a proper database, but it is possible to use csv files.
If moving the data to a DB is not a solution in your case, you could do the following:
Read the CSV file into PHP via fgetcsv(), line by line, into an
array
Sort the array based on your criteria and get the latest entry
Update the entry
Write the entries back into the file via fputcsv()
I actually found a much simpler solution to this. Was the simplest one I could come up with.
$line = $_SESSION[username].",".$_SESSION[LOGIN_TIME].",".'pending'."\r\n";
$newline = $_SESSION[username].","$_SESSION[LOGIN_TIME].",".date("ymd_H:i:s"). "\r\n";
$data = file_get_contents('login.log');
$data = str_replace($line,$newline,$data);
file_put_contents('login.log', $data);
So there is no need for scanning for the latest entry. I just store the previous entry values and replace them with the new ones.
Thank you kainaw and BigScar
My situation is this: the first ~500kb of a CSV file is encoded as a string and sent as a request to my server, running Laravel. How can I make Laravel parse that string (which is, by itself, not a complete CSV file) to get the column headers and first few rows of data?
I've looked into Goodby/CSV, but it looks as though that can only take a file as input, and not a string. Is there a CSV interpreter plugin that can handle this scenario, or should I expect to have to write a parser of my own?
Edit:
Looks like I can do something like this:
$Data = str_getcsv($CsvString, "\n"); //parse the rows
foreach($Data as &$Row) $Row = str_getcsv($Row, ","); //parse the items in rows
I was overthinking it, of course.
I've looked into Goodby/CSV, but it looks as though that can only take
a file as input, and not a string
If you have fopen wrappers enabled, you can take advantage of the fact that GoodbyCSV uses the SplFileObject class internally, and pass the string as a data protocol:
$lexer->parse('data://text/plain;base64,' . base64_encode($csv_string), $interpreter);
I found this useful when I needed to handle large files as well as simple strings.
http://pastebin.com/raw.php?i=7NTGXU5R
I have about a hundred of those listened in the same file
I tried working on a php solution, but I wasn't sure how to parse the space, I could only find fgetcsv which does commas
What direction should I head to in order to make sense
I remember some C++ from years ago, I was thinking I do something like a getline, then store the line (or row in our case) into an array
Once that is done, just write a bunch of if statements to go through each line and classify the first element (column) in each array to be the designated 'header'.
Tasks like that always boil down to a large amount of custom string-munching code. Your best weapons of choice will be regular expressions. Forget about fgetcsv if files look like your file does.
The basic logic might look something like this:
Fetch all rows of the file via the file function.
Save each table area to an own array containing the rows:
foreach ($filelines as $line)
{
$lefreportlines[] = trim(substr($line, 0, 93));
$middlereportlines[] = trim(substr($line, 67, 135));
...
}
When you're done, start processing each report as it deserves. For example, the leftmost report might simply be parsed with preg_split('/\s+/', $line);
Either way, you'll have a lot of work to do. Good luck!