Import a "|" delimited text file into MySQL DB using PHP? - php

I'm trying to load a text file locally into a MySQL database using PHP. The data file looks like:
Device Name|DE:VI:CE:MAC:AD:DR:ES:SS|1376493754086|1376493754086
There are many lines of data formatted exactly as above.
I'm trying to load this data into a pre-established table like this:
$file = #fopen("LOCAL_data_file.txt","r");
$values = '';
while(!feof($file))
{
$con=mysqli_connect("127.0.0.1","root","password","my_db");
if(mysqli_connect_errno()){
echo "Error connecting to my_db: " . mysqli_connect_error();
}
$buffer = fgets($file, 4096);
list($a,$b,$c,$d)=explode("|",$buffer);
/*echo $a;
echo $b;
echo $c;
echo $d;*/
$ins="INSERT INTO 'tablet3' (Name, Address, StartTime, EndTime) VALUES
($a,$b,$c,$d)";
mysqli_query($con,$ins);
}
The data gets read from the file fine, because If I decomment the echoes of the list variables, it echos all of the data in the table. If I don't run the insert query, the table gets created with proper columns of "Name | Address | StartTime | Endtime" but with the insert query, the code returns a table with null data. I simply cannot figure out what I'm doing wrong.

I recreated it and this is the query I got to work
This is assuming the database structure of:
Name(varchar or text) Address(varchar or text) StartTime(int) EndTime(int)
Query:
$ins="INSERT INTO `tablet3` (Name, Address, StartTime, EndTime) VALUES
('$a','$b',$c,$d)";
Take note of the quotes around $a and $b. Also the tick marks around tablet3.
P.S. You don't really need anything around a table name.

Related

MySQL - Why is LOAD DATA INFILE replacing data in columns I didn't specify with NULL?

I'm trying to take a TSV file and 'POST'ed inputs and load the TSV file's contents into a DB table, replacing any existing data for specified columns. The TSV may contain any number of columns and rows and the 1st row specifies the columns that are supposed to be modified.
My problem concerns data in columns that ARE NOT supposed to be modified when running the code-generated LOAD DATA INFILE ... REPLACE INTO TABLE ... MySQL statement. When I run my code (see below), data of columns that are NOT specified in $columnsText (which is generated from the 1st row of the TSV file) end-up getting set to NULL or their default value. On the other hand, data of columns that ARE specified in $columnsText have their contents replaced just as intended.
An example of the MySQL statement that is generated by my code and is working as described above is:
LOAD DATA INFILE 'C:\\MyProject\\public\\1459772537-cities7.tsv' REPLACE INTO TABLE cities FIELDS TERMINATED BY ' ' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"' LINES TERMINATED BY ' ' IGNORE 1 LINES (id,UNLOCODE,name_english,UN_subdiv) -- for all TSV-file mentioned rows, this statement will update the mentioned columns (id,UNLOCODE,name_english,UN_subdiv) correctly, but then all unmentioned columns for that row will be set to NULL!
How do I modify this code to keep the data of unspecified columns from being set to their default/NULL values? Or more simply, getting to the root of the problem, how do I fix the MySQL statement that is being generated to achieve my objective?
I'm using PHP with Laravel.
// Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
echo "POST has file <br>";
$file = Input::file('file');
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "location: ".$pathName."<br>";
// Determine whether to use IGNORE OR REPLACE in MySQL query.
if (isset($_POST['replace']) && $_POST['replace'] == true){
$ignoreOrReplace = "REPLACE";
}
else {$ignoreOrReplace = "IGNORE";}
echo "ignore or replace: ".$ignoreOrReplace."<br>";
// Determine columns to insert in DB, based on values of input file's 1st row.
$columnsText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columns = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columns as $column){
if ($firstIteration){$firstIteration=false;}
else {$columnsText .= ",";}
$columnsText .= $column;
}
echo "DB columns to load: ".$columnsText;
fclose($handle);
}
$query = sprintf(
"LOAD DATA INFILE '%s' %s INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$ignoreOrReplace,$_POST['mytable'],$columnsText
);
echo "<br>Here's the query: ".$query."<br>";
echo "<br><br> Database update should be complete!<br><br>";
echo 'Return to Home Page<br>';
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // I forgot what this does.
return DB::connection()->getpdo()->exec($query);
The documentation states:
If you specify REPLACE, input rows replace existing rows. In other words, rows that have the same value for a primary key or unique index as an existing row. See Section 13.2.8, “REPLACE Syntax”.
REPLACE is not UPDATE. REPLACE is a MySQL extension to the SQL that first deletes the row if it exists, then it inserts the new one.
On INSERT, MySQL uses the default values for the fields that are not provided in the query. These fields probably default to NULL in your case.
There is no way to update the existing rows using LOAD DATA INFILE.
I suggest you to create a working table and use it only for the purpose of loading data into it as follows:
TRUNCATE it before using it.
LOAD DATA INFILE in it.
Join it against the table you want to update and use UPDATE on the join to copy the fields you need from the working table to the final table.
Use INSERT ... SELECT to get from the join the rows that are not in the final table and insert them.
TRUNCATE it.
Don't delete the table after it is used, you'll need it again next time. The last step aims to keep its disk usage at a minimum; the table definition doesn't take much space.
The answer provided by axiac is broadly-speaking the correct answer.
In case it is useful to anyone, I have included below the specific code (PHP/Laravel/MySQL) that solved my problem. I can't necessarily say this is the most efficient way to solve this problem, but it it working! :)
// (1) setup
DB::connection()->disableQueryLog();
// (2) Get file, put it in a folder on the server.
if (Input::hasFile('file')) {
$file = Input::file('file');
}
else {
echo "<br>Input file not found! Please review inputed information.<br>";
return null;
}
$name = time() . '-' . $file->getClientOriginalName();
$path = public_path();
$file->move($path, $name);
$pathName= $path .'\\'.$name;
echo "Input file location: ".$pathName."<br>";
// (3) Determine main table and staging table.
$mainTable = $_POST['mytable'];
$stagingTable = $_POST['mytable'].'_staging'; // All staging tables are named: 'standardtable_staging'.
// (4) Determine destination DB table's columns and columns to be inserted into that table (based on values of input file's 1st row).
$columnsMain = Schema::getColumnListing($mainTable);
$columnsInput = [];
$columnsInputText = "";
if (($handle = fopen("$pathName", "r")) !== FALSE) { //"r" parameter = read-only, w file-pointer at start of file.
$columnsInput = fgetcsv($handle,0,"\t"); // makes an array of the column names that are in the 1st row of TSV file.
$firstIteration = true;
foreach ($columnsInput as $columnInput){
if ($firstIteration){$firstIteration=false;}
else {$columnsInputText .= ",";}
$columnsInputText .= $columnInput;
}
echo "<br>DB columns to load: ".$columnsInputText."<br>";
fclose($handle);
}
// (5) Create a new empty staging table.
$statement = "DROP TABLE IF EXISTS ".$stagingTable; // we drop rather than truncate b/c we want to re-determine columns.
DB::connection()->getpdo()->exec($statement);
$statement = "CREATE TABLE ".$stagingTable." LIKE ".$mainTable;
DB::connection()->getpdo()->exec($statement);
// (6) The staging table only needs to have columns that exist in the TSV file, so let's minimize its columns.
$columnsToDrop = [];
foreach ($columnsMain as $columnMain){
if (! in_array($columnMain,$columnsInput)){
array_push($columnsToDrop,$columnMain);
}
}
if (count($columnsToDrop) > 0){
Schema::table($stagingTable, function($t) use ($columnsToDrop) {$t->dropColumn($columnsToDrop);});
}
// (7) Load data to the staging table.
$statement = sprintf(
"LOAD DATA INFILE '%s' INTO TABLE %s
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY '\"'
ESCAPED BY '\"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES (%s)",
addslashes($pathName),$stagingTable,$columnsInputText
);
echo "<br>Here's the MySQL staging statement: <br>".$statement."<br>";
DB::connection()->getpdo()->exec("SET sql_mode ='';"); // don't actually recall why I put this here.
DB::connection()->getpdo()->exec($statement);
// (8) 'INSERT...ON DUPLICATE KEY UPDATE' is used here to get data from staging table to the actually-used table.
// Note: Any new columns in the staging table MUST already be defined in the main table.
$statement = sprintf("INSERT INTO %s (%s) SELECT * FROM %s ON DUPLICATE KEY UPDATE ", $mainTable,$columnsInputText,$stagingTable);
$firstClause = true;
foreach ($columnsInput as $columnInput) {
if (strtoupper($columnInput) != "ID"){
if ($firstClause){$firstClause=false;}
else {$statement .= ", ";}
$clause = $mainTable.".".$columnInput." = IF (".$stagingTable.".".$columnInput." <=> NULL,".
$mainTable.".".$columnInput.",".
$stagingTable.".".$columnInput.")";
$statement .= $clause;
}
}
echo "<br>Here's the staging-to-actual-table statement:<br>".$statement."<br>";
DB::connection()->getpdo()->exec($statement);
echo "<br>New information added to database!<br>";

Adding commas at the end of each line when importing CSV

I am quite new to PHP and need your help for an issue. I will explain my problem in the following steps:
1- I have a CSV file and I would like to import it to MySQL database with PHP.
2- When I add commas at the end of each lines in the CSV file, I could import it by using IMPORT option in Mysql db. I can not import it without adding commas at the end of lines.
3- I want to add a new line into my PHP code which will add commas at the end of each record/line and will enable the data imported without any error.
Here is my code:
<?php
//read file
$csvfile=file_get_contents("/samba/import/Tbl_HRList.csv");
$lines = explode(PHP_EOL, $csvfile);
$array = array();
foreach ($lines as $line) {
$field = str_getcsv($line);
if ($field[0] != ''){
$sql="INSERT INTO HR_Tbl (Employee_ID, First Name, Prefix, Surname, Location, Organizational Code, Organizational Unit, Team, Team Code, Function, Function code, T24 Department Code, Date in service (GBI), Company email, End Date Contract, End Date Systems, Temp. Leave Date, Temp. Leave End, Temp. Leave Code)
VALUES
('$field[0]','$field[1]','$field[2]','$field[3]','$field[4]','$field[5]','$field[6]','$field[7]','$field[8]','$field[9]','$field[10]','$field[11]','$field[12]','$field[13]','$field[14]','$field[15]','$field[16]','$field[17]','$field[18])";
}
//insert record to database
if ($conn->query($sql) === TRUE) {
echo "New record created successfully". PHP_EOL;
} else {
echo "Error: " . $sql . "<br>" . $conn->error;
}
}
else
{
continue;
}
}
$conn->close();
?>
Thanks in advance
Alex
Use Load Data Infile and completely abandon the above idea.
You can have csv lines terminate with \n or \r\n
You can have row one of that import file either have or not have column names.
if you have column names, you skip the first row.
If you have column names on row1, you can have, say, 12 columns, and just use 4 if you want.
It is a very flexible construct. Check it out.
https://dev.mysql.com/doc/refman/5.1/en/load-data.html
LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;

PHP CSV Import Problems

So I'm trying to make it so that I can update a MySQL database by importing a CSV file, only problem is I am seeing some of my data has commas, which is causing the data to be imported into the wrong tables. Here's my existing import code.
if ($_FILES[csv][size] > 0) {
//get the csv file
$file = $_FILES[csv][tmp_name];
$handle = fopen($file,"r");
//loop through the csv file and insert into database
do {
if ($data[0]) {
mysql_query("INSERT INTO songdb (artist, title) VALUES
(
'".addslashes($data[0])."',
'".addslashes($data[1])."'
)
") or die (mysql_error());
}
} while ($data = fgetcsv($handle,1000,",","'"));
//
//redirect
header('Location: import.php?success=1'); die;
}
Is there a way I can set it to ignore the commas, quotes and apostrophes in the CSV file?
I would also let to set it to ignore the first line in the csv, seeing as how it's just column information. If that is at all possible.
** EDIT **
For example if the CSV contains data such as "last name, first name", or "User's Data", these are literally just examples of the data that's actually in there. The data is imported each month and we've just noticed this issue.
Sample Data:
Column 1, Column 2
Item 1, Description
Item 2, Description
Item, 3, Description
Item, 4, Description
"Item 5", Description
"Item, 6", Description
Above is the sample data that was requested.
You might want to use MySQL's built-in LOAD DATA INFILE statement which not only will work faster, but will let you use the clause FIELDS OPTIONALLY ENCLOSED BY '"' to work with that kind of files.
So your query will be something like that:
mysql_query(<<<SQL
LOAD DATA LOCAL INFILE '{$_FILES['csv']['tmp_name']}'
INTO TABLE songdb
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\\n'
IGNORE LINES 1 (artist, title)
SQL
) or die(mysql_error());
If your data is dirty, the easiest way to handle this will be to clean it up manually, and either use data entry forms that strip out bad characters and/or escape the input data, or tell the users who are generating this data to stop putting commas in fields.
Your example has inconsistent column count and inconsistent fields due to lack of escaping input in whatever they used to generate this data.
That said, you could do some advanced logic to igore any comma after Item but before a space or digit, using regular expressions, but that is getting kind of ridiculous and depending on the number of rows, it may be easier to clean it up manually before importing.
In terms of skipping the header row, you can do this:
if ($_FILES[csv][size] > 0) {
//get the csv file
$file = $_FILES[csv][tmp_name];
$handle = fopen($file,"r");
$firstRow = false;
//loop through the csv file and insert into database
do {
if ($data[0]) {
// skip header row
if($firstRow) {
$firstRow=false;
continue;
}
mysql_query("INSERT INTO songdb (artist, title) VALUES
(
'".addslashes($data[0])."',
'".addslashes($data[1])."'
)
") or die (mysql_error());
}
} while ($data = fgetcsv($handle,1000,",","'"));
//
//redirect
header('Location: import.php?success=1'); die;
}
Oh I just read your comment, 5gb. Wow. Manual cleanup is not an option. You need to look at the range of possible ways the data is screwed up and really assess what logic you need to use to capture the right columns.
Is your example above a representative sample or could other fields without enclosures have commas?
Try this, this is working fine for me.
ini_set('auto_detect_line_endings',TRUE);
$csv_data=array();
$file_handle = fopen($_FILES['file_name']['tmp_name'], 'r');
while(($data = fgetcsv($file_handle) ) !== FALSE){
$update_data= array('first'=>$data['0'],
'second'=>$data['1'],
'third'=>$data['2'],
'fourth'=>$data['34']);
// save this array in your database
}

SELECT DISTINCT still showing duplicate result

I have a HTML select form filled by SQL Query using SELECT DISTINCT...
The idea is to not show duplicate values from database, and it's almost working, but in some case it's giving a problem. To fill the SQL columns I'm using a PHP reading a TXT file with delimiters and explode function. If I have in the TXT file 10 duplicate columns, on my HTML it shows 2, instead of only 1... And I note that 1 is with 9 of the entries from database, and the other one have 1 entry that is always the last line from TXT file.
Resuming: the last line of TXT file always duplicate on the HTML select form.
Checking the database, everything looks ok, I really don't know why it's duplicating always in the last one.
I told about the PHP that make the SQL entry because i'm not sure if the problem is in the PHP that contains the HTML select or in the PHP that fill the database... I believe that the problem is in the PHP with HTML select since I'm looking in the database and everything is ok. The SQL query in this php is like this:
<td class="formstyle"><select name="basenamelst" class="formstyle" id="basenamelst">
<option value="Any">Any</option>
<?
$sql = mysql_query("SELECT DISTINCT basename FROM dumpsbase where sold=0");
while($row = mysql_fetch_assoc($sql))
{
if($row['basename'] == "")
{
echo '<option value="'.htmlspecialchars($row['basename'], ENT_QUOTES, 'UTF-8').'">unknOwn</option>';
}
else
{
echo '<option value="'.htmlspecialchars($row['basename'], ENT_QUOTES, 'UTF-8').'">'.htmlspecialchars($row['basename'], ENT_QUOTES, 'UTF-8').'</option>';
}
}
?>
</select>
Remember: if I upload to database 10 duplicate columns, it shows 2 on select. One with 9 entries, and another with 1 entry (always the last line of my TXT file)...
Okay, many people told me to trim() the columns and it still showing duplicate... So I came to the conclusion that I have some issue while loading the TXT for database. Here is the code where I get the values to put on database:
$file = fopen($targetpath, "r") or exit("Unable to open uploaded file!");
while(!feof($file))
{
$line = fgets($file);
$details = explode(" | ", $line);
foreach($details as &$value) // clean each field
{
$value = mysql_real_escape_string($value);
if($value == "")
{
$value = "NONE";
}
}
unset($value);
mysql_query("INSERT INTO dumpsbase VALUES('NULL', '$details[0]', '$details[1]', '$details[2]', '$details[3]', '$details[4]', '0', '$price', 'NONE', now(), 'NONE', 'NONE')") or die ("Uploading Error!");
It sounds to me like the error is when you are populating the table from the file, and that one of the values is ending up subtly different to the others.
The fact that it's the last line that differs makes me wonder if there are newline characters being included in each value (except that last line).
If this is the case, you should be able to correct it by running trim() or similar in your DB.
[Edit] Ideally, you want to do this as early as possible, i.e. correct the data rather than remembering it's wrong when you access it. If you can't find why the initial import is messing it up, you could correct the data immediately afterwards with UPDATE dumpsbase SET basename = TRIM(basename)
Try changing your query to the following:
SELECT DISTINCT TRIM(basename) FROM dumpsbase WHERE sold=0
Hope this helps.

Transform MySQL table and rows

I have one problem here, and I don't even have clue what to Google and how to solve this.
I am making PHP application to export and import data from one MySQL table into another. And I have problem with these tables.
In source table it looks like this:
And my destination table has ID, and pr0, pr1, pr2 as rows. So it looks like this:
Now the problem is the following: If I just copy ( insert every value of 1st table as new row in second) It will have like 20.000 rows, instead of 1000 for example.
Even if I copy every record as new row in second database, is there any way I can fuse rows ? Basically I need to check if value exists in last row with that ID_, if it exist in that row and column (pr2 for example) then insert new row with it, but if last row with same ID_ does not have value in pr2 column, just update that row with value in pr2 column.
I need idea how to do it in PHP or MySQL.
So you got a few Problems:
1) copy the table from SQL to PHP, pay attention to memory usage, run your script with the PHP command Memory_usage(). it will show you that importing SQL Data can be expensive. Look this up. another thing is that PHP DOESNT realese memory on setting new values to array. it will be usefull later on.
2)i didnt understand if the values are unique at the source or should be unique at the destination table.. So i will assume that all the source need to be on the destination as is.
I will also assume that pr = pr0 and quant=pr1.
3) you have missmatch names.. that can also be an issue. would take care of that..also.
4) will use My_sql, as the SQL connector..and $db is connected..
SCRIPT:
<?PHP
$select_sql = "SELECT * FROM Table_source";
$data_source = array();
while($array_data= mysql_fetch_array($select_sql)) {
$data_source[] = $array_data;
$insert_data=array();
}
$bulk =2000;
foreach($data_source as $data){
if(isset($start_query) == false)
{
$start_query = 'REPLACE INTO DEST_TABLE ('ID_','pr0','pr1','pr2')';
}
$insert_data[]=implode(',',$data).',0)';// will set 0 to the
if(count($insert_data) >=$bulk){
$values = implode('),(',$insert_data);
$values = substr(1,2,$values);
$values = ' VALUES '.$values;
$insert_query = $start_query.' '.$values;
$mysqli->query($insert_query);
$insert_data = array();
} //CHECK THE SYNTAX IM NOT SURE OF ALL OF IT MOSTLY THE SQL PART>> SEE THAT THE QUERY IS OK
}
if(count($insert_data) >=$bulk) // IF THERE ARE ANY EXTRA PIECES..
{
$values = implode('),(',$insert_data);
$values = substr(1,2,$values);
$values = ' VALUES '.$values;
$insert_query = $start_query.' '.$values;
$mysqli->query($insert_query);
$insert_data = null;
}
?>
ITs off the top off my head but check this idea and tell me if this work, the bugs night be in small things i forgot with the QUERY structure, print this and PASTE to PHPmyADMIN or you DB query and see its all good, but this concept will sqve a lot of problems..

Categories