I have a rough and complete working CSV transformer. The way my current system works is it imports the CSV file into an SQL database table with static column names, and exports only specific (needed) columns. This system works great but is only specific to one type of CSV file (because the column names are pre-defined.) I'm wondering how I can make this universal. Instead of having it insert column1, column2, column3. I want to insert Spreadsheet Column1, Spreadsheet Column2, Spreadsheet Column3, etc. How would I go about pulling the column names from the CSV file, and creating a new table in the database with the column names being those from the first row of the CSV file.
The current system:
Client uploads CSV file.
A table is created with predefined column names (column 1, column 2, column 3)
Using LOAD DATA INFILE -> PHP scripts will insert the information from the CSV file into the recently created table.
The next query that is ran is simply something along the lines of taking only specific columns out of the table and exporting it to a final CSV file.
The system that would be ideal:
Client uploads CSV file.
PHP scripts read the CSV file and takes only the first row (column names), after taking these column names, it'll create a new table based on the column names.
PHP scripts now use LOAD DATA INFILE.
The rest is the same as current system.
Current code:
import.php
include("/inc/database.php");
include("/inc/functions.php");
include("/inc/data.php");
if($_SERVER['REQUEST_METHOD'] == 'POST'){
$string = random_string(7);
$new_file_name = 'report_'. $string .'.csv';
$themove = move_uploaded_file($_FILES['csv']['tmp_name'], 'C:/xampp/htdocs/uploads/'.$new_file_name);
mysql_query("CREATE TABLE report_". $string ."(". $colNames .")") or die(mysql_error());
$sql = "LOAD DATA INFILE '/xampp/htdocs/uploads/report_". $string .".csv'
INTO TABLE report_". $string ."
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n'
(". $insertColNames .")";
$query = mysql_query($sql) or die(mysql_error());
header('Location: download.php?dlname='.$string.'');
}
data.php (shortened most of this. In reality there are about 200 columns going in, twenty-thirty coming out)
<?php
$colNames = "Web_Site_Member_ID text,
Master_Member_ID text,
API_GUID text,
Constituent_ID text";
$insertColNames = "Web_Site_Member_ID,
Master_Member_ID,
API_GUID,
Constituent_ID";
$exportNames = "Web_Site_Member_ID, Date_Membership_Expires, Membership, Member_Type_Code";
?>
functions.php just includes the block of code for generating a random string/file name.
For CSV file reading please look at the fgetcsv() function. You should easily be able to extract a row of data and access each individual field in the resulting array for your column header definitions.
Related
This question already has answers here:
Importing CSV data using PHP/MySQL
(6 answers)
Closed 5 years ago.
I have a csv file which has 13 columns, each of these columns has lots and lots of data and what I need is to be able to extract this data from the .csv file into mysql database.
Firstly I need help with creating specific tables for each of the columns of the csv file, as I'm quite new with mysql and I wasn't too sure on what attributes to assign to each table.
Here is the structure of the csv file...
Columns
pid, start_time, end_time, epoch_start, epoch_end, complete_title, media_type, masterband, service, brand_pid, is_clip, categories, tags
Data under columns
p00547jm (pid), 1003394820 (start_time), 1003999620 (end_time), 2001-10-18T08:47:00 (epoch_start), 2001-10-25T08:47:00 (epoch_end), in_our_time:_democracy (complete_title), audio (media_type), bbc_radio_four (masterband), bbc_radio_four (service), b006qykl (brand_pid),0 (is_clip), [9100005:1:factual.9200041:2:arts_culture_and_the_media.9200055:2:history] (categories), [democracy.history.philosophy.plato.ancient_greece] (tags)
Create your table first and then use
LOAD DATA LOCAL INFILE '/home/dummy/dummy.csv' INTO TABLE tablename
FIELDS TERMINATED BY ',';
to insert the data from the file into your new table.
Also, make sure when you login into MySQL you use
mysql -uusername -ppassword --local-infile
As for creating a table:
We can give the syntax but we cannot write the whole query for you. You can try it yourself as you have the necessary data with you.
create table table_name (col1 datatype(size) NOT NULL primary key,
col1......);
This will generate the create table query $1 will become the table name
[Create mysql table directly from CSV file using the CSV Storage engine?
#!/bin/sh
# pass in the file name as an argument: ./mktable filename.csv
echo "create table $1 ( "
head -1 $1 | sed -e 's/,/ varchar(255),\n/g'
echo " varchar(255) );"
Then you can read the file and put your insert logic in the while loop
[Import CSV into MYSQL but ignore header row
//open the csv file for reading
$handle = fopen($file_path, 'r');
// read the first line and ignore it
fgets($handle);
while (($data = fgetcsv($handle, 1000, ',')) !== FALSE) {
// do your thing
}
My code for fetching mysql rows and write to csv file using fputscv is working fine. But I want to omit first column while writing to the csv file. Clearly telling, all the fetched values are different points of a graph and I would be directly importing that generated csv file for a graph generating code. The id (which is not a graph point) will be included in csv file if I use a query like,
$sql = "SELECT * FROM mybase WHERE id='$id'";
$qry = $dbo->prepare($sql);
$qry->execute();
$row = $qry->fetch(PDO::FETCH_ASSOC);
fputcsv($data, $row);
Anyone knows the best method to eliminate the ID before writing into the csv file.?
I saw an obvious and simple result here. Unfortunately, I have too many columns and it is difficult to specify each column in sql query. Thanks..
Just UNSET it like
unset($row['ID']);
fputcsv($data, $row);
Orelse You can fetch all the columns except ID in your query itself like
$sql = "SELECT other_than_ID_column FROM mybase WHERE id='$id'";
I need to insert data from a plain text file, explode each line to 2 parts and then insert to the database. I'm doing in this way, But can this programme be optimized for speed ?
the file has around 27000 lines of entry
DB structure [unique key (ext,info)]
ext [varchar]
info [varchar]
code:
$string = file_get_contents('list.txt');
$file_list=explode("\n",$string);
$entry=0;
$db = new mysqli('localhost', 'root', '', 'file_type');
$sql = $db->prepare('INSERT INTO info (ext,info) VALUES(?, ?)');
$j=count($file_list);
for($i=0;$i<$j;$i++)
{
$data=explode(' ',$file_list[$i],2);
$sql->bind_param('ss', $data[0], $data[1]);
$sql->execute();
$entry++;
}
$sql->close();
echo $entry.' entry inserted !<hr>';
If you are sure that file contains unique pairs of ext/info, you can try to disable keys for import:
ALTER TABLE `info` DISABLE KEYS;
And after import:
ALTER TABLE `info` ENABLE KEYS;
This way unique index will be rebuild once for all records, not every time something is inserted.
To increase speed even more you should change format of this file to be CSV compatible and use mysql LOAD DATA to avoid parsing every line in php.
When there are multiple items to be inserted you usually put all data in a CSV file, create a temporary table with columns matching CSV, and then do a LOAD DATA [LOCAL] INFILE, and then move that data into destination table. But as I can see you don't need much additional processing, so you can even treat your input file as a CSV without any additional trouble.
$db->exec('CREATE TEMPORARY TABLE _tmp_info (ext VARCHAR(255), info VARCHAR(255))');
$db->exec("LOAD DATA LOCAL INFILE '{$filename}' INTO TABLE _tmp_info
FIELDS TERMINATED BY ' '
LINES TERMINATED BY '\n'"); // $filename = 'list.txt' in your case
$db->exec('INSERT INTO info (ext, info) SELECT t.ext, t.info FROM _tmp_info t');
You can run a COUNT(*) on temp table after that to show how many records were there.
If you have a large file that you want to read in I would not use file_get_contents. By using it you force the interpreter to store the entire contents in memory all at once, which is a bit wasteful.
The following is a snippet taken from here:
$file_handle = fopen("myfile", "r");
while (!feof($file_handle)) {
$line = fgets($file_handle);
echo $line;
}
fclose($file_handle);
This is different in that all you are keeping in memory from the file at a single instance in time is a single line (not the entire contents of the file), which in your case will probably lower the run-time memory footprint of your script. In your case, you can use the same loop to perform your INSERT operation.
If you can use something like Talend. It's an ETL program, simple and free (it has a paid version).
Here is the magic solution [3 seconds vs 240 seconds]
ALTER TABLE info DISABLE KEYS;
$db->autocommit(FALSE);
//insert
$db->commit();
ALTER TABLE info ENABLE KEYS;
I have a CSV file which contains data seperated with tabs. I need to import the data into a MySQL table which consists of two columns. The first CSV column should go into the first column of the table and similarly for the second.
<?php
$con=mysql_connect("localhost","root","");
mysql_select_db("translation",$con);
$open=fopen("EH_excel.txt","r");
while(($get=fgetcsv($open,1000,","))!==false) {
mysql_query("insert into dictionary(english,croatian)
values('".$get[0]."','".$get[1]."')");
}
fclose($open); echo "Import Done.";
?>
Can anybody help me?
Since what you have is called Tab Delimited Files
This is the way you import it to
SQL
LOAD DATA LOCAL INFILE 'sample.txt' INTO TABLE sample
FIELDS TERMINATED BY '\t'
OPTIONALLY ENCLOSED BY ''
ESCAPED BY ''
LINES TERMINATED BY '\n';
I am trying to import an XLS file into PHP, where I can then edit the information and import it into mySQL. I have never done anything related to this, so I am having a hard time grasping how to approach it.
I have looked at a few open source projects:
PHP Excel Reader
ExcelRead
PHPExcel
None of these options perfectly fit what I want to do or maybe I just haven't gone deep enough into the documentation.
There are some things that needed to be taken into consideration. The XLS file cannot be converted into any other file format. This is being made for ease-of-access for nontechnical users. The XLS file is a report generated on another website that will have the same format (columns) every time.
For example, every XLS file with have the same amount of columns (this would be A1):
*ID |Email |First Name |Last Name |Paid |Active |State |Country|*
But, there are more columns in the XLS file than what is going to be imported into the DB.
For example, the rows that are being imported (this would be A1):
*ID |Email |First Name |Last Name |Country*
I know one of two ways to do edit the data would be A. Use something like PHPExcel to read in the data, edit it, then send it to the DB or B. Use something like PHPExcel to convert the XLS to CSV, do a raw import into a temp table, edit the data, and insert it into the old table.
I have read a lot of the PHPExcel documentation but, it doesn't have anything on importing into a database and I don't really even know where to start with editing the XLS before or after importing.
I have googled a lot of keywords and mostly found results on how to read/write/preview XLS. I am looking for advice on the best way of doing all of these things in the least and simplest steps.
See this article on using PHP-ExcelReader, in particular the short section titled "Turning the Tables".
Any solution you have will end up looking like this:
Read a row from the XLS (requires an XLS reader)
Modify the data from the row as needed for your database.
Insert modified data into the database.
You seem to have this fixation on "Editing the data". This is just PHP--you get a value from the XLS reader, modify it with PHP code, then insert into the database. There's no intermediate file, you don't modify the XLS--it's just PHP.
This is a super-simple, untested example of the inner loop of the program you need to write. This is just to illustrate the general pattern.
$colsYouWant = array(1,2,3,4,8);
$sql = 'INSERT INTO data (id, email, fname, lname, country) VALUES (?,?,?,?,?)';
$stmt = $pdo->prepare($sql);
$sheet = $excel->sheets[0];
// the excel reader seems to index by 1 instead of 0: be careful!
for ($rowindex=2; $rowindex <= $sheet['numRows']; $rowindex++) {
$xlsRow = $sheet['cells'][$rowindex];
$row = array();
foreach ($colsYouWant as $colindex) {
$row[] = $xlsRow[$colindex];
}
// now let's "edit the row"
// trim all strings
$row = array_map('trim', $row);
// convert id to an integer
$row[0] = (int) $row[0];
// capitalize first and last name
// (use mb_* functions if non-ascii--I don't know spreadsheet's charset)
$row[2] = ucfirst(strtolower($row[2]));
$row[3] = ucfirst(strtolower($row[3]));
// do whatever other normalization you want to $row
// Insert into db:
$stmt->execute($row);
}