Create MySQL table dynamically from Excel CSV file - php

My goal is to create a MySQL table containing data from my CSV file.
I know how to create a MySQL table and how to load data from excel in it.
But the problem is:
I have a large CSV file containing long column names (questions labels for example "Q27 : Are you happy with the after sales service?") so it would be boring to create a MySQL table by copying all column names(almost 35) and add 'VARCHAR(100) NOT NULL'.
That's why I would like to write a small php script to create a MySQL table by getting the first row of my file, and then fill it with the rest of the csv file data.
For now, my script looks like this :
<?php
$host = 'localhost';
$user = 'root';
$pass = '';
$database = 'test';
$db = #mysql_connect($host, $user, $pass) or die('mysql connection pb');
#mysql_select_db($database) or die('database selection pb');
/********************************************************************************/
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) { $file = $argv[1]; }
else {
echo "Please provide a file name\n"; exit;
}
if($argv[2]) {
$table = $argv[2];
}
else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
/**************************************************************************** ****/
// Get the first row to create the column headings
$fp = fopen($file, 'r');
$frow = fgetcsv($fp,";");
$columns=false;
print_r($frow);
foreach($frow as $column) {
if($columns) $columns .= ', ';
$columns .= "`$column` VARCHAR(250) NOT NULL";
}
$create = "create table if not exists $table ($columns);";
#mysql_query($create, $db) or die('table creation pb');
/**************************************************************************** ****/
// Import the data into the newly created table.
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$q = "load data infile '$file' into table $table fields terminated by ',' ignore 1 lines";
#mysql_query($q, $db);
?>
And when i run in command line : php myscript.php csvfile.csv mytable, it appears that the problem is in the table creation query.
And on top of that the column names are not well identified even though they are separated by ";" in the csv.

As mentioned refrain from using mysql_ functions which as of PHP 7 (current version) this extension is no longer available. Use either mysqli or PDO.
Below is a PDO example with try/catch (more informative than die()). Also, the csv read is handled slightly different and its concatenation in SQL create table string.
<?php
$host="localhost";
$username="root";
$password="password";
$database="test"
// Parameters: filename.csv table_name
$argv = $_SERVER['argv'];
if($argv[1]) {
$file = $argv[1];
} else {
echo "Please provide a file name\n";
exit;
}
if($argv[2]) {
$table = $argv[2];
} else {
echo "Please provide a table name\n";
$table = pathinfo($file);
$table = $table['filename'];
}
// Read in only first row of CSV file
$handle = fopen($file, "r");
$row = 1;
$columns = [];
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE AND $row==1) {
$columns = $data;
$row++;
}
//SQL string commands
$createSQL = "CREATE TABLE IF NOT EXISTS $table
(".implode(" VARCHAR(255) NOT NULL, ", $columns). "
VARCHAR(255) NOT NULL);";
$file = addslashes(realpath(dirname(__FILE__)).'\\'.$file);
$loadSQL = "LOAD DATA INFILE '$file'
INTO TABLE $table
FIELDS TERMINATED BY ','
IGNORE 1 LINES";
// Open database connection
try {
$dbh = new PDO("mysql:host=$host;dbname=$database",$username,$password);
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// Execute queries
$S1TH = $dbh->query($createSQL);
$S2TH = $dbh->query($loadSQL);
}
catch(PDOException $e) {
echo $e->getMessage();
}
# Close database connection
$dbh = null;
?>

Related

How can I use a for loop to insert data from a csv table to a sql database?

This is the sample csv file I'm using:
Column1,Column2,Column3
data1,data2,data3
data1,data2,data3
data1,data2,data3
data1,data2,data3
The purpose of my program is to insert this table into a php file via command line and insert the data into an SQL database. I'm using this project as a way of learning how to use MySql.
A csv file is taken and the data is then converted into an array or arrays. A database is then created and the data is supposed to be inserted into the table.
Instead of my data being inserted, I get only one row with null values.
<?php
/**
* Created by PhpStorm.
* User:
* Date: 6/16/2017
* Time: 11:32 AM
*/
$servername = "localhost";
$username = "pop-user";
$password = "pop-pw";
$database = 'popdb';
parse_str(implode('&', array_slice($argv, 1)), $_GET);
$file = array_map('str_getcsv', file($argv[1]));
// connecting to MySQL
$link = mysqli_connect($servername,$username,$password);
// check if connection completed
if ($link->connect_error) {
die("Connection failed: ". $link->connect_error);
}
//creating database
printf("Creating database...\n");
$dbcheck = mysqli_select_db($link, $database);
// if database doesn't exist, then one will be created
if (!$dbcheck) {
$sql = 'CREATE DATABASE '. $database;
if (mysqli_query($link, $sql)) {
echo "Database ". $database ." created\n";
}
else {
echo "Failed to create database:\n";
echo $link->error."\n";
}
}
printf("Creating table...");
//creating table to hold information
$sql = 'USE '. $database. ';';
printf("\r\n");
mysqli_query($link,$sql);
$sql2 = "CREATE TABLE popCensus (";
foreach ($file[0] as $rows) {
if ($rows != end($file[0]))
$sql2 .= "{$rows} varchar(33), ";
else
$sql2 .= "{$rows} varchar(33)";
}
$sql2 .= ");";
echo $sql2;
printf("\r\n");
mysqli_query($link,$sql2);
printf("Inserting data into table...\n");
$cnt = 1;
$sql3 = "";
//Not inserting data
foreach ( $file as $file[$cnt]) {
$sql3 = "INSERT INTO popcensus VALUES ( ";
foreach ($file[$cnt] as $rows) {
if ($rows != end($file[$cnt]))
$sql3 .= "{$rows} , ";
else
$sql3 .= "{$rows});";
}
printf($sql3);
printf("\n");
mysqli_query($link, $sql3);
$cnt++;
printf("cnt: ". $cnt."\n");
$sql3 = "";
}
printf("\nDone\n");
mysqli_close($link);
?>
edit: I'm able to parse the information into an arrays of arrays. the problem I'm having is trying to insert them into a table afterwards.
you need to use prepared statement inside the loop
Try checking the query for an error result inside the foreach loop.
if (false === mysqli_query($link, $sql3)) {
throw new Exception(mysqli_error($link));
}
Beyond that, you should really be using a prepare statement.

How to get field name using the improved MySQL (MySQLI)? [duplicate]

This question already has answers here:
Building a HTML table using MySQL/PHP results
(3 answers)
Closed 5 years ago.
The following is a PHP code that I'm using to convert a MySQL table into csv output. However, I couldn't figure out how to convert MySQL_field_name to the equivalent in MySQLI. Can anyone assist?
<?php
/* vars for export */
// database record to be exported
$db_record = 'manufacturing_';
// optional where query
$where = 'WHERE 1 ORDER BY 1';
// filename for export
$csv_filename = 'db_export_'.$db_record.'_'.date('Y-m-d').'.csv';
// database variables
$hostname = "localhost";
$user = "XXXXXXXXX";
$password = "XXXXXXXXX";
$database = "XXXXXXXXX";
// Database connecten voor alle services
$connection = mysqli_connect($hostname, $user, $password) or die('Could not connect: ' . mysqli_error());
mysqli_select_db($connection, $database) or die ('Could not select database ' . mysqli_error());
// create empty variable to be filled with export data
$csv_export = '';
// query to get data from database
$query = mysqli_query($connection, "SELECT * FROM ".$db_record." ".$where);
$field = mysqli_num_fields($query);
// create line with field names
for($i = 0; $i < $field; $i++) {
$csv_export.= mysqli_field_name($connection, $query,$i).';';
}
// newline (seems to work both on Linux & Windows servers)
$csv_export.= '
';
// loop through database query and fill export variable
while($row = mysqli_fetch_array($connection, $query)) {
// create line with field values
for($i = 0; $i < $field; $i++) {
$csv_export.= '"'.$row[mysqli_field_name($connection, $query,$i)].'";';
}
$csv_export.= '
';
}
// Export the data and prompt a csv file for download
header("Content-type: text/x-csv");
header("Content-Disposition: attachment; filename=".$csv_filename."");
echo($csv_export);
?>
Error Message:
Fatal error: Call to undefined function mysqli_field_name()
You can get this infos from INFORMATION_SCHEMA with a query like this. you only must change the TABLE_SCHEMA and the TABLE_NAME
SELECT COLUMN_NAME
FROM information_schema.columns
WHERE TABLE_NAME ='owner' AND TABLE_SCHEMA='test'
ORDER BY ORDINAL_POSITION;

CSV uploading without browse button

I want to upload the contents of a CSV while one PHP page is running. I don't want any browse button to upload the CSV. Whenever the page is running the page should find the CSV which the path is already defined in the PHP page and contents should be inserted into the table. Now I am getting error related with fopen.
Here is my code
<?php
//database connection details
$connect = mysql_connect('localhost', 'root', '');
if (!$connect) {
die('Could not connect to MySQL: ' . mysql_error());
}
//your database name
$cid = mysql_select_db('test', $connect);
// path where your CSV file is located
define('CSV_PATH', 'D:/xamp/htdocs/test/');
// Name of your CSV file
$csv_file = CSV_PATH . "test.csv";
echo $csv_file;
if (($handle = fopen($csv_file, "r")) !== FALSE) {
fgetcsv($handle);
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$num = count($data);
for ($c = 0; $c < $num; $c++) {
$col[$c] = $data[$c];
}
$col1 = $col[0];
$col2 = $col[1];
$col3 = $col[2];
$col4 = $col[3];
$col5 = $col[4];
$col6 = $col[5];
// SQL Query to insert data into DataBase
$query = "INSERT INTO testcsv(Line,Part No,Make,Model,Year,Part Type) VALUES('" . $col1 . "','" . $col2 . "','" . $col3 . "','" . $col4 . "','" . $col5 . "','" . $col6 . "')";
$s = mysql_query($query, $connect);
}
fclose($handle);
}
echo "File data successfully imported to database!!";
mysql_close($connect);
?>
I am getting this error
Warning: fopen(D:/xamp/htdocs/test/test.csv): failed to open stream: No such file or directory in D:\xamp\htdocs\test\test.php on line 22
File data successfully imported to database!!
Can anyone help me?
I'm not sure why you are getting that particular error - one might assume that the file does not exist or that the directory is not readable but you are using the now deprecated mysql_ functions and directly embedding variables in the sql - thus making it vulnerable to sql injection. However, as this looks to be only a test that is probably not an issue.
The preferred method for this type of thing would be to use either mysqli or PDO in conjunction with prepared statements - below is an example of how you might implement that - I tested this with different data and database details and it seemed to work fine.
define('CSV_PATH','D:/xamp/htdocs/test/');
$filepath = CSV_PATH . "test.csv";
/* database connection details */
$host = 'localhost';
$uname = 'xxx';
$pwd = 'xxx';
$db = 'xxx';
/* create db connection */
$con = new mysqli( $host, $uname, $pwd, $db );
/* construct required sql statement */
$sql='insert into `testcsv` (`Line`,`Part No`,`Make`,`Model`,`Year`,`Part Type`) values (?,?,?,?,?,?)';
/* create prepared statement */
$stmt=$con->prepare( $sql );
if( !$stmt ){
echo 'error preparing sql statement!';
$con->close();
} else {
/* bind the columns to variables which will be populated later */
/* use "i" for integer and "s" for string values */
$stmt->bind_param( 'ssssss', $line,$part,$make,$model,$year,$type );
/* access csv file */
$file=new SplFileObject( $filepath );
/* Process each row of the csv file */
while( !$file->eof() ) {
/* read the line into a variable */
$data=$file->fgetcsv();
if( !empty( $data ) ){
/* assign a variable to each field value for this row */
list( $line,$part,$make,$model,$year,$type )=$data;
/* execute statement with the now defined variables */
$stmt->execute();
}
}
/* tidy up */
$stmt->close();
$con->close();
echo 'database updated with new records from csv';
}

Error Importing CSV to MySQL with PHP

I am trying to use this code to import a csv file into my MySQL database:
... Thanks for solved that question!! ....
What am I doing wrong?
Guys I could solved that problem but now... I have an other problem, this is the code now:
<?php
$databasehost = "localhost";
$databasename = "cauctti";
$databasetable = "sample";
$databaseusername="root";
$databasepassword = "toor";
$fieldseparator = ";";
$lineseparator = "\r|n";
$csvfile = "Reporting.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION,
PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
When I try to run it, only insert one row on the table...
Loaded a total of 1 records from this csv file.
Why it only gets one row when my csv file have like 5.000 entries...
As found here
You should set CHARACTER SET UTF8 in your query.
Example:
LOAD DATA INFILE 'file'
IGNORE INTO TABLE table
CHARACTER SET UTF8
FIELDS TERMINATED BY ';'
OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
Have a look here --> Here
<?php
$table = "tableName";
$fileName = "States.csv";
$ignoreFirstRow = 1;
if (($handle = fopen($fileName, "r")) !== FALSE){
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE)
{
if($ignoreFirstRow != 1){
$sql = "insert into ".$table." values(";
$sql .= '"'.implode('","',$data).'"';
echo "".$sql.');';
$sql = "";
}
$ignoreFirstRow++;
}
fclose($handle);
}
?>

Import a CSV file into MySQL using PHP

I'm trying to import CSV data into a MySQL database using the fgetcsv function.
if(isset($_POST['submit'])) {
$fname = $_FILES['sel_file']['name'];
$var = 'Invalid File';
$chk_ext = explode(".",$fname);
if(strtolower($chk_ext[1]) == "csv") {
$filename = $_FILES['sel_file']['tmp_name'];
$handle = fopen($filename, "r");
$res = mysql_query("SELECT * FROM vpireport");
$rows = mysql_num_rows($res);
if($rows>=0) {
mysql_query("DELETE FROM vpireport") or die(mysql_error());
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
if($i==1)
continue;
$sql = "INSERT into vpireport
(item_code,
company_id,
purchase,
purchase_value)
values
(".$data[0].",
".$data[1].",
".$data[2].",
".$data[3].")";
//echo "$sql";
mysql_query($sql) or die(mysql_error());
}
}
fclose($handle);
?>
<script language="javascript">
alert("Successfully Imported!");
</script>
<?
}
The problem is it gets stuck in between the import process and displays the following error:
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'S',0,0)' at line 1
The file is imported only partially each time. Only between 200-300 lines out of a 10000 line file are imported.
Here is the DDL of my table:
create table vpireport (
id int not null auto_increment,
item_code int,
company_id int,
purchase double,
primary key(id),
foreign key(company_id) references users(userid)
);
I haven't been able to find the problem so far, any help appreciated. Thanks.
You probably need to escape quotes, which you could accomplish using PDO and prepared statements.
I've skipped most of your code in the example for brevity and just focused on the for loop.
<?php
// Use PDO to connect to the DB
$dsn = 'mysql:dbname=YOUR_DB;host=localhost';
$user = 'DB_USERNAME';
$password = 'DB_PASSWORD';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
for($i =1;($data = fgetcsv($handle, 10000, ",")) !== FALSE; $i++) {
// The query uses placeholders for data
$sql = "INSERT INTO vpireport
(item_code,company_id,purchase,purchase_value)
VALUES
(:item_code,:company_id,:purchase,:purchase_value)";
$sth = $dbh->prepare($sql);
// The data is bound to the placeholders
$sth->bindParam(':item_code', $data[0]);
$sth->bindParam(':company_id', $data[1]);
$sth->bindParam(':purchase', $data[2]);
$sth->bindParam(':purhcase_value', $data[3]);
// The row is actually inserted here
$sth->execute();
$sth->closeCursor();
}
That won't get rid of any problem characters, though, so you may want to look at some kind of data sanitization if that poses a problem.
uncomment the //echo "$sql"; and look what is the last query (with error) - it may be that the csv data contains strange characters or the query is cut off.
BTW: you can also import csv file by mysql:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
$row = 1;
if (($handle = fopen("albums.csv", "r")) !== FALSE) {
while (($data = fgetcsv($handle, 1000, ',','"')) !== FALSE) {
if($row!=1){
$num = count($data);
$albumIndex=0;
//Insert into tbl_albums
$sqlAlbums="INSERT INTO tbl_albums(albumName) VALUES ('".$data[$albumIndex]."')";
$resultAlbums=mysql_query($sqlAlbums);
}
}
$row++;
}
}
fclose($handle);

Categories