I have recently asked how to insert a CSV into a MySQL database. It was suggested to me to use LOAD DATA LOCAL INFILE, however it turns out that this is disabled on my host, so no longer an option. Back to PHP loops..
I'm having an issue looping through the results of a temp upload, since I'm mapping the values to an array on insert. On multiple lines therefore, this causes the same entry to be entered twice (the first line's values), as the array values are explicitly defined.
It's inserting 1, 2, 3, 4 and then 1, 2, 3, 4. I want to insert 1, 2, 3, 4 then 5, 6, 7, 8 of the array.
What's the solution (aside from hacky for's and row++)?
Thanks in advance.
$handle = fopen($_FILES['csv']['tmp_name'], "r");
$sql = "INSERT INTO tbl (col1, col2, col3, col4) VALUES (?, ?, ?, ?)";
while (($data = fgetcsv($handle, 1000, ",")) !== FALSE) {
$query = $db->prepare($sql);
if ($query->execute(array($data[0],$data[1],$data[2],$data[3]))) return true;
else return false;
}
The only thing I can think of is that your loop is only executing once, but you run the loop twice. (You have a "return" statement in your loop.)
The following should work:
function loadCsv($db, $filename){
$fp = fopen($filename, 'rb');
$sql = 'INSERT INTO tbl (col1, col2, col3, col4) VALUES (?,?,?,?)';
$pstmt = $db->prepare($sql);
while (FALSE !== ($data = fgetcsv($fp, 1000))) {
$cols = array_slice($data, 0, 4);
$query->execute($cols);
}
$pstmt->closeCursor();
fclose($fp);
}
For maximum compatibility and performance, I recommend connecting to PDO with a function like this connect_PDO function.
First of all, you only need to prepare the query once (that's one of the two main advantages of using prepared statements, with injection prevention being the other one), so put the call to prepare() before the while loop, not inside it.
Outside of that, I see no reason why the code you've posted would behave the way you claim it does, unless your data is duplicated in your CSV file.
The issue was with the return statement. Removing the return instantly fixed the issue.
Unfortunately the user who posted this answer has since removed it.
Thanks everyone for your suggestion and help with this!
Related
So I am attempting to write a generic sqlite insert that can be used no matter how many items a row has. This is for a single row, assumes all columns other than ID, which is set to autoincrementing integer, are assigned, and bindparam must be used. I have been attempting it like so:
Table Quarterbacks
ID---firstName---lastName
public static function insert($table, $values)
{
$pdo = new PDO('sqlite:testTable.sqlite');
$inputString = implode(",", $values);
$statement = $pdo->prepare("insert into $table values (:value)");
$statement->bindParam(':value', $inputString);
$statement->execute();
}
$new = array("Steve", "Young");
Query::insert("Quarterbacks", $new);
The idea being that the table will now add a new row, increment the ID, and add Steve Young. But I get the generic error that the prepare statement is false. I know my pdo is connecting to the database, as other test methods work. There's a lot of array related threads out there but it seems like they're much more complicated than what I'm trying to do. I'm pretty sure it has to do with it treating the single string as invalid, but it also won't take an array of strings for values.
Edit:I'm starting to lean towards a compromise like bash, ie provide a large but not infinite amount of function parameters. Also open to the ...$ style but I feel like that ends up with the same problem as now.
I was able to get this to work
$name = "('daunte' , 'culpepper')";
$cats = "('firstName', 'lastName')";
$statement = $pdo->prepare("insert into $table" .$cats ." values" .$name);
$statement->execute();
But not this
$name = "('reggie' , 'wayne')";
$cats = "('firstName', 'lastName')";
$statement = $pdo->prepare("insert into $table:cats values:name");
$statement->bindParam(':cats', $cats);
$statement->bindParam(':name', $name);
$statement->execute();
I have to import a csv to a mysql database
I can't use load data infile because it's disabled on the webserver.
are there any other ways to do so?
If you have a scripting language available, you can loop through the CSV lines and have it generate SQL code:
PHP example:
<?php
$lines = file('file.csv');
foreach($lines as $line){
//sepatates each cell by the delimiter "," (watch for delimiters in the cell, escaped or not)
$cell = explode(",",$line);
$sql = "INSERT INTO table (col1,col2,col3) VALUES (";
$sql.= "'".$cell[0]."','".$cell[1]."','".$cell[2]."');";
echo $sql;
}
?>
Loop through the file and insert using a prepared query. Prepared querys should be quicker too, since the DB doesn't have to recompile every SQL string you send it. That will be more noticeable when you have thousands and thousands of lines.
<?php
// assume $db is a PDO connection
$stmt = $db->prepare('INSERT INTO table (col1, col2, col3) VALUES(?, ?, ?)');
// read file contents to an array
$lines = file('file.csv');
// insert each line
foreach ($lines as $line) {
// see manual to specify $delimter, $enclousure, or $more
$cols = str_getcsv($lines);
$stmt->execute($cols);
}
That'll work. Since we're using file(), the script can consume a lot of memory if your CSV file is HUGE. To make better use of resources, do the following to keep only one line in memory at a time:
<?php
// assume $db is a PDO connection
$stmt = $db->prepare('INSERT INTO table (col1, col2, col3) VALUES(?, ?, ?)');
$handle = fopen('test.csv', 'r');
while ($cols = fgetcsv($handle)) {
$stmt->execute($cols);
}
I am inserting the $_POST contents of my PHP array into a table with PDO. I was looking at the following lines of code and I had one of those "there has to be a better way to do this" moments. If the key name matches the column name in the table, is there a more simple way to insert all of it?
Code for example:
$statement = $db->prepare("INSERT INTO `applications`(`username`, `email`, `password`, `name`) VALUES (?,?,?,?)");
$statement->execute(array($_POST['username'], $_POST['email'],$_POST['password'],$_POST['name']));
This code WORKS but it just seems a bit over-the-top (especially as more and more columns are added).
I would do it this way:
Declare the columns first. We'll use these to extract a subset of $_POST for use as columns. Otherwise a user could pass bogus request parameters that don't match any columns of the table, which would break our SQL.
$columns = array('username','email','password','name');
$column_list = join(',', $columns);
Create named parameter placeholders i.e. :username.
$param_list = join(',', array_map(function($col) { return ":$col"; }, $columns));
Form the SQL separately, because it's easier to read and debug if it's in its own variable.
$sql = "INSERT INTO `applications` ($column_list) VALUES ($param_list)";
Always check for error status returned from prepare() and execute().
$statement = $db->prepare($sql);
if ($statement === false) {
die(print_r($db->errorInfo(), true));
}
Here we take only the fields of $_POST that match the columns we want to insert.
$param_values = array_intersect_key($_POST, array_flip($columns));
And pass that array to execute(). Again, check for error return status.
$status = $statement->execute($param_values);
if ($status === false) {
die(print_r($statement->errorInfo(), true));
}
How can I implement recursive MySQL Queries. I am trying to look for it but resources are not very helpful.
Trying to implement similar logic.
public function initiateInserts()
{
//Open Large CSV File(min 100K rows) for parsing.
$this->fin = fopen($file,'r') or die('Cannot open file');
//Parsing Large CSV file to get data and initiate insertion into schema.
$query = "";
while (($data=fgetcsv($this->fin,5000,";"))!==FALSE)
{
$query = $query + "INSERT INTO dt_table (id, code, connectid, connectcode)
VALUES (" + $data[0] + ", " + $data[1] + ", " + $data[2] + ", " + $data[3] + ")";
}
$stmt = $this->prepare($query);
// Execute the statement
$stmt->execute();
$this->checkForErrors($stmt);
}
#Author: Numenor
Error Message: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '0' at line 1
This Approach inspired to look for an MySQL recursive query approach.
Here is the Approach I was using Earlier:
Current Code:
public function initiateInserts()
{
//Open Large CSV File(min 100K rows) for parsing.
$this->fin = fopen($file,'r') or die('Cannot open file');
//Parsing Large CSV file to get data and initiate insertion into schema.
while (($data=fgetcsv($this->fin,5000,";"))!==FALSE)
{
$query = "INSERT INTO dt_table (id, code, connectid, connectcode)
VALUES (:id, :code, :connectid, :connectcode)";
$stmt = $this->prepare($query);
// Then, for each line : bind the parameters
$stmt->bindValue(':id', $data[0], PDO::PARAM_INT);
$stmt->bindValue(':code', $data[1], PDO::PARAM_INT);
$stmt->bindValue(':connectid', $data[2], PDO::PARAM_INT);
$stmt->bindValue(':connectcode', $data[3], PDO::PARAM_INT);
// Execute the statement
$stmt->execute();
$this->checkForErrors($stmt);
}
}
Updated Code
public function initiateInserts()
{
//Open Large CSV File(min 100K rows) for parsing.
$this->fin = fopen($file,'r') or die('Cannot open file');
//Prepare insertion query to insert data into schema.
$query = "INSERT INTO dt_table (id, code, connectid, connectcode)
VALUES (:id, :code, :connectid, :connectcode)";
$stmt = $this->prepare($query);
// Then, for each line : bind the parameters
$stmt->bindValue(':id', $data[0], PDO::PARAM_INT);
$stmt->bindValue(':code', $data[1], PDO::PARAM_INT);
$stmt->bindValue(':connectid', $data[2], PDO::PARAM_INT);
$stmt->bindValue(':connectcode', $data[3], PDO::PARAM_INT);
//Loop through CSV file and execute inserts prepared, but this is not working
//and there are not data being populated into database.
while (($data=fgetcsv($this->fin,5000,";"))!==FALSE)
{
// Execute the statement
list($id, $code, $connid, $conncode)=$data;
$stmt->execute();
$this->checkForErrors($stmt);
}
}
This was my Main Question for which I am looking for suggestions !!!
There's nothing recursive in that code snippet.
The wrong operator is used to concatenate the strings, it's . (dot) not +
You'd have to use something like mysqli::multi_query() to execute more than one statement with a single function call and the statements would have to be separated by a delimiter character (by default a semicolon)
Since you're already using prepare() and execute() why not simply make it a parametrized prepared statement and then assign the values in each iteration of the loop and execute the statement? (Exactly what is $this and what type of object does $this->prepare() return?)
edit and btw: $this->prepare() indicates that your class extends a database class. And it also holds a file descriptor $this->fin. This has a certain code smell. My guess is that your class uses/has a database/datasink object and a file/datasource, but not is a database+readfile class. Only extend a class if your derived class is something.
edit: a simple example
class Foo {
protected $pdo;
public function __construct(PDO $pdo) {
$this->pdo = $pdo;
}
public function initiateInserts($file)
{
$query = '
INSERT INTO
dt_table_tmp
(id, code, connectid, connectcode)
VALUES
(:id, :code, :connid, :conncode)
';
$stmt = $this->pdo->prepare($query);
$stmt->bindParam(':id', $id);
$stmt->bindParam(':code', $code);
$stmt->bindParam(':connid', $connid);
$stmt->bindParam(':conncode', $conncode);
$fin = fopen($file, 'r') or die('Cannot open file');
while ( false!==($data=fgetcsv($fin,5000,";")) ) {
list($id, $code, $connid, $conncode)=$data;
$stmt->execute();
}
}
}
$pdo = new PDO("mysql:host=localhost;dbname=test", 'localonly', 'localonly');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
// set up a demo table and some test data
$pdo->exec('CREATE TEMPORARY TABLE dt_table_tmp (id int, code int, connectid int, connectcode int)');
$sourcepath = 'sample.data.tmp';
$fh = fopen($sourcepath, 'wb') or die('!fopen(w)');
for($i=0; $i<10000; $i++) {
fputcsv($fh, array($i, $i%4, $i%100, $i%3), ';');
}
fclose($fh); unset($fh);
// test script
$foo = new Foo($pdo);
$foo->initiateInserts($sourcepath);
a few tips about speeding up mysql data import
check if your data really requires to be parsed, sometimes load data works just fine for csv
if possible, create an sql file first via php and then execute it with mysql command line client
use multivalue inserts
disable keys before inserting
multivalue insert statement is something like
INSERT INTO users(name, age) VALUES
("Sam", 13),
("Joe", 14),
("Bill", 33);
this is much faster than three distinct insert statements.
Disabling keys is important to prevent indexing each time you're executing an INSERT:
ALTER TABLE whatever DISABLE KEYS;
INSERT INTO whatever .....
INSERT INTO whatever .....
INSERT INTO whatever .....
ALTER TABLE whatever ENABLE KEYS;
further reading http://dev.mysql.com/doc/refman/5.1/en/insert-speed.html
Inspired by this question I would say you should do something similar. If you really have so many data, then a bulk import is the most appropriate approach for this. And you already have the data in a file.
Have a look at the LOAD DATA INFILE command.
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
If you are interested in the speed differences then read Speed of INSERT Statements.
E.g. you can do this:
$query = "LOAD DATA INFILE 'data.txt' INTO TABLE tbl_name
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES;
"
This will also ignore the first line assuming that it only indicates the columns.
How would you write a prepared MySQL statement in PHP that takes a differing number of arguments each time? An example such query is:
SELECT `age`, `name` FROM `people` WHERE id IN (12, 45, 65, 33)
The IN clause will have a different number of ids each time it is run.
I have two possible solutions in my mind but want to see if there is a better way.
Possible Solution 1 Make the statement accept 100 variables and fill the rest with dummy values guaranteed not to be in the table; make multiple calls for more than 100 values.
Possible Solution 2 Don't use a prepared statement; build and run the query checking stringently for possible injection attacks.
I can think of a couple solutions.
One solution might be to create a temporary table. Do an insert into the table for each parameter that you would have in the in clause. Then do a simple join against your temporary table.
Another method might be to do something like this.
$dbh=new PDO($dbConnect, $dbUser, $dbPass);
$parms=array(12, 45, 65, 33);
$parmcount=count($parms); // = 4
$inclause=implode(',',array_fill(0,$parmcount,'?')); // = ?,?,?,?
$sql='SELECT age, name FROM people WHERE id IN (%s)';
$preparesql=sprintf($sql,$inclause); // = example statement used in the question
$st=$dbh->prepare($preparesql);
$st->execute($parms);
I suspect, but have no proof, that the first solution might be better for larger lists, and the later would work for smaller lists.
To make #orrd happy here is a terse version.
$dbh=new PDO($dbConnect, $dbUser, $dbPass);
$parms=array(12, 45, 65, 33);
$st=$dbh->prepare(sprintf('SELECT age, name FROM people WHERE id IN (%s)',
implode(',',array_fill(0,count($parms),'?'))));
$st->execute($parms);
There is also the FIND_IN_SET function whose second parameter is a string of comma separated values:
SELECT age, name FROM people WHERE FIND_IN_SET(id, '12,45,65,33')
decent sql wrappers support binding to array values.
i.e.
$sql = "... WHERE id IN (?)";
$values = array(1, 2, 3, 4);
$result = $dbw -> prepare ($sql, $values) -> execute ();
Please take #2 off the table. Prepared statements are the only way you should consider protecting yourself against SQL injection.
What you can do, however, is generate a dynamic set of binding variables. i.e. don't make 100 if you need 7 (or 103).
I got my answer from: http://bugs.php.net/bug.php?id=43568.
This is my working mysqli solution to my problem. Now I can dynamically use as many parameters as I want. They will be the same number as I have in an array or as in this case I am passing the ids from the last query ( which found all the ids where email = 'johndoe#gmail.com') to the dynamic query to get all the info about each of these id no matter how many I end up needing.
<?php $NumofIds = 2; //this is the number of ids I got from the last query
$parameters=implode(',',array_fill(0,$NumofIds,'?'));
// = ?,? the same number of ?'s as ids we are looking for<br />
$paramtype=implode('',array_fill(0,$NumofIds,'i')); // = ii<br/>
//make the array to build the bind_param function<br/>
$idAr[] = $paramtype; //'ii' or how ever many ?'s we have<br/>
while($statement->fetch()){ //this is my last query i am getting the id out of<br/>
$idAr[] = $id;
}
//now this array looks like this array:<br/>
//$idAr = array('ii', 128, 237);
$query = "SELECT id,studentid,book_title,date FROM contracts WHERE studentid IN ($parameters)";
$statement = $db->prepare($query);
//build the bind_param function
call_user_func_array (array($statement, "bind_param"), $idAr);
//here is what we used to do before making it dynamic
//statement->bind_param($paramtype,$v1,$v2);
$statement->execute();
?>
If you're only using integer values in your IN clause, there's nothing that argues against constructing your query dynamically without the use of SQL parameters.
function convertToInt(&$value, $key)
{
$value = intval($value);
}
$ids = array('12', '45', '65', '33');
array_walk($ids, 'convertToInt');
$sql = 'SELECT age, name FROM people WHERE id IN (' . implode(', ', $ids) . ')';
// $sql will contain SELECT age, name FROM people WHERE id IN (12, 45, 65, 33)
But without doubt the solution here is the more general approach to this problem.
I had a similiar problem today and I found this topic. Looking at the answers and searching around the google I found a pretty solution.
Although, my problem is a little bit more complicated. Because I have fixed binding values and dynamic too.
This is the mysqli solution.
$params = array()
$all_ids = $this->get_all_ids();
for($i = 0; $i <= sizeof($all_ids) - 1; $i++){
array_push($params, $all_ids[$i]['id']);
}
$clause = implode(',', array_fill(0, count($params), '?')); // output ?, ?, ?
$total_i = implode('', array_fill(0, count($params), 'i')); // output iiii
$types = "ss" . $total_i; // will reproduce : ssiiii ..etc
// %% it's necessary because of sprintf function
$query = $db->prepare(sprintf("SELECT *
FROM clients
WHERE name LIKE CONCAT('%%', ?, '%%')
AND IFNULL(description, '') LIKE CONCAT('%%', ?, '%%')
AND id IN (%s)", $clause));
$thearray = array($name, $description);
$merge = array_merge($thearray, $params); // output: "John", "Cool guy!", 1, 2, 3, 4
// We need to pass variables instead of values by reference
// So we need a function to that
call_user_func_array('mysqli_stmt_bind_param', array_merge (array($query, $types), $this->makeValuesReferenced($merge)));
And the function makeValuesreferenced:
public function makeValuesReferenced($arr){
$refs = array();
foreach($arr as $key => $value)
$refs[$key] = &$arr[$key];
return $refs;
}
Links for getting this 'know-how': https://bugs.php.net/bug.php?id=49946, PHP append one array to another (not array_push or +), [PHP]: Error -> Too few arguments in sprintf();, http://no2.php.net/manual/en/mysqli-stmt.bind-param.php#89171, Pass by reference problem with PHP 5.3.1