I am trying to insert data from an Excel file into an Access 2007 database using PDO prepared statements. What the code is trying to do is to check if the value in the Excel sheet exists in the database and if not to add it but it doesn't do anything
Here is the code:
$conn = connect_to_impact();
//******************************************
//select statement
$select = 'SELECT < ? FROM < ?';
$select_query = $conn->prepare($select);
for ($h = 2; $h<count($clmn[0]); $h++){
$value = $clmn[0][$h];
// query execution
$select_query->execute(array($clmn[0][1], $clmn[0][1]));
$impact_no_result = $select_query->fetchAll();
//query result in multidimensional array
$impact_no_arr = impact_no_select($impact_no_result);
// create an indexed array of results
$impact_no_r = impact_no_indexed($impact_no_arr, $clmn[0][1]);
if(is_null($impact_no_r)){
$insert_impact_no = 'INSERT INTO < ? (< ?) VALUES (< ?)';
$simple_arr = [$clmn[0][1], $clmn[0][1], $value];
$insert_query = $conn->prepare($insert_impact_no);
$insert_query->execute($simple_arr);
}
// then if the value in the column is not in the indexed array insert it
elseif(!in_array($value, $impact_no_r)){
$insert_impact_no = 'INSERT INTO < ? (< ?) VALUES (< ?)';
$simple_arr = [$clmn[0][1], $clmn[0][1], $value];
$insert_query = $conn->prepare($insert_impact_no);
$insert_query->execute($simple_arr);
}
}
Indeed replacements cannot be used for table name and column names in PDO prepared statements. Here is the code that fixed the issueȘ
$conn = connect_to_impact();
//******************************************
//select statement
$select = "SELECT ". $clmn[0][1] . " FROM " . $clmn[0][1];
//repeat for each row the the excel sheet starting from row 2
for ($h = 2; $h<count($clmn[0]); $h++){
//take each value from each row from column A
$value = $clmn[0][$h];
// query execution
$impact_no_result = $conn->query($select);
if($impact_no_result){
$impact_no_result_ass = $impact_no_result->fetchAll();
}
//if there is no result in the select query
if($impact_no_result == false){
$insert_impact_no = "INSERT INTO ". $clmn[0][1] . "(" . $clmn[0][1] . ") " . "VALUES ('" . $value . "');";
$execute = $conn->query($insert_impact_no);
}
//if there is a result in the select query
else {
//put query result in array
$impact_no_arr = impact_no_select($impact_no_result_ass);
//if value is not in array insert it
if(!in_array($value, $impact_no_arr)){
$insert_impact_no = "INSERT INTO ". $clmn[0][1] . "(" . $clmn[0][1] . ") " . "VALUES ('" . $value . "');";
$execute = $conn->query($insert_impact_no);
}
}
}
Related
I am receiving an internal server error for my PHP script timing out. I am taking 8000 rows from mySQL database and breaking each row into 3 different tables.
I did some research on stack and ran into this as a solution:
set_time_limit(0);
ignore_user_abort(1);
Now, I am just not receiving the error anymore in chrome, but still not adding to the database.
Here is a little bit of pseudo of what I am doing:
$db = new myConnectDB();
$query = "SELECT * FROM tableName";
$result = $db->query($query);
$numRows = $result->num_rows;
//beerName', '$style' , '$brewery', '$abv','$ibu','$breweryID', '$icon', '$iconBrew' , '$lat', '$long' , '$beerRating')";
for ($i=0; $i < 8000 ; $i++) {
//fetch associate
$row = $result->fetch_assoc();
//get all 13 variables from rom
$db2 = new myConnectDB();
//insert some variables into 1st table
mysqli_close($db2);
//todo: insert BeerTable / check and
$db3 = new myConnectDB();
//insert variables into second table
mysqli_close($db3);
//todo: insert BreweryTable / check and update
$db4 = new myConnectDB();
//insert variables into third table.
mysqli_close($db4);
}
echo $i;
This is bad practice to establish and break connection inside a loop. Try to refactor your code this way:
$db = new myConnectDB();
$db2 = new myConnectDB();
$db3 = new myConnectDB();
$db4 = new myConnectDB();
$query = "SELECT * FROM tableName";
$result = $db->query($query);
$numRows = $result->num_rows;
$insertQuery1 = "INSERT INTO tbl1 (col1, col2, col3) VALUES";
$insertQuery2 = "INSERT INTO tbl2 (col1, col2, col3) VALUES";
$insertQuery3 = "INSERT INTO tbl3 (col1, col2, col3) VALUES";
for ($i=0; $i < 8000 ; $i++) {
//fetch associate
$row = $result->fetch_assoc();
//get all 13 variables from rom
//insert some variables into 1st table
$insertQuery1 .= '(' . $val1 . ',' . $val2 . ',' . $val3 ')';
//todo: insert BeerTable / check and
//insert variables into second table
$insertQuery2 .= '(' . $val4 . ',' . $val5 . ',' . $val6 ')';
//todo: insert BreweryTable / check and update
//insert variables into third table.
$insertQuery1 .= '(' . $val7 . ',' . $val8 . ',' . $val9 ')';
}
$db1->query($insertQuery1);
$db2->query($insertQuery2);
$db3->query($insertQuery3);
mysqli_close($db2);
mysqli_close($db3);
mysqli_close($db4);
In this concept you won't query 3 inserts for each row. You collect 3 big insert queries and execute them after the loop. So this code would work faster.
I'm trying to create a php function to split up the data into batches as it fails when I try to insert them fairly quickly.
I'm trying to insert thousands of records of user-data into a different format in the same database, later to be exported to a seperate database. However the query fails.
Based on comments and answers below I've updated the code to the following. Still fails, though.
The code inserting values:
function insertUsers( $users ){
error_reporting(E_ALL);
ini_set('display_errors',1);
global $pdo;
//insert into database
$i = 0;
$base = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered,
user_firstname, user_lastname ) VALUES ';
$sql = '';
var_dump($users);
while( $i < count( $users ) ){
$sql = $sql . ' ("' .
$users[$i]['user_login'] . '", "' .
$users[$i]['user_pass'] . '", "' .
$users[$i]['user_email'] . '", "' .
$users[$i]['user_registered'] . '", "' .
$users[$i]['meta_value']['first_name'] . '", "' .
$users[$i]['meta_value']['last_name'] . '")';
if (!( $i % 25 === 0 )){
$sql = $sql . ', ';
}
if ($i % 25 === 0) {
//execute $base + $query here
$sql = $base . $sql;
$query = $pdo->prepare( $sql );
echo 'check query: <br />';
print_r( $query );
if( $query->execute() ){
echo '50 users succesfully added to the database';
} else {
echo 'Query failed: ';
print_r( $pdo->errorInfo() );
echo '<br />';
}
$sql = ''; //Re-init query string
}
$i++;
}
if ( strlen( $sql ) > 0 ) { // Execute remainder, if any
//execute $base + $query here
$sql = $base . $sql;
$query = $pdo->prepare( $sql );
echo 'check query: <br />';
print_r($query);
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
echo 'Query failed: ';
print_r( $pdo->errorInfo() );
echo '<br />';
}
}
}
check query:
PDOStatement Object ( [queryString] => INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname ) VALUES ("John Smith", "4\/\/350M3 P4sS\/\/0r|)", "john.smith#greatmail.com", "2013-04-11 11:18:58", "John", "Smith") )
Query failed: Array ( [0] => 00000 [1] => [2] => )
Tried it with a %25 and %50, both don't work. Keep getting the 00000 error which is supposed to lead to victory (success, though for me it still fails, nothing in the DB)
I'd do it manually if I had the time but this won't be a one-time event so I need a solution to this issue. Is there a good way to split up the query into batches (and how?) that would allow this to be repeated and queries to be executed one after the other? I've been looking at a whole bunch of questions on SO (and elsewhere) already and can't find one that suits my needs.
UPDATE - has been answered, need a small modification as shown below:
if (!( $i % 25 === 0 )){
if(!( $i == ( count( $users ) - 1 ))){
$sql = $sql . ', ';
}
}
You're mixing PDO with mysql_ functions. Pick one and follow the respective library's error handling.
It would also be beneficial print out your $sql to yourself to see if it's formatted correctly. Additionally, if you're handling POSTed data, you will want to use prepared statements,
You might need to increase the max_allowed_packet value which defaults to 1Mb. If you want to split up the query in batches you can do so using the modulus operator.
$base = 'INSERT INTO ...';
$sql = '';
while( $i < count( $users ) ){
$sql = $sql . ' ("' ... //etc.
if ($i % 50 === 0) {
//execute $base + $qry here
$sql = ''; //Re-init query string
}
}
if (strlen($qry)>0) { // Execute remainder, if any
//execute $base + $query here
}
Or as an array (described here):
$base = 'INSERT INTO ...';
$sql = array();
while( $i < count( $users ) ){
$sql[] = ' ("' ... //etc.
if ($i % 50 === 0) {
//execute $base + implode(',', $sql) here
$sql = array(); //Re-init query string
}
}
if (sizeof($qry)>0) { // Execute remainder, if any
//execute $base + implode(',', $sql) here
}
Also please make sure you're using prepared statements correctly so you're not vulnerable to SQL injections.
Finally: you might need to enable error reporting so failures won't be silent; if they're still silent after enabling error reporting (e.g. error_reporting(-1);) you might need to set MySQL to strict mode (not sure if that will help). If they still fail silently file a bugreport.
Edit
Oh, I missed the fact that you're mixing mysql_ and PDO; that will probably be the reason why you're not seeing any errors... D'uh. Go read the manual on PDO error handling.
#Downvoters: If you're downvoting at least have the decency to leave a comment on why.
Is there a reason why you want to do it as one single statement? Why not just iterate through the users and insert each one with something like:
while($i < count( $users )) {
$sql = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname ) VALUES ';
$sql = $sql + "(";
$sql = $sql + "'" . $users[$i]['user_login'] . '",';
$sql = $sql + "'" . $users[$i]['user_pass'] . '",';
$sql = $sql + "'" . $users[$i]['user_email'] . '",';
$sql = $sql + "'" . $users[$i]['user_registered'] . '",';
$sql = $sql + "'" . $users[$i]['meta_value']['first_name'] . '",';
$sql = $sql + "'" . $users[$i]['meta_value']['last_name'] . '"';
$sql = $sql + ")";
$query = $pdo->prepare( $sql );
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
die ("Query failed: " . $pdo->errorInfo());
}
}
Even better is to prepare the statement once and then bind the parameters. As pointed out by RobIII, you can introduce SQL injection vulnerabilities by building up your SQL statements as strings so instead, you could do something like:
$sql = 'INSERT INTO tth_user_accounts (user_login, user_pass, user_email, user_registered, user_firstname, user_lastname) ';
$sql += ' VALUES (:user_login, :user_pass, :user_email, :user_registered, :user_firstname, :user_lastname)';
$query = $pdo->prepare( $sql );
while ($i < count($users)) {
$query->bindParam(":user_login", $users[$i]['user_login']);
$query->bindParam(":user_pass", $users[$i]['user_pass']);
$query->bindParam(":user_email", $users[$i]['user_email']);
$query->bindParam(":user_registered", $users[$i]['user_registered']);
$query->bindParam(":user_firstname", $users[$i]['user_firstname']);
$query->bindParam(":user_lastname", $users[$i]['user_lastname']);
if( $query->execute() ){
echo 'User succesfully added to the database';
} else {
die ("Query failed: " . $pdo->errorInfo());
}
}
Another issue you may be encountering is a simple fact that the user's name / information may contain invalid characters and killing your command. Take for instance a person's last name is "O'Maley". When building your string of values and wrapping the name with hard quotes, you would get
'O'Maley'
I ran into something similar to this about 8 years ago and had to validate certain values within the data. Once you find it, you'll know how to correct each respective value, then go back to batch mode.
Also, consider someone supplying a bogus value of a bogus / bad value of "--" which indicates rest of line is a comment... or even a ";" to indicate end of statement. That is most likely what you are running into.
As RobIII responded, and I was too quick in my response, don't test one at a time, but query the data an look at it for such anomolies / bad escape posts in the data. Fix what you need to BEFORE trying the insert.
im having a problem inserting some variables into my database, here's my code.
preg_match('/Tel\. P(.)liza :(.*?)Localidad/s', $a->output(), $tel);
echo "Tel. Poliza: " . $tel[2] . "<br><br>";
$tel = $tel[2];
preg_match('/Tel\. Expt :(.*?)D\.P\./s', $a->output(), $tel1);
$tel1 = $tel1[1];
preg_match_all('/\\b[0-9]{9}\\b/s', $tel1, $tel2);
$tel2 = implode(" / ", $tel2[0]);
echo "Tel. Expt: " . $tel2 . "<br><br>";
$conn = new PDO('mysql:host=localhost;port=3306;dbname=something', 'something', 'something');
$sql = "INSERT INTO clients (tel poliza, tel expt) VALUES ( ? , ? )";
$q = $conn->prepare($sql);
$q->execute(array ($tel, $tel2));
i can echo this variables(they are just numbers from a pdf file) but when i run the sql query nothing is inserted into the db. i have tried this with other variables in my code and they get inserted but when i had this 2 to the query nothing is inserted at all.
Your table column names have spaces in then therefore you have to put them in back quotes
$sql = "INSERT INTO clients (`tel poliza`, `tel expt`) VALUES ( ? , ? )";
I'm new to PHP, and I do not it's syntax and principles very well.
I have such code:
function exportFromTransbase($table_name) {
//$odbc_query = "SELECT * FROM " . $table_name. " WHERE ((CDS_CTM subrange(248 cast integer) = 1) AND (CDS_LNG_ID = 16))";
$odbc_query = "SELECT * FROM " . $table_name. "";
$data = odbc_exec($this->odbc_id, $odbc_query);
odbc_longreadlen($data, 10485760);
$oufile=fopen($table_name.".sql", 'w') or die("error writing file");
$q1='INSERT INTO `' . substr($table_name, 4) . '` VALUES';
fwrite($oufile, $q1);
while($row = odbc_fetch_array($data))
{
foreach($row as $key => $value) {
$keys[] = "`" . $key . "`";
if ($value == ""){
$value = 'NULL';
$values[] = "" . mysql_real_escape_string($value) . "";
}
else{
$values[] = "'" . mysql_real_escape_string($value) . "'";
}
//echo "\n \n ololo ".$value;
}
$mysql_query = "\n (".implode(",", $values).")," ;
fwrite($oufile, $mysql_query);
//mysql_query($mysql_query);
set_time_limit(0);
unset($keys);
unset($values);
unset($row);
}
$stat = fstat($oufile);
ftruncate($oufile, $stat['size']-1);
fseek($oufile, 0, SEEK_END);
fwrite($oufile, ";".$r);
//} while ($r < 5 );
fclose($oufile);
if ($mysql_query){
print "Ýêñïîðò äàííûõ èç òàáëèöû " . $table_name . " çàâåðøåí!";
//strtolower(substr($table_name, 4))
}
}
what and where i need to custom, so that i export all table fields except one, for example called Size, i must insert in db is this field nulls....
Also if it is easy, how to split my sql query in batches of 5000 rows? so insert (5000 rows) then insert another 5000....
But first i need to export all fields, except one...
To the best of my knowledge, it can't be possible.
You can use simply:
SELECT field1, field2, field3, field4 FROM table
See Select all columns except one in MySQL?
And if you have more fields in your table then you can use * only.
At the time of insert you can make a condition which ignores the field which you don't want in new table.
Can you use
"select * from {table} where {sth} != {condition}"
Then you can fetch all data except one. The one is excrpted follow your condition.
Meanwhile,if you want to insert by 5000, you can read 5000 lines one time and insert into.
I'm inserting data from oracle database to mysql database with php than insert and update, but I have a strange behaviour. The table oracle have 20851 records.
The problem is, in the record x, the query to mysql return me a empty result object, but the same query executed on MySQL returned objects with data.
With the following code I can insert and update data from oracle to mysql.
$stid = oci_parse($conn, 'SELECT * FROM B_PROGRAMA_EVALUACION_BIPS'); //oracle db
oci_execute($stid); //oracle db
while ($row = oci_fetch_array($stid, OCI_ASSOC+OCI_RETURN_NULLS)) {
$sql = "select ID_ITEM, ID_PROGRAMA from B_PROGRAMA_EVALUACION_BIPS WHERE ID_ITEM=".$row['ID_ITEM']." AND ID_PROGRAMA=".$row['ID_PROGRAMA'];
$result = $db->query($sql);
$rows = mysqli_fetch_array($result);
if ($rows['ID_ITEM'] == $row['ID_ITEM'] && $rows['ID_PROGRAMA'] == $row['ID_PROGRAMA']) {
$sql = "UPDATE B_PROGRAMA_EVALUACION_BIPS SET ID_PROGRAMA='".$row['ID_PROGRAMA'] . "', ANO='" . $row['ANO'] . "', COPIA='" . $row['COPIA'] . "', TIPO_PROGRAMA_ER='"
. $row['TIPO_PROGRAMA_ER'] . "', EVALUACION='".mysqli_real_escape_string($db, $row['EVALUACION'])."', ID_CARACTERICACION='".$row['ID_CARACTERICACION']."', NOTA='".$row['NOTA']."' WHERE ID_ITEM=".$row['ID_ITEM'];
} else {
$sql = "INSERT INTO B_PROGRAMA_EVALUACION_BIPS VALUES('".$row['ID_PROGRAMA'] . "','" . $row['ANO'] . "','" . $row['COPIA'] . "','"
. $row['ID_ITEM'] . "','".$row['ID_CARACTERICACION']."','".mysqli_real_escape_string($db, $row['EVALUACION'])."','".$row['TIPO_PROGRAMA_ER']."','".$row['NOTA']."')";
}
$result = $db->query($sql);
if ($result != 1) {
$resultado['B_PROGRAMA_EVALUACION_BIPS'] = $resultado['B_PROGRAMA_EVALUACION_BIPS'] + 1;
}
}
If I delete this line the second execute query, the first query working perfectly.
$result = $db->query($sql);
You should add single quotes around the values:
$sql = "select ID_ITEM, ID_PROGRAMA from B_PROGRAMA_EVALUACION_BIPS WHERE ID_ITEM='".$row['ID_ITEM']."' AND ID_PROGRAMA='".$row['ID_PROGRAMA']"'";