bulk data insert sql issue - php

i am importing a huge data from a csv file into my database, but the issue is , if there is a error in the sql , my insertion stops , thus making my bulk insertion useless.i have to go back , delete the uploaded data and remove that entry which is causing issue in my insertion and start again . i want programme to skip it and continue insertion, i know i have to apply try and catch, i have applied it in my algo but i cnt understand how to use so that it continues its insertion .
here is my code
$num=35; //number of columns
$dum=true; // a check
$sum=0;// count total entries
if (($handle = fopen($_FILES['file']['name'], 'r')) !== FALSE) {
while (($data = fgetcsv($handle, 10000, ',')) !== FALSE)
{
if($dum)
{
for ($qwe=0; $qwe < $num; $qwe++) { //searching the columns exact position in case they have been changed
if($data[$qwe]=='ID')
{$a=$qwe;}
.
.
.
.
.else if($data[$qwe]=='PowerMeterSerial')
{$aa=$qwe;}
else if($data[$qwe]=='Region')
{$ab=$qwe;}
else if($data[$qwe]=='Questions')
{$ac=$qwe;}
}
}
if(!$dum)
{
for($qwe=0;$qwe<$num;$qwe++)
{
if($qwe==7||$qwe==8)
{
if($qwe==7){$asd=$data[$qwe];}
$data[$qwe]=date('Y-m-d h-i-s',strtotime($data[$qwe]));
}
}
$data[57]=date('Y-m-d ',strtotime($asd));try {
$sql="INSERT INTO pm (ID, .......... Questions, dateofdata ,Unsuccessful) VALUES ('$data[$a]','$data[$b]','$data[$c]','$data[$d]','$data[$e]','$data[$f]','$data[$g]','$data[$h]' ,'$data[$i]','$data[$j]','$data[$k]', '$data[$l]', '$data[$m]','$data[$n]','$data[$o]','$data[$p]','$data[$q]','$data[$r]','$data[$s]','$data[$t]','$data[$u]','$data[$v]','$data[$w]', '$data[$x]','$data[$y]','$data[$z]','$data[$aa]','$data[$ab]','$data[$ac]','$data[57]','$data[30]')";
if (!mysql_query($sql,$con))
{
die('Error: ' . mysql_error(). $sql);
}
}
catch (Exception $e)
{
echo'<br>'; echo $sql;
}
$sum++;
}$dum=false;
}
}
?>
kindly note there is no issue in uploading algorithme or sql , its when input data does not match the data type than sql generates a error , for that i am trying try and catch .. please help

Change your die() command to a print(). You'll see what the error was, and the script will move on to the next line.
Given the structure of your code, I'm guessing it'll blow up anytime you're inserting a string (particularly with quotes inside it), causing SQL syntax errors. You MUST pass each text field from your csv through mysql_real_escape_string() BEFORE you insert those values into the query string.

Related

How to execute 2 SQL queries one after the other using mysqli_multi_query in PHP

I was trying insert values simultaneously into MySQL database using mysqli_multi_query but it's not executing and going to if part showing alert message stating Record Insertion Failed.
Below is my PHP code with query
while (($emapData = fgetcsv($file, 10000, ",")) !== FALSE) {
$sql_tableone = "INSERT into inverterlog (`id`,`timestamp`,`irradiance`,`ambienttemp`,`photovoltaictemp`,`pv1voltage`,`pv2voltage`,`pv3voltage`,`pv1current`,`pv2current`,`pv3current`,`pv1power`,`pv2power`,`pv3power`,`pv1energy`,`pv2energy`,`pv3energy`,`gridvoltagegv1`,`gridvoltagegv2`,`gridvoltagegv3`,`gridcurrentgc1`,`gridcurrentgc2`,`gridcurrentgc3`,`gridpowergp1`,`gridpowergp2`,`gridpowergp3`,`sumofapparentpower`,`gridpowertotal`,`gridenergyge1`,`gridenergyge2`,`gridenergyge3`,`socounter`,`gridcurrentdcgc1`,`gridcurrentdcgc2`,`gridcurrentdcgc3`,`gridresidualcurrent`,`gridfrequencymean`,`dcbusupper`,`dcbuslower`,`temppower`,`tempaux`,`tempctrl`,`temppower1`,`temppowerboost`,`apparentpowerap1`,`apparentpowerap2`,`apparentpowerap3`,`sovalue`,`reactivepowerrp1`,`reactivepowerrp2`,`reactivepowerrp3`,`opmode`,`latestevent`,`pla`,`reactivepowermode`,`overexcitedunderexcited`,`reactivepowerabs`,`inverter`)
values('','$newDate','$emapData[1]','$emapData[2]','$emapData[3]','$emapData[4]','$emapData[5]','$emapData[6]','$emapData[7]','$emapData[8]','$emapData[9]','$emapData[10]','$emapData[11]','$emapData[12]','$emapData[13]','$emapData[14]','$emapData[15]','$emapData[16]','$emapData[17]','$emapData[18]','$emapData[19]','$emapData[20]','$emapData[21]','$emapData[22]','$emapData[23]','$emapData[24]','$emapData[25]','$emapData[26]','$emapData[27]','$emapData[28]','$emapData[29]','$emapData[30]','$emapData[31]','$emapData[32]','$emapData[33]','$emapData[34]','$emapData[35]','$emapData[36]','$emapData[37]','$emapData[38]','$emapData[39]','$emapData[40]','$emapData[41]','$emapData[42]','$emapData[43]','$emapData[44]','$emapData[45]','$emapData[46]','$emapData[47]','$emapData[48]','$emapData[49]','$emapData[50]','$emapData[51]','$emapData[52]','$emapData[53]','$emapData[54]','$emapData[55]','$inverter')";
$sql_tabletwo = "INSERT into data (`id`,`timestamp`,`gridpowertotal`,`inverter`) values ('','$newDate','$emapData[26]','$inverter')";
$sql= $sql_tableone.";".$sql_tabletwo;
$result = mysqli_multi_query( $con,$sql);
if (! $result ) {
echo "<script type=\"text/javascript\">
alert(\"multi query Record Insertion Failed.\");
</script>";
}
fclose($file);
}
//throws a message if data successfully imported to mysql database from excel file
echo "<script type=\"text/javascript\">
alert(\"CSV File has been successfully Imported.\");
window.location = \"four.php\"
/</script>";
//close of connection
mysqli_close($con);
}
}
If the id column is auto-incremented in the tables, then omit the column (and the empty value) from your query. Alternatively, you can use NULL (not quoted) if you are going to mention the column in your query.
Many, many people struggle to find the errors with their mysqli_multi_query() code block because it is not set up to properly output affected rows and errors.
I recommend having a look at a general purpose code block that will help to isolate troublesome queries, and read this answer.
It also looks like you are while-looping mysqli_multi_query()'s two queries. For efficiency, I recommend building up the full array of queries, finishing the loop, then calling mysqli_multi_query() only once.
p.s. Do any of your insert values have quotes in them? Prepared statements would help with that issue. Use the code block from my link and check the error message.
UPDATE: Here is my spoon-fed answer (Of course, I didn't actually test it before posting):
// I assume $newdate is not user declared and considered safe.
// I am using NULL for your auto-incremented primary key `id`.
// If you want to be assured that each pair has an identical `id`, perhaps use LAST_INSERT_ID() on second query of pair.
// establish variables for future use
$inverterlog_sql="INSERT INTO `inverterlog` (`id`,`timestamp`,`irradiance`,`ambienttemp`,`photovoltaictemp`,`pv1voltage`,`pv2voltage`,`pv3voltage`,`pv1current`,`pv2current`,`pv3current`,`pv1power`,`pv2power`,`pv3power`,`pv1energy`,`pv2energy`,`pv3energy`,`gridvoltagegv1`,`gridvoltagegv2`,`gridvoltagegv3`,`gridcurrentgc1`,`gridcurrentgc2`,`gridcurrentgc3`,`gridpowergp1`,`gridpowergp2`,`gridpowergp3`,`sumofapparentpower`,`gridpowertotal`,`gridenergyge1`,`gridenergyge2`,`gridenergyge3`,`socounter`,`gridcurrentdcgc1`,`gridcurrentdcgc2`,`gridcurrentdcgc3`,`gridresidualcurrent`,`gridfrequencymean`,`dcbusupper`,`dcbuslower`,`temppower`,`tempaux`,`tempctrl`,`temppower1`,`temppowerboost`,`apparentpowerap1`,`apparentpowerap2`,`apparentpowerap3`,`sovalue`,`reactivepowerrp1`,`reactivepowerrp2`,`reactivepowerrp3`,`opmode`,`latestevent`,`pla`,`reactivepowermode`,`overexcitedunderexcited`,`reactivepowerabs`,`inverter`) VALUES (NULL,$newdate";
$data_sql="INSERT INTO `data` (`id`,`timestamp`,`gridpowertotal`,`inverter`) VALUES (NULL,'$newDate'";
$tally=0;
$x=0;
// build all queries
while(($emapData=fgetcsv($file,10000,","))!==false){
++$x;
$sql[$x]=$inverterlog_sql; // start first query of pair
for($i=1; $i<56; ++$i){
$sql[$x].=",'".mysqli_real_escape_string($con,$emapData[$i])."'";
}
$sql[$x].=",'".mysqli_real_escape_string($con,$inverter)."');"; // end first query of pair
$sql[$x].="$data_sql,'".mysqli_real_escape_string($con,$emapData[26])."','".mysqli_real_escape_string($con,$inverter)."')"; // whole second query of pair
fclose($file);
}
// run all queries
if(mysqli_multi_query($con,implode(';',$sql)){
do{
$tally+=mysqli_affected_rows($con);
} while(mysqli_more_results($con) && mysqli_next_result($con));
}
// assess the outcome
if($error_mess=mysqli_error($con)){
echo "<script type=\"text/javascript\">alert(\"Syntax Error: $error_mess\");</script>";
}elseif($tally!=$x*2){ // I don't expect this to be true for your case
echo "<script type=\"text/javascript\">alert(\"Logic Error: Only $tally row",($tally!=1?"s":"")," inserted\");</script>";
}else{
echo "<script type=\"text/javascript\">alert(\"CSV File has been successfully Imported.\"); window.location = \"four.php\"/</script>";
}
mysqli_close($con);

Mysql does't insert query using php

This could sound stupid (maybe because it is) but I'm having so much problem in an insert query, this is the thing, I have in my first page a query that inserts into a table, the problem is that in my page2, I have EXACTLY the same query, but in this page doesn't work, the weirdest thing is that I echoed a message in the true statement of the query and it echoes, meaning that the query supposedly is already made, but is not, because when I check into the DB, is not there, and other thing! if I want to update the values, it works!, but not in create, I don't know why, but is bothering me a lot!, if someone could help me i really would appreciated, thanks.
Here is the code:
if($radio==2){
echo $_SESSION["in"];
$sqlb = "SELECT * FROM table WHERE idus='$idus';";
$resb = mysqli_query($con, $sqlb);
$res_b = mysqli_fetch_array($resb);
if( !$res_b) {//if not exist on table, create
$sqly="INSERT INTO table (x,y,z)
VALUES ('$x','$y','$z');";
if ($con->query($sqly) === TRUE) {
$_SESSION["in"]=1;
} else {
echo "Error: " . $sqly . "<br>" . $con->error;
}
}else{ //update if exist on table
$sqly="UPDATE table
SET y='$y',z='$z'
WHERE idus='$idus';";
if ($con->query($sqly) === TRUE) {
} else {
echo "Error: " . $sqly . "<br>" . $con->error;
}
}
header('Location: page2.php');
}
The weird thing of all is that passes through true when doing the query and do nothing, also when pasting the exact same query directly into the DB, it works.
It's hard to tell because we can't see all the code, but I think you should replace these lines, using as a reference the update query a few lines below:
$sqly="INSERT INTO table (x,y,z)
VALUES (x,y,z);";
To:
$sqly="INSERT INTO table (x,y,z)
VALUES ('$x','$y','$z');";
An explanation: x, y and z are not defined as SQL variables, but in another query below you use them as php variables, so I updated the code putting them as they should. Hope it helps.
Check your column names and properties carefully. may be x column is something wrong if update which have y,z is working fine.
I did not understand the question correctly but by looking at your code. You could achieve same behaviour by using INSERT...ON DUPLICATE query. see more here
if($radio==2){
echo $_SESSION["in"];
$sqlb = 'INSERT INTO table (x,y,z) VALUES ($x, $y, $z) ON DUPLICATE KEY UPDATE x=$x, y=$y, z=$z';
if ($con->query($sqlb) === TRUE) {
$_SESSION["in"]=1;
} else {
echo "Error: " . $sqlb . "<br>" . $con->error;
}
header('Location: page2.php');
}
Check the variables $x ,$y, $z variables .
If its is not empty then your code works otherwise insertion don't work
I don't know what actually was the problem, but I fix it by deleting all my DB structure and re do it again, thanks for all your help!

Insert loop from CSV bug using PHP PDO

I'm tryin to insert datas (160,000+ rows) using INSERT INTO and PHP PDO but i have a bug.
When I launch the PHP script, i see more than the exact number of lines in my CSV inserted in my database.
Can someone say me if my loop is not correct or something ?
Here the code I have :
$bdd = new PDO('mysql:host=<myhost>;dbname=<mydb>', '<user>', '<pswd>');
// I clean the table
$req = $bdd->prepare("TRUNCATE TABLE lbppan_ticket_reglements;");
$req->execute();
// I read and import line by line the CSV file
$handle = fopen('<pathToMyCsvFile>', "r");
while (($data = fgetcsv($handle, 0, ',')) !== FALSE) {
$reqImport =
"INSERT INTO lbppan_ticket_reglements
(<my31Columns>)
VALUES
('$data[0]','$data[1]','$data[2]','$data[3]','$data[4]','$data[5]','$data[6]','$data[7]','$data[8]',
'$data[9]','$data[10]','$data[11]','$data[12]','$data[13]','$data[14]','$data[15]','$data[16]',
'$data[17]','$data[18]','$data[19]','$data[20]','$data[21]','$data[22]','$data[23]','$data[24]',
'$data[25]','$data[26]','$data[27]','$data[28]','$data[29]','$data[30]')";
$req = $bdd->prepare($reqImport);
$req->execute();
}
fclose($handle);
The script works a little because datas are in the table but i dunno why it bugs and inserts more datas. I think maybe, due to the file size (18 Mo) maybe the script crash and attempts to relaunch inserting same rows again.
I can't use LOAD DATA on the server I'm using.
Thanks for your help.
This is not an answer but adding this much into comments is quite tricky.
Start by upping the maximum execution time
If that does not solve your issue, start working your way through the code line by line and handle every exception you can think of. For example, you are truncating the table BUT you say you have loads more data after execution, could the truncate be failing?
try {
$req = $bdd->prepare("TRUNCATE TABLE lbppan_ticket_reglements;");
$req->execute();
} catch (\Exception $e) {
exit($e->getMessage()); // Die immediately for ease of reading
}
Not the most graceful of try/catches but it will allow you to easily spot a problem. You can also apply this to the proceeding query...
try {
$req = $bdd->prepare($reqImport);
$req->execute();
} catch (\Exception $e) {
exit($e->getMessage());
}
and also stick in some diagnostics, are you inserting 160k rows? You could optionally echo out $i on each loop and see if you can spot any breaks or abnormalities.
$i = 0;
while (($data = fgetcsv($handle, 0, ',')) !== FALSE) {
// ... your stuff
$i++;
}
echo "Rows inserted " . $i . "\n\n";
Going beyond that you can the loop print out the SQL content for you to look at manually, perhaps its doing something weird and fruity.
Hope that helps.
Assuming $data[0] is the unique identifier then you can try this to spot the offending row(s):
$i = 0;
while (($data = fgetcsv($handle, 0, ',')) !== FALSE) {
echo 'Row #'.++$i.' - '.$data[0];
}
Since you are not using prepared statements, it is very possible that one of the $data array items are causing a double-insert or some other unknown issue.

UPDATE MySQL table from a CSV using PHP

I found and followed the directions contained within this StackOverflow thread: Update MySql Table from CSV using PHP
I've got an error somewhere that I'm unable to detect, I think there's a problem with my query, which works fine in actual MySQL but seems to not quite translate to PHP.
In short, I'm trying to UPDATE the value of several rows within a single table (catalog_product_entity_varchar) with CSV column $data[1], but only where certain skus are concerned AND attribute_id = 523 AND entity_id matches $data[0] of my CSV. Here's my code (actual PW/username, etc, obviously removed)
$con=mysqli_connect("localhost","username","password","some_db");
if (!$con){
die('Could not connect: ' . mysql_error());
}
if (($file = fopen("upload.csv", "r")) !== FALSE) {
while (($data = fgetcsv($file)) !== FALSE) {
$sql = "UPDATE catalog_product_entity_varchar
JOIN catalog_product_flat_1
ON catalog_product_flat_1.entity_id = catalog_product_entity_varchar.entity_id
SET catalog_product_entity_varchar.value='{$data[1]}'
WHERE catalog_product_entity_varchar.entity_id='{$data[0]}'
AND catalog_product_entity_varchar.attribute_id = 523
AND (catalog_product_flat_1.sku LIKE '%PR%'
OR catalog_product_flat_1.sku LIKE '%PT%'
OR catalog_product_flat_1.sku LIKE '%PF%')";
if (mysql_query($con,$sql)) {
echo "Updated!";
} else {
echo "Error updating " . mysql_error();
}
}
}
fclose($file);
It simply returns "Error updating" for every line of the spreadsheet. This query, when simply done in MySQL (without the PHP) and modified to have actual values instead of $data[1] or $data[0] works just fine. What am I missing?
If you're unclear of what I'm trying to achieve, I did post this question yesterday (trying to do it via pure mySQL) and there's more context here - https://stackoverflow.com/questions/21170245/updating-a-joined-table-in-mysql-from-a-csv
Wow.
So I feel stupid. Apparently mixing mysqli_connect and mysql_query doesn't work very well. Adding the "i" to the "mysql" of mysql_query solved it. Thanks for looking everyone!

PHP MySQL INSERT query no longer works (was working yesterday)

I have a problem with an INSERT query.
Here is the problem:
Yesterday I was using this code to upload data, it was working fine. Today, when I hit submit on the form, it just shows a blank page. No errors, just blank. Nothing in the error log. All SELECT queries are working fine, so the SELECT Count(id) query still works.
Here is what I have tried:
Re-uploading to server
syntax adjustments eg '".$v.'" instead of '$v'
adding print lines to check that none of the variables are null. All is okay, all data is present just before the INSERT query.
Test insert via PHP my admin, all okay
The function call is correct - it is and remains unchanged from Yesterday
The function takes a list of species, a family and a genus, then adds them to the database.
Here is the code (the un-santised version - both were working yesterday):
error_reporting(E_ALL);
ini_set('display_errors', '1');
function submit($family, $genus, $species){
//require statement
require 'databaseConnect.php';
//get num of species
if(!($result = mysql_query("SELECT Count(id) as num FROM speciesList", $connection))) mysql_error();
$nums = mysql_fetch_row($result);
$num=$nums[0];
//parse species
$holder="";
$array = Array();
while(strlen($species)!=0){
if($species[0]==';'){
$array[] = $holder;
$holder="";
}else{
$holder = $holder . $species[0];
}
$species=substr($species, 1);
}
foreach($array as $v){
$num++;
if(!(mysql_query("INSERT INTO speciesList VALUES($num, '$family', '$genus', '.$v')", $connection))){
mysql_error();
}else{
print "success ";
}
}
mysql_close($connection);
}
Thank you very much in advance, this problem is rather mysterious to me!
Em
You never ever want to use string replacement with parameters to build SQL statements as it leaves you vulnerable to SQL injection attacks. Instead, bind your parameters.
Your code isn't returning an error because you call mysql_error() and ignore the return value. It returns the error string, so you want your code to be more like this:
if(!(mysql_query("INSERT INTO speciesList VALUES($num, '$family', '$genus', '.$v')", $connection))){
print_r( mysql_error() );
}else{
print "success ";
}
If you need help understanding the error once you see it, please post it here.
if(!(mysql_query("INSERT INTO speciesList VALUES($num, '$family', '$genus', '.$v')", $connection))){
echo mysql_error();
}else{
There maybe other things that are amiss, but the mysql_error() function returns a string. Your script needs to take some action to display the string.
I find that a better way of doing this is to construct the variables separately, so they can be displayed. And I'm not sure you want that dot before $v. See if this makes sense to you.
foreach($array as $v)
{
$num++;
$sql = "INSERT INTO speciesList VALUES($num, '$family', '$genus', '$v')";
$res = mysql_query($sql);
if (!$res) die("FAIL: $sql BECAUSE: " . mysql_error());
echo "<br/>SUCCESS: $sql";
}

Categories