Import csv into database with php and with NOT EXIST - php

SQL agent (in SSMS) is exporting output.csv (from query). Then it moves this csv file (comma delimited) into another server, where I run php script, which is automatically importing (task scheduler) this csv file into mysql database (xampp - phpmyadmin).
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator));
echo "Loaded a total of $affectedRows records from this csv file.\n";
I would like to add something like: if this value U_ScID exists dont import this value.
WHERE T3.U_CreateDate > DATEADD(hour,-12,GETDATE()-1)
AND NOT EXISTS (SELECT T5.U_ScID FROM [xxx].[dbo].[table] where T5.U_ScID = T0.U_ScId)
Do you have any idea, how should I do it?

Related

HTTP 504 timeouts by running this PHP script

I have a PHP script below that I'm running every hour using a cronjob. Sometimes this script runs without any errors and other times it throws a 504 Gateway Timeout.
Can someone please help to improve the execution and efficiency of this script? I believe it can be written cleaner, but not sure where to start.
This is on a shared server and I do not have much control over the settings. Max timeout is 120.
Cronjob:
curl www.mysite.com/apps/script.php
PHP script:
<?php
set_time_limit(500);
$databasehost = "localhost";
$databasename = "foodb";
$databasetable = "footable";
$databaseusername="foouser";
$databasepassword = "foopass";
$fieldseparator = ",";
$lineseparator = "\n";
$csvfile = "inventory.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
$sql = "TRUNCATE TABLE $databasetable";
$command = $pdo->prepare($sql);
$command->execute();
echo "Removed records from $databasename.\n<br />";
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)."IGNORE 1 LINES");
echo "Loaded a total of $affectedRows records from this csv file.\n";
/****/
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
$sql = $pdo->exec("
UPDATE wp37_pmxi_posts
JOIN wp37_postmeta USING (post_id)
JOIN inventoryImport ON wp37_pmxi_posts.unique_key = inventoryImport.sku
SET meta_value = inventoryImport.qty
WHERE meta_key = '_stock'");
echo "Updated $affectedRows records after the import.\n";
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
?>
Thanks for your time and I appreciate any help.

I had the exact same issue last week and even though I am on a VPS server with a full root access, I didn't want to mess about with the files in the etc/ folder or php.ini or any other PHP settings...
So I came up with my own solution..
Please note this solution and this answer intended to just give you and other people an idea of how you could solve this issue without fiddling with server settings...
The solution was quite simple actually...
First i created an extra column in the MYSQL database with a pre-defined value of 0. You can name it anything you want.
What I did next was basically LIMIT my SELECT to 100.
something like:
SELECT * FROM mytable WHERE newColumn=0 LIMIT 100
And then I did whatever i wanted with this 100 selected items and every time I did that i UPDATE the database and changed the newColumn=1
And I repeated the same process until there is no more data in the MYSQL database with newColumn=0.
once I reach this stage, I UPDATE the table again and reset the newColumn back to 0.
Note that I run the process above on cron job.
This way your PHP won't get killed as you are only selecting 100 rows from the MYSQL.
hope this helps.

load data infile function with PHP

I have tried to import CSV file into MySQL database using PHP and MySQL load data Infile function.The query looks correct but still there is an error, while embedding it inside php code. Would you Please help me.
<?php
//connect to the database
$host='localhost';
$username='';
$password='';
$database_name='cricket';
$connection=mysql_connect($host,$username,$password,$database_name);
if(!$connection)
{
echo "no connection made";
}
//
else {
$query="LOAD DATA INFILE 'cricket.csv'
INTO TABLE cricket_batsman
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'";
mysql_query($query);
}
?>

LOAD DATA INFILE from cvs enclosed with double quotes fields

So, I got this. Just trying to insert a csv file into a MySQL through PHP PDO driver:
<?php
$databasehost = "localhost";
$databasename = "db";
$databasetable = "table";
$socketPath = "/home/mysql/mysql.sock";
$databaseusername="user";
$databasepassword = "pass";
$fieldseparator = ",";
$fieldenclosed = '"';
$lineseparator = "\r\n";
$csvfile = "file.csv";
if(!file_exists($csvfile)) {
die("File not found. Make sure you specified the correct path.");
}
try {
$pdo = new PDO("mysql:host=$databasehost;dbname=$databasename;unix_socket=$socketPath",
$databaseusername, $databasepassword,
array(
PDO::MYSQL_ATTR_LOCAL_INFILE => true,
PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION
)
);
} catch (PDOException $e) {
die("database connection failed: ".$e->getMessage());
}
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator).", ENCLOSED BY ". $pdo->quote($fieldenclosed)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)." IGNORE 1 LINES;");
echo "Loaded a total of $affectedRows records from this csv file.\n";
?>
The csv file is something like this(comma separated and enclosed by double quotes).
"X410","","4114068500","000010","04/15/2014","04/16/2015"
"X410","","4220521243","000030","04/08/2014","04/08/2015"
"X410","","4130003659","000010","04/02/2014","04/05/2014"
"X410","","4220524277","000010","04/08/2014","04/08/2015"
"X410","","4114038136","000010","04/07/2014","04/07/2015"
"X410","","4130003594","000110","03/14/2014","03/14/2015"
"X410","","4130003675","000010","04/04/2014","04/04/2015"
"X410","","4130003623","000010","03/12/2014","03/12/2015"
"X410","","4130003679","000010","04/09/2014","04/09/2015"
"X410","","4130003679","000020","04/09/2014","04/09/2015"
THe ENCLOSED BY part is the one that's giving me trouble, I've done my homework and tried $fieldenclosed = '\"\"', $fieldenclosed = "\"\"", with and withouth the $pdo->quote($fieldenclosed) and any other weird concat stuf I've though of or found out in other topics with similar issues.
MySQL throws this error:
PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ' ENCLOSED BY '"'
LINES TERMINATED BY '\r\n' IGNORE 1 LINES' at line 2' in /root/csvUpload.php:32
Im running MySQL 5.1 in a CentOS 6.4 server.
Any ideas?

There shouldn't be a comma before ENCLOSED BY. It should be:
$fieldenclosed = '"';
Just a single quote -- it's the character that's supposed to be at the beginning and end of the field.
You also mistyped the variable name, didn't end the string and concatenate around calling $pdo->quote.
$affectedRows = $pdo->exec("
LOAD DATA LOCAL INFILE ".$pdo->quote($csvfile)." INTO TABLE `$databasetable`
FIELDS TERMINATED BY ".$pdo->quote($fieldseparator)." ENCLOSED BY ".$pdo->quote($fieldenclosed)."
LINES TERMINATED BY ".$pdo->quote($lineseparator)." IGNORE 1 LINES;");

MySQL data import issues

I have tried just about everything i can think of (and searched plenty of threads here on Stack)
I am using the following:
try {
$db = new PDO('mysql:host=localhost;dbname=data', 'user', 'pass');
}catch (PDOException $e){
exit('Database Error.');
}
$fileName="file.txt";
$query = <<<eof
LOAD DATA LOCAL INFILE '$fileName' INTO TABLE data FIELDS TERMINATED BY '|'  LINES TERMINATED BY '\r\n'  (email, ip, referalurl, date, firstname, lastname);
eof;
$db->query($query);
echo "success";
now in my file (yes im aware its a .txt) its formated like below:
email#gmail.com|IP.IP.IP.IP|yourdomain.com|"2013-12-22 12:03:29"|firstname|lastname
any suggestions would be great.. btw the database connection is successful but no data ever gets saved into mysql for some reason..

I decided to go with mysqlimport (bash)

import glob text file from php to mysql

My glob code finds the file and attempts to insert it into the table but fails. I couldn't find anyone that is asking the same question. I will have many files that need to be inserted into the same table. It is a coma delimited text file.
Updated code*
The following code works as long as I rem the load data line. When I use the load data command manually, it works. I have not found a post example that has solved the problem "Why does the load data command not work?"
$con = new mysqli('localhost', 'username', 'password', 'schaadan_hash');
if (mysqli_connect_errno($con))
{
echo "Failed to connect to MySQL: " . mysqli_connect_error();
}
else
{
echo "Connected. ";
echo "<br />";
}
$upload = "LOAD DATA LOCAL INFILE 'ntlmhash2_1.txt' INTO TABLE ntlm2 FIELDS TERMINATED BY ',' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n'(`clear` , `hash`);";
mysqli_query($con,$upload);
?>

Categories