I have the following script which uploads the data from a CSV file to my database.
Problem occurs though when one of the fields in the CSV has a apostrophe (')
Sample data from CSV:
"12345","John","Smith","john.smith#gmail.com","Company Name"
"12346","Joe","Blogg","joe.blogg#gmail.com","Company's Name"
Code I'm using:
<?
$link = mysql_connect("localhost", "######", "######") or die("Could not connect: ".mysql_error());
$db = mysql_select_db("######") or die(mysql_error());
$row = 1;
$handle = fopen ("file.csv","r");
while ($data = fgetcsv ($handle, 1000, ",")) {
$query = "INSERT INTO suppliers(`regid`, `firstname`, `lastname`, `email`, `company`) VALUES('".$data[0]."', '".$data[1]."', '".$data[2]."', '".$data[3]."', '".$data[4]."') ON DUPLICATE KEY UPDATE REGID='".$data[0]."', firstname='".$data[1]."', lastname='".$data[2]."', email= '".$data[3]."', company= '".$data[4]."'";
$result = mysql_query($query) or die("Invalid query: " . mysql_error().__LINE__.__FILE__);
$row++;
}
fclose ($handle);
?>
Can anyone suggest a solution to get around this?
Many thanks
The best solution would be to upgrade to PDO or mysqli, and make use of their parametrized queries.
If you can't, you should escape the data before inserting it into queries:
while ($data = fgetcsv($handle, 1000, ",")) {
$data = array_map('mysql_real_escape_string', $data);
$query = "INSERT INTO suppliers(`regid`, `firstname`, `lastname`, `email`, `company`) VALUES('".$data[0]."', '".$data[1]."', '".$data[2]."', '".$data[3]."', '".$data[4]."') ON DUPLICATE KEY UPDATE REGID='".$data[0]."', firstname='".$data[1]."', lastname='".$data[2]."', email= '".$data[3]."', company= '".$data[4]."'";
$result = mysql_query($query) or die("Invalid query: " . mysql_error().__LINE__.__FILE__);
$row++;
}
You should always be using mysql_real_escape_string on any user-supplied data, to protect against SQL injection or deal with syntax problems like this.
I would use LOAD DATA for this.
LOAD DATA LOCAL INFILE 'file.csv' REPLACE INTO TABLE suppliers
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
(regid, firstname, lastname, email, company);
No need to fopen(), fgetcsv(), or execute INSERT so many times. No need to worry about apostrophes or other special characters.
If you want to carry on using mysql_query as described, i'd suggest running $data though array _map to escape the single quotes. Something like
$data = array_map(function($string){
return str_replace("'", "\'", $string);
}, $data);
The best way to do this is using the LOAD DATA INFILE statement to bulk load a CSV into MySQL:
LOAD DATA INFILE 'csvfile' into suppliers FIELDS TERMINATED BY ','
The added advantage of this is that you don't need to escape your strings at all -- LOAD DATA does it for you. Let me know if you have further questions.
Related
I need to upload csv and import this data to MySQL table.
public function insertData()
{
while (($column = fgetcsv($file, 10000, ",")) !== FALSE) {
$query = "INSERT into deleted_users (email)
values ('" . $column[0] . "'); $query->execute();
}
}
Perhaps the fastest way to bulk load data from PHP into MySQL is to use the LOAD DATA tool, something like this:
$sql = "LOAD DATA LOCAL INFILE 'path/to/yourfile.csv'
INTO TABLE deleted_users
FIELDS TERMINATED BY ','
OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\r\n'
(#col1)
SET email = #col1;";
mysqli_query($conn, $sql);
This answer loads records with only the email field being assigned. If you want to assign a value for the id and type columns, then you will need a CSV file containing these values.
So, I've got a few txt files, each container around 400,000 lines.
Each line is a word, I need to add to my database, if it isn't in there already.
Currently my code for checking/adding every word is
$sql = mysql_sql("SELECT `id` FROM `word_list` WHERE `word`='{$word}' LIMIT 1");
$num = mysql_num($sql);
if($num == '0'){
$length = strlen($word);
$timestamp = time();
#mysql_sql("INSERT INTO `word_list` (`word`, `length`, `timestamp`) VALUES ('{$word}', '{$length}', '{$timestamp}')");
}
and the functions being called are:
function mysql_sql($sql){
global $db;
$result = $db->query($sql);
return $result;
}
function mysql_num($result){
return $result->num_rows;
}
I'm looking for a better way to insert each word into the database.
Any ideas would be greatly appreciated.
I can think of some ways to do this.
First, if you have access to the MySQL server's file system you can use LOAD DATA INFILE to create a new table, then do an insert from that new table into your word_list table. This will most likely be your fastest option.
Second (if you don't have access to the MySQL server's file system), put a primary key or unique index on word_list.word. Then get rid of your SELECT query and use INSERT IGNORE INTO word_list .... That will allow MySQL automatically to skip the duplicate items without any need for you to do it with a query/insert operation.
Third, if your table uses an access method that handles transactions (InnoDB, not MyISAM), issue a BEGIN; statement before you start your insert loop. Then every couple of hundred rows issue COMMIT;BEGIN; . Then at the end issue COMMIT;. This will wrap your operations in multirow transactions so will speed things up a lot.
Try out this code. It will first create query with all your values and you will run query only ONCE ... Not again and again for ever row
$values = array();
$sql = mysql_sql("SELECT `id` FROM `word_list` WHERE `word`='{$word}' LIMIT 1");
$num = mysql_num($sql);
$insert_query = "INSERT INTO `word_list` (`word`, `length`, `timestamp`) VALUES ";
if ($num == '0') {
$length = strlen($word);
$timestamp = time();
$values[] = "('$word', '$length', '$timestamp')";
}
$insert_query .= implode(', ', $values);
#mysql_sql($insert_query);
I have been using the following query to upload my data into mysql database:
$sql = array();
foreach( $data as $row ) {
$sql[] = '("'.mysql_real_escape_string($row['text']).'", '.$row['category_id'].')';
}
mysql_query('INSERT INTO table (text, category) VALUES '.implode(',', $sql));
Since a bit I have started using PDO and my query looks like this:
$query="INSERT INTO mytable (name, use) VALUES(:sname, :usee)";
$res = $db_conn->prepare($query);
$res->bindValue(':sname',$value);
$res->bindValue(':usee',$_SESSION['usee']);
$res->execute();
Now the above code block is fine, but now when I am going through my CSV upload thing, I again looking backward and using the first code. Want to use the same PDO now for CSV upload also.
Is there a trick to upload multiple values in database using PDO at once?
Yes, you can use some loop in which you will execute, something like this:
$query = $db->prepare(
'INSERT INTO mytable (name, use) VALUES(:sname, :usee)'
);
foreach($mainArrayOfveluus AS $arrayOfValue){
$query->execute(array(
':sname' => $arrayOfValue['sname'],
':usee' =>$arrayOfValue['usee']
));
}
$query->commit();
This is my code:
$q=mysql_query("SELECT * FROM `table1` WHERE name like '%$searchText%'");
while($e=mysql_fetch_assoc($q))
//$output[]=$e;
//echo $e['NAME'];
{
$name = $e['NAME'];
$brand = $e['BRAND'];
$category = $e['CATEGORY'];
$query = "INSERT INTO table2 (brand, name, category) VALUES ('$brand', '$name', '$category')";
$result = mysql_query($query) or die("Unable to insert because : " . mysql_error());
}
Since in "BRAND", there may be some data like "First's Choice".
In this case, I cannot insert to database due to error.
How can I insert data that contain single quotes?
Thx
you need to use mysql_real_escape_string on the value, which you should be doing anyway. That should properly escape your value for insertion.
$name = mysql_real_escape_string($e['NAME']);
$brand = mysql_real_escape_string($e['BRAND']);
$category = mysql_real_escape_string($e['CATEGORY']);
$query = "INSERT INTO table2 (brand, name, category) VALUES ('$brand', '$name', '$category')";
Use mysql_real_escape_string
You must use :
$brand = mysql_real_escape_string($brand)
See PHP Documentation.
string mysql_real_escape_string ( string $unescaped_string [, resource $link_identifier = NULL ] )
Escapes special characters in
the unescaped_string, taking into account the current character set of
the connection so that it is safe to place it in a mysql_query(). If
binary data is to be inserted, this function must be used. (..)
Try below code
$q=mysql_query("SELECT * FROM `table1` WHERE name like '%$searchText%'");
while($e=mysql_fetch_assoc($q))
//$output[]=$e;
//echo $e['NAME'];
{
$name = $e['NAME'];
$brand = mysql_real_escape_string($e['BRAND']);
$category = $e['CATEGORY'];
$query = "INSERT INTO table2 (brand, name, category) VALUES ('$brand', '$name', '$category')";
$result = mysql_query($query) or die("Unable to insert because : " . mysql_error());
}
There are two ways of accomplishing that. You can first run an escape string on it:
$newbrand = mysql_real_escape_string($brand);
and insert $newbrand. When you call it, you have to do strpslashes($newbrand);
OR you could do:
$search = array("'");
$newbrand = str_replace($search,'',$brand);
I was pulling my hair to solve this, finally i am ok with this solution. Try this
I'm trying to load data from a few hundred text files into a database.
I believe MYSQL is exiting out of the loop without inserting all the rows.
Can anyone suggest how to insert blocks of 1000 rows to the end of data, with PHP code?
$filenames_array = array();
foreach($filenames_array as $filename)
{
$file_array = file($filename);
$file_value = $file_array[0];
$new_array = explode(",", $file_value);
$length = count($new_array);
for($i = 0; $i < $length; $i++)
{
$sql = "INSERT INTO `names`
(`id`, `name`)
VALUES
('',
'" . $new_array[$i] . "'
)";
$result = mysql_query($sql) or die(mysql_error());
echo $i . 'Row Inserted<br />';
}
}
you're probably trying to run too many INSERT statements in a single query.
look into PDO and prepared statements or use SQL syntax like this:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
Is it possible that one of the entries you're trying to insert contains a single quote '? In this case, an error would occur and the the loop wouldn't finish.
You should always escape the values you insert into the database with mysql_real_escape_string to prevent problems like that, and to make sure you're not vulnerable to sql injection.
$sql = "INSERT INTO `names`
(`id`, `name`)
VALUES
('',
'" . mysql_real_escape_string($new_array[$i]) . "'
)";
Why not combine every txt file into one big text file, and read it line by line? See the examples here http://php.net/manual/en/function.fgets.php
Mainly:
<?php
$handle = #fopen("/tmp/inputfile.txt", "r");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
echo $buffer;
}
fclose($handle);
}
?>