I have a csv file that I parse and display it by storing it in an array. I was wondering how can I validate the mail or make sure it's the right format before inserting it into mysql? And, if the mail is not validate it should display an error and not insert that row to mysql.
I know I can use that filter_var($email, FILTER_VALIDATE_EMAIL) !== false; but I am not sure how to use it in my code as it is inside an array (the value of email). Any help is appreciated.
My code:
<?php
require "database.php";
$lines = file('users.csv', FILE_IGNORE_NEW_LINES|FILE_SKIP_EMPTY_LINES);
$csv = array_map('str_getcsv', $lines);
$col_names = array_shift($csv);
$users = [];
try {
$stmt = $dbh->prepare("INSERT INTO users(name, surname, email)
VALUES(:name, :surname, :email)");
foreach($csv as $row) {
$users[] = [
$col_names[0] => ucfirst(strtolower($row[0])),
$col_names[1] => ucfirst(strtolower($row[1])),
$col_names[2] => strtolower(strtolower($row[2])), //that's the mail
];
$stmt->bindValue(':name', $col_names[0], PDO::PARAM_STR);
$stmt->bindValue(':surname', $col_names[1], PDO::PARAM_STR);
$stmt->bindValue(':email', $col_names[2], PDO::PARAM_STR);
$stmt->execute();
}
}catch(PDOException $e) {
echo "Error: " . $e->getMessage();
}
$dbh = null;
?>
You can simply check it within your loop something like below
foreach($csv as $row) {
if (filter_var(strtolower(strtolower($row[2])), FILTER_VALIDATE_EMAIL) === false) {
echo("".strtolower($row[2]))." is not a valid email address"); // You can break the loop and return from here
}
$users[] = [
$col_names[0] => ucfirst(strtolower($row[0])),
$col_names[1] => ucfirst(strtolower($row[1])),
$col_names[2] => strtolower(strtolower($row[2])), //that's the mail
];
$stmt->bindValue(':name', $col_names[0], PDO::PARAM_STR);
$stmt->bindValue(':surname', $col_names[1], PDO::PARAM_STR);
$stmt->bindValue(':email', $col_names[2], PDO::PARAM_STR);
$stmt->execute();
}
Apart from this I would suggest you to create a batch insert for this.. Never execute the insertion query in a loop
Related
I have list of data in CSV and need to insert this data into a MySQL database. These data should be safely inserted i.e sanitation. So, I have used PDO object to rectify SQL injection. But, it fails to get data from CSV file and inserts null values.
Here is the example,
<?php
$servername = "localhost";
$username = "root";
$password = "";
try {
$conn = new PDO("mysql:host=$servername;dbname=contact_list",$username,$password);
$conn->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
echo "connection successfully";
}
catch(PDOException $e)
{
echo "connection Failed:" . $e -> getMessage();
}
// Create CSV to Array function
function csvToArray($filename = '', $delimiter = ',')
{
if (!file_exists($filename) || !is_readable($filename)) {
return false;
}
$header = NULL;
$result = array();
if (($handle = fopen($filename, 'r')) !== FALSE) {
while (($row = fgetcsv($handle, 1000, $delimiter)) !== FALSE) {
if (!$header)
$header = $row;
else
$result[] = array_combine($header, $row);
}
fclose($handle);
}
return $result;
}
// Insert data into database
$all_data = csvToArray('contact.csv');
foreach ($all_data as $data) {
$data = array_map(function($row){
return filter_var($row, FILTER_SANITIZE_STRING, FILTER_SANITIZE_FULL_SPECIAL_CHARS);
}, $data);
$sql = $conn->prepare("INSERT INTO contact
(title, first_name,last_name,company_name,date_of_birth,notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
$sql->bindParam(':t', $data[1], PDO::PARAM_STR);
$sql->bindParam(':fname', $data[2], PDO::PARAM_STR);
$sql->bindParam(':lname', $data[3], PDO::PARAM_STR);
$sql->bindParam(':cname', $data[0], PDO::PARAM_STR);
$sql->bindParam(':dob', $data[4], PDO::PARAM_STR);
$sql->bindParam(':note', $data[15], PDO::PARAM_STR);
print_r($data);
$sql->execute();
}
?>
Can anyone help me to solve this?
If you take a look at the documentation for array_combine() you'll see that its purpose is to build an associative array. You use this function in csvToArray() but later in your code you are trying to get data using numeric keys. I wouldn't expect you'd ever have anything inserted.
On a side note, you are completely defeating the purpose of prepared statements by repeatedly preparing the same statement over and over again. Prepare once and execute many times. Individually binding parameters is rarely needed, in almost all cases you can provide the data to PDOStatement::execute() as an array. It's also bad form to store HTML entities in a database; if you need to output to HTML, you perform escaping at that point.
Something like this should work (adjust array key names as necessary.)
$all_data = csvToArray('contact.csv');
$sql = $conn->prepare("INSERT INTO contact
(title, first_name, last_name, company_name, date_of_birth, notes)
VALUES (:t, :fname, :lname,:cname,:dob,:note)");
foreach ($all_data as $data) {
$params = [
":t" => $data["t"],
":fname" => $data["fname"],
":lname" => $data["lname"],
":dob" => $data["dob"],
":note" => $data["note"],
];
$sql->execute($params);
}
I need to get an array of id's of the subscribers added to the database within this function:
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
return array('status'=> true);
} else {
return array('status'=> false);
}
}
However I am stuggling with figuring out how to get the values and return them within the array like so:
return array('status'=> true,'ids'=> $ids);
I have read up and found that SCOPE_IDENTITY() is probably the most reliable way of getting these values but I don't know where to put it in this function to return the values that I need. The Primary key in the database is the id column.
Would it be something like this:
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) VALUES (GETDATE(), GETDATE(), :fullname, :email, 1) SELECT SCOPE_IDENTITY()"))
And then bind a parameter after that called $ids or am I over thinking this completely?
Any help would be greatly appreciated!
EDIT: I have tried using a similar function to the one in this question and was receiving an invalid cursor error so adapted it to look like this (notice the closeCursor was how to fix the invalid cursor error):
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) OUTPUT INSERTED.id VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
$SQL->closeCursor();
$ids = $SQL->fetchAll(PDO::FETCH_ASSOC);
return array('status'=> true, 'id' => $ids);
} else {
$pdo = null;
return array('status'=> false);
}
}
The return from this function is now this:
Array
(
[status] => 1
[id] => Array
(
)
)
So it seems the output is not working as it should? This is getting stranger and stranger...
Figured it out guys!
There was a problem with the position of the closeCursor(); method the function now looks like this:
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) OUTPUT INSERTED.id VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
$ids = $SQL->fetchAll(PDO::FETCH_ASSOC);
$SQL->closeCursor();
foreach ($ids as $id) {
return $id['id'];
}
} else {
$pdo = null;
return false;
}
}
So when insert_test($pdo, $fullname, $email); is within a for each loop containing the data being inserted the function returns each id as desired.
If anyone can see any inherent problems with this please let me know!
I need to get an array of id's of the subscribers added to the database within this function:
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
return array('status'=> true);
} else {
return array('status'=> false);
}
}
However I am stuggling with figuring out how to get the values and return them within the array like so:
return array('status'=> true,'ids'=> $ids);
I have read up and found that SCOPE_IDENTITY() is probably the most reliable way of getting these values but I don't know where to put it in this function to return the values that I need. The Primary key in the database is the id column.
Would it be something like this:
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) VALUES (GETDATE(), GETDATE(), :fullname, :email, 1) SELECT SCOPE_IDENTITY()"))
And then bind a parameter after that called $ids or am I over thinking this completely?
Any help would be greatly appreciated!
EDIT: I have tried using a similar function to the one in this question and was receiving an invalid cursor error so adapted it to look like this (notice the closeCursor was how to fix the invalid cursor error):
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) OUTPUT INSERTED.id VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
$SQL->closeCursor();
$ids = $SQL->fetchAll(PDO::FETCH_ASSOC);
return array('status'=> true, 'id' => $ids);
} else {
$pdo = null;
return array('status'=> false);
}
}
The return from this function is now this:
Array
(
[status] => 1
[id] => Array
(
)
)
So it seems the output is not working as it should? This is getting stranger and stranger...
Figured it out guys!
There was a problem with the position of the closeCursor(); method the function now looks like this:
function insert_test($pdo, $fullname, $email) {
if ($SQL = $pdo->prepare("INSERT INTO subscribers ([dateAdded],[dateUpdated],[fullname],[email],[isActive]) OUTPUT INSERTED.id VALUES (GETDATE(), GETDATE(), :fullname, :email, 1)")) {
$SQL->bindValue(':fullname', $fullname, PDO::PARAM_STR);
$SQL->bindValue(':email', $email, PDO::PARAM_STR);
$SQL->execute();
$ids = $SQL->fetchAll(PDO::FETCH_ASSOC);
$SQL->closeCursor();
foreach ($ids as $id) {
return $id['id'];
}
} else {
$pdo = null;
return false;
}
}
So when insert_test($pdo, $fullname, $email); is within a for each loop containing the data being inserted the function returns each id as desired.
If anyone can see any inherent problems with this please let me know!
I have the following code to read a JSON and store the results in a DDBB.
The code works, but it takes more than a minute to insert just 400 records.
If I open the json, it loads pretty fast.
What I'm doing wrong?
$db = new PDO('', '', '');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
if(tableExists($db, 'locations') == 1)
{
$sql=$db->prepare("DROP TABLE locations");
$sql->execute();
}
$sql ="CREATE TABLE `locations` (id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, evid INT(6) NOT NULL, place VARCHAR(150), country VARCHAR(150), reg_date TIMESTAMP)" ;
$db->exec($sql);
$json = file_get_contents('thejson.php');
$data = array();
$data = json_decode($json);
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
$stmt->bindValue(':evid', $id, PDO::PARAM_INT);
$stmt->bindValue(':places', $place, PDO::PARAM_STR);
$stmt->bindValue(':country', $country, PDO::PARAM_STR);
$stmt->execute();
}
}
So first 2 things i would try would be moving the prepare outside the loop, and wrapping it in a transaction:
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt->bindValue(':evid', $id, PDO::PARAM_INT);
$stmt->bindValue(':places', $place, PDO::PARAM_STR);
$stmt->bindValue(':country', $country, PDO::PARAM_STR);
$stmt->execute();
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
Another thing you could do is try using bindParam to bind the vars by reference - this way you only need to call bindParam once on each variable name at the start, then just overwrite the values of those variables on each iteration and call execute.
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
$id = 0;
$place = '';
$country = '';
$stmt->bindParam(':evid', $id, PDO::PARAM_INT);
$stmt->bindParam(':places', $place, PDO::PARAM_STR);
$stmt->bindParam(':country', $country, PDO::PARAM_STR);
foreach ($data as $key => $object)
{
if(is_object($object))
{
$id = $object->id;
$place = $object->name;
$country = substr(strrchr($object->name, "-"), 2);
$stmt->execute();
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
Similar to this instead of calling bind* you could just pass the values in via execute:
try {
$db->beginTransaction();
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
foreach ($data as $key => $object)
{
if(is_object($object))
{
$params = array(
':id' => $object->id,
':places' => $object->name,
':country' => substr(strrchr($object->name, "-"), 2)
);
$stmt->execute($params);
}
}
$db->commit();
} catch (Exception $e) {
$db->rollBack();
throw $e;
}
I suspect using a transaction will get you a performance gain, but I dont know that there will be a ton of difference between switching up the binding methods.
Your best bet is probably to insert all the records in a single query as #PavanJiwnani suggests:
// first we need to compile a structure of only items
// we will insert with the values properly transformed
$insertData = array_map(function ($object) {
if (is_object($object)) {
return array(
$object->id,
$object->name,
substr(strrchr($object->name, "-"), 2)
);
} else {
return false;
}
}, $data);
// filter out the FALSE values
$insertData = array_filter($insertData);
// get the number of records we have to insert
$nbRecords = count($insertData);
// $records is an array containing a (?,?,?)
// for each item we want to insert
$records = array_fill(0, $nbRecords, '(?,?,?)');
// now now use sprintf and implode to generate the SQL like:
// INSERT INTO `locations` (evid, place, country) VALUES (?,?,?),(?,?,?),(?,?,?),(?,?,?)
$sql = sprintf(
'INSERT INTO `locations` (evid, place, country) VALUES %s',
implode(',', $records)
);
$stmt = $db->prepare($sql);
// Now we need to flatten our array of insert values as that is what
// will be expected by execute()
$params = array();
foreach ($insertData as $datum) {
$params = array_merge($params, $datum);
}
// and finally we attempt to execute
$stmt->execute($params);
Try to echo timestamps with milliseconds to see what is running slow. Probably executing 400 insert queries (including opening/closing connections).
There are many factors that affects the performance of the database, please provide detailed information about the database system, PHP version and related hardware.
The bottleneck could be at:
file_get_contents('thejson.php')
if the JSON contents are fetched from a remote host, i.e. DB is running normally, network is slow.
You may also want to consider move:
$stmt = $db->prepare("INSERT INTO `locations` (evid, place, country) VALUES (:evid, :places, :country)");
out of the foreach loop.
I'm working with a sequence of queries created with PDO class, in some case, my queries needs the same parameter.
I've created an array used in a foreach statement which save the data but some variables come from outside, can I use both data in one query?
the example:
// $connection is the PDO object;
// $full_data contains:
// $full_data[$i]["address"]
// $full_data[$i]["phone"]
// $full_data[$i]["email"]
// $full_data[$i]["user_id"]
// $full_data[$i]["surname"] // not used but present
// $full_data[$i]["name"] // not used but present
$sql = "UPDATE users_table SET city = :address, phone = :phone, email = :email, admin_id = :admin_id, admin_name = :admin_name WHERE user_id = :user_id";
$statement = $connection->prepare ($sql);
$statement->bindParam (':admin_id', trim($admin_id), PDO::PARAM_INT);
$statement->bindParam (':admin_name', trim($admin_name), PDO::PARAM_STR);
foreach ($full_data as $value) {
$ok = $statement->execute ($value);
$num = $statement->rowCount ();
}
} catch (PDOException $e) {
return $e->getMessage ();
}
this page return me the error:
SQLSTATE[HY093]: Invalid parameter number: number of bound variables does not match number of tokens
what is exactly the problem, on an UPDATE statement the technique works
damn, I've found the problem after hours...
// $connection is the PDO object;
// $full_data contains:
// $full_data[$i]["address"]
// $full_data[$i]["phone"]
// $full_data[$i]["email"]
// $full_data[$i]["user_id"]
// ==> $full_data[$i]["surname"] // not used but present
// ==> $full_data[$i]["name"] // not used but present
the array data not saved in the query ["surname"] and ["name"] generate the error.
It seems like execute (); needs precise array data structure.
I've solved the problem by using this:
$sql = "UPDATE users_table SET city = :address, phone = :phone, email = :email, admin_id = :admin_id, admin_name = :admin_name WHERE user_id = :user_id";
$statement = $connection->prepare ($sql);
// must be removed ==> $statement->bindParam (':admin_id', trim($admin_id), PDO::PARAM_INT);
// must be removed ==> $statement->bindParam (':admin_name', trim($admin_name), PDO::PARAM_STR);
for ($i = 0; $i < count($full_data); $i++) {
$full_data[$i]["admin_name"] = "the admin name";
$full_data[$i]["admin_id"] = "100";
unset ($full_data[$i]["surname"]); // IMPORTANT: must remove the unused vars
unset ($full_data[$i]["name"]); // IMPORTANT: must remove the unused vars
}
foreach ($full_data as $value) {
// bindParam can be avoided, but it's recommended for data type security
$statement->bindParam(':address', trim($value['address']), PDO::PARAM_STR);
$statement->bindParam(':phone', trim($value['phone']), PDO::PARAM_STR);
$statement->bindParam(':email', trim($value['email']), PDO::PARAM_STR);
$statement->bindParam(':admin_id', trim($value['admin_id']), PDO::PARAM_INT);
$statement->bindParam(':admin_name', trim($value['admin_name']), PDO::PARAM_STR);
$ok = $statement->execute ($value);
$num = $statement->rowCount ();
}
} catch (PDOException $e) {
return $e->getMessage ();
}
You need to bind the :address, :phone, and :email parameters.
To elaborate on BD answer you're missing the following lines of code:
$statement->bindParam (':address', trim($address), PDO::PARAM_STR);
$statement->bindParam (':phone', trim($phone), PDO::PARAM_STR);
$statement->bindParam (':email', trim($email), PDO::PARAM_STR);
Plus, something seems to be wrong with your foreach loop, I think this is what you want:
$sql = "UPDATE users_table SET city = :address, phone = :phone, email = :email, admin_id = :admin_id, admin_name = :admin_name";
$statement = $connection->prepare($sql);
$statement->bindParam(':admin_id', trim($admin_id), PDO::PARAM_INT);
$statement->bindParam(':admin_name', trim($admin_name), PDO::PARAM_STR);
foreach ($full_data as $value)
{
$statement->bindParam(':address', trim($value['address']), PDO::PARAM_STR);
$statement->bindParam(':phone', trim($value['phone']), PDO::PARAM_STR);
$statement->bindParam(':email', trim($value['email']), PDO::PARAM_STR);
$ok = $statement->execute();
$num = $statement->rowCount();
}