I have multiple csv files generated everyday with same name and they should be imported to Mysql database . I can import a single file to table where I have defined the csv file and table name before importing, but how to match the csv file to table names and import them. here is my code:
$conn = mysqli_connect($servername, $username, $password);
// Check connection
if (mysqli_connect_error()) {
echo "Connection failed!!!" .mysqli_connect_error();
exit();
}
else{
echo "Connected successfully \n";
}
$serverDir = $_SERVER['DOCUMENT_ROOT'];
//$filename = "OwlertonGreenIN.CSV";
echo getcwd();
print_r($filename);
//$table_name = strtolower( $filename );
//$filename = "$serverDir.$filename";
if (($handle = fopen($filename, 'r')) !== FALSE)
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE)
{
// print_r($data);
$import="INSERT into table name (`camera_name`,`plate`) values(`camera_name`,`plate`)";
}
I created functions for myself a while back that would essentially solve your problem if implemented correctly.
Only thing is I created them inside a DB class that uses/extends PDO and doesn't use mysqli. If you were to change how you connect to your database and use PDO, you could use the below class and utilize the insertQuery function that I made.
<?php
class DB extends PDO{
protected $host = "localhost";
protected $user = "admin";
protected $password = "";
public $connect;
public function __construct(string $dbname){
try{
$this->connect = new \PDO("mysql:host=".$this->host.";dbname=".$dbname, $this->user, $this->password);
$this->connect->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
}catch(PDOException $e){ }
}
public function insertQuery(string $table, array $data){
try{
// declare our query peices
$half1 = "INSERT INTO `${table}` (";
$half2 = ") VALUES (";
// build both halfs of query w/ the keys of the array being used as the column name
foreach($data as $column => $v){
$half1 .= "${column}, ";
$half2 .= ":${column}, ";
}
// remove extra commas and spaces from the ends of our two halfs and then combine them
$query = substr($half1, 0, strlen($half1) - 2).substr($half2, 0, strlen($half2) - 2).")";
// prepare query
$prep = $this->connect->prepare($query);
// loop through & bind all values in array to the corresponding keys in our query
foreach($data as $column => $value){
if(is_string($value)){
$prep->bindValue(":${column}", $value, PDO::PARAM_STR);
}elseif(is_int($value)){
$prep->bindValue(":${column}", $value, PDO::PARAM_INT);
}elseif(is_bool($value)){
$prep->bindValue(":${column}", $value, PDO::PARAM_BOOL);
}
}
// execute query
$prep->execute();
// if row added, return true
if($prep->rowCount() > 0) return true;
// if not, return false
else return false;
}catch(PDOException $e){ return false; }
}
public static function csvParse($input, $callback = false){
$results = [];
$raw_array = (is_file($input)) ? array_map('str_getcsv', file($input)):array_map('str_getcsv', explode("\n", $input));
$array = array_splice($raw_array, 1, count($raw_array));
foreach($raw_array[0] as $c) $columns[] = $c;
foreach($array as $key0 => $val0) foreach($val0 as $key1 => $val1) $results[$key0][$columns[$key1]] = $val1;
if(is_callable($callback) && !empty($results)) call_user_func_array($callback, array($results));
elseif(!empty($results)) return $results;
else throw new Exception("Results Empty: Can not read the string or open file.");
}
}
$csv_str = "name,age,occupation,city\nCrimin4L,24,Programmer,New York\nMrAwesome,20,Gamer,Los Angeles";
// parse CSV file or string into a readable array (if file, use file location as parameter):
$csv_array = DB::csvParse($csv_str);
// print_r($csv_array) output:
/* Array
(
[0] => Array
(
[name] => Crimin4L
[age] => 24
[occupation] => Programmer
[city] => New York
)
[1] => Array
(
[name] => MrAwesome
[age] => 20
[occupation] => Gamer
[city] => Los Angeles
)
) */
// with that you can now use the DB::insertQuery function, but since it
// is a 2 dimentional array (multiple rows) you would need to loop
// through with a foreach
# start your database connection
$db = new DB("database_name_to_connect_into");
// Now loop through array with foreach
foreach($csv_array as $row){
# insert current row
$db->insertQuery("table_to_insert_data_into", $row);
}
/*
As long as the table column(s) match the key(s) in
the array, it should work flawlessly.
----
For this example, the table columns in the database
would need to be named: name, age, occupation, & city;
because that is what the array keys are named, basically
the array values are inserted into their corresponding table
columns by matching the array keys (if that any makes sense).
*/
?>
I can't really give you an example as its dealing with databases but if you'd like to play around with the code here it is in a sandbox: https://sandbox.onlinephpfunctions.com/c/20365
I'd use this:
$tablename = pathinfo($filename, PATHINFO_FILENAME);
Prepare an INSERT statement with placeholders:
$import="INSERT INTO `{$tablename}` (`camera_name`,`plate`) VALUES(?, ?)";
$stmt = $conn->prepare($import);
The tablename should be within back-ticks because you don't know if it contains whitespace or punctuation or an SQL reserved keyword.
Then execute that prepared statement for each row you read from the CSV file.
while (($data = fgetcsv($handle, 1000, ";")) !== FALSE)
{
$stmt->bind_param('ss', $data[0], $data[1]);
$stmt->execute();
}
This may run pretty slowly, inserting one row at a time. You might like to read my presentation Load Data Fast! for more tips on performance improvement.
Related
I am making a PDO update statement but it is entering 0's on empty values.
I want a empty value to enter null
I want a zero value to enter zero
Here is my code:
include 'dbconfig.php';
$id = $_POST['id'];
$data = $_POST['data'];
try {
$options = [
\PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION,
\PDO::ATTR_DEFAULT_FETCH_MODE => \PDO::FETCH_ASSOC,
\PDO::ATTR_EMULATE_PREPARES => false,
];
$pdo = new PDO("mysql:charset=utf8mb4;host=$servername;dbname=$dbname", $username, $password);
$setStr = "";
$values = array();
foreach($data as $field => $value) {
$setStr .= "$field=:$field,";
$values[$field] = $value;
}
$setStr = rtrim($setStr, ",");
$values['id'] = $id;
$stmt = $pdo->prepare("UPDATE products SET $setStr WHERE id = :id");
$stmt->execute($values);
$count = $stmt->rowCount();
if ($count > 0) {
echo json_encode(array('response'=>'success','message'=>'Product ID # '.$id." successfully updated."));
}else{
echo json_encode(array('response'=>'danger','message'=>'Product not successfully updated'));
}
}catch(PDOException $e){
echo json_encode(array('response'=>'danger','message'=>$e->getMessage()));
}
$conn = null;
What is the best way to do this? Thank you
Check for an empty string in your loop, and convert it to null.
foreach($data as $field => $value) {
$setStr .= "$field=:$field,";
$values[$field] = $value === "" ? null : $value;
}
PDOStatement::execute considers every parameter to be a string. In some databases the driver deals with this without issue, but if you want to specifically type values you need instead to use bindValue. With this typing, we can type null (or empty) values to the database-null type (instead of whatever the column default is) - see my bindEmptyAsNull function.
Also, since I'm using a trusted source for the column names, I can safely use your $setStr pattern for building the update params, but note you might not want to update a column that isn't included in the $dangerData associative array (as it is, if it isn't submitted, the column will be cleared).
<?php
//... your setup code, including $pdo
// List all the names of the columns in the products table here - you havent' provided enough info so I'm guessing
$keys=["name","cost","weight_grams"];
$dangerId = $_POST['id'];
$dangerData = $_POST['data'];
/**
* Convert null values into PARAM_NULL types, not
* #param $type Should be one of the [PDO::PARAM_*](https://www.php.net/manual/en/pdo.constants.php) constants
*/
function bindEmptyAsNull($stmt,$name,$value,$type=PDO::PARAM_STR) {
//If you want empty string to be considered null too:
//if ($value===null || $value==="") {
if ($value===null) {
//Note the column needs to support null types
$stmt->Value($name,null,PDO::PARAM_NULL);
} else {
$stmt->bindValue($name,$value,$type);
}
}
//Build the set string
$setStr='';
foreach($keys as $col) {
$setStr.=$col.'=:'.$col.',';
}
$setStr=substr($set,0,-1);//Remove the trailing comma
//Create the statement
$stmt=$pdo->prepare("UPDATE products SET $setStr WHERE id=:id");
//Bind the danger values
// NOTE: The only way you could automate this (re-read the keys array) is to also have an
// associative array of key:type mapping too
$stmt->bindValue(":id",$dangerId,PDO::PARAM_INT);
bindEmptyAsNull($stmt,":name",$dangerData["name"]);
bindEmptyAsNull($stmt,":cost",$dangerData["cost"]);
bindEmptyAsNull($stmt,":weight_grams",$dangerData["weight_grams"],PDO::PARAM_INT);
$stmt->execute();
// .. Further processing
I'm getting null values after I run the DBEscape($data) function that is for SQL injection protection. Can someone help?
My inputs are all multiple arrays, ex: name="quote[][dt_flight]", name="quote[][acft]", etc.
Method is POST.
function DBEscape($data){
$link = DBConect();
if(!is_array($data)){
$data = mysqli_real_escape_string($link,$data);
}
else {
$arr = $data;
foreach ($arr as $key => $value){
$key = mysqli_real_escape_string($link, $key);
$value = mysqli_real_escape_string($link, $value);
$data[$key] = $value;
}
}
DBClose($link);
return $data;
}
function DBCreate($table, array $data, $insertId = false){
$table = DB_PREFIX.'_'.$table;
$data = DBEscape($data);
var_dump($data);
$fields = implode(", ", array_keys($data));
$values = "'".implode("', '", $data)."'";
$query = "INSERT INTO {$table} ({$fields}) VALUES ({$values});";
var_dump($query);
return DBExecute($query, $insertId);
}
if(isset($_POST["quote"]) && is_array($_POST["quote"])){
foreach($_POST["quote"]["dt_flight"] as $key => $text_field){
$last_id = DBCreate('quote',$_POST['quote'],true);
$i++;
}
}
The connection works since it is inserting the rows into the tables. I used vardump before and after the DBEscape to figure out that it is deleting the values, the keys are fine.
PS: The proposed answer is for a single variable not an array.
As you can see in your var_dump-result, the data you sent to DBCreate and thus to DBEscape looks like
array(
'dt_flight' => array(0 => '2018-06-13'),
'acft' => array(0 => 'VQ-BFD',
// and so on
)
Therfore the data you sent to
// $value = array(0 => '2018-06-13') here
$value = mysqli_real_escape_string($link, $value);
And well, mysqli_real_escape_string doesn't like arrays very much, thus will return NULL and thus inserting empty data in your table.
You most likely want to resolve this error within your foreach($_POST["quote"]["dt_flight"]) loop, since I suppose you sent multiple flight-data:
foreach($_POST["quote"]["dt_flight"] as $key => $text_field) {
// $key would be 0, for $_POST["quote"]["dt_flight"][0] = '2018-06-13'
$keyData = [];
foreach($_POST["quote"] as $field => $allFieldValues) {
// Walk over every field, and add the value for the same $key
if (is_array($data) && isset($allFieldValues[$key])) {
// Would add for example $keyData['acft'] = $_POST['quote']['acft'][0] = 'VQ-BFD';
$keyData[$field] = $allFieldValues[$key];
}
}
var_dump($keyData);
// Would look like array(
// 'dt-flight' => '2018-06-13',
// 'acft' => 'VQ-BFD',
// and so on
// )
$last_id = DBCreate('quote',$keyData,true);
$i++;
}
Although this is not part of your question, I really suggest you also take care of my comment on your question about mysqli_real_escape_string not being a safe way to escape column-names (or table-names and so on). For example with following solution:
function DBCreate($table, array $data, $insertId = false) {
// For each table the known columns
$columns = array( 'quote' => array('dt_flight', 'acft', '...') );
// Verify valid table given
if (!isset($columns[$table])) {
throw new InvalidArgumentException('No such table: ' . $table);
}
// Remove everything from data where the key is not in $columns[$table]
// = Remove everything where the column-name is non-existing or even an attempt to hack your system
$data = array_intersect_key($data, array_fill_keys($columns[$table], null));
if (!count($data)) {
throw new InvalidArgumentException('No (valid) data given at all');
}
// Next, continue with your implementation
}
I'm trying to create a csv file direct from a mysql table using php with an OOP approach. Here is my code:
$mysqli = new mysqli($host, $user, $pass, $database);
// Checking connection.
if ($mysqli->connect_errno) {
printf("Connect failed: %s\n", $mysqli->connect_error);
exit();
}
$query = 'SELECT * FROM table';
$result = $mysqli->query($query);
while ($row = $result->fetch_array()) {
$rows[] = $row;
}
$array = "";
foreach($rows as $row) {
$array .= "\"".$row["genus"].",".$row["species"]."\",";
}
$list = array($array);
header('Content-type: application/ms-excel');
header('Content-Disposition: attachment; filename=export.csv');
$file = fopen("php://output","w");
foreach ($list as $line) {
fputcsv($file,explode(',',$line));
}
fclose($file);
// Free result set.
$result->close();
// Close connection.
$mysqli->close();
My problem is with the following part, when creating the array:
$array = "";
foreach($rows as $row) {
$array .= "\"".$row["genus"].",".$row["species"]."\",";
}
because when I do a print_r ($list); I don't receive the expected array:
Array ( [0] => "homo,sapiens", [1] => "homo,erectus",
[2] => "homo,ergaster", )
but this one:
Array ( [0] => "homo,sapiens","homo,erectus","homo,ergaster", )
and moreover when I create the array manually
$list = array("homo,sapiens","homo,erectus","homo,ergaster");
it works just fine and I receive the correct csv file.
I would appreciate if someone could help me, explain me what I'm doing wrong and what I should do to fix the code.
That's because you're treating $array as a string, not an array.
This should work:
$array = array();
foreach($rows as $row) {
$array[] = "\"".$row["genus"].",".$row["species"]."\",";
}
I think you're expecting the $list = array($array); line to magically convert a comma separated string to individual elements in the array, but it doesn't.
I've got a MySQL table with fields a1,a2,a3,b1,...,d1,d2, each field was declared as a BOOLEAN in the CREATE statement. (I also tried TINYINT(1) but had the same problem).
Then I have this PHP function which receives data from an HTML form:
public function add($a) {
$sql = "INSERT INTO property_classification
(a1,a2,a3,b1,b2,b3,b4,b5,b6,b7,b8,c1,c2,c3,d1,d2)
VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?);";
// creating the classification_id
// e.g. "a1a2a3" => ["a1","a2","a3"]
$classifications = str_split($a['classifications'], 2);
$data = array();
// compile $data array
foreach (self::$classification_fields as $classification) {
// if user array contained any classification, set to true
if (in_array($classification, $classifications)) {
$data[$classification] = "1"; // I tried `true` too
} else {
$data[$classification] = "0"; // I tried `false` here
}
}
// set type for binding PDO params
foreach ($data as $key=>$value) settype($data[$key], 'int'); // tried 'bool'
$this->db->query($sql, $data);
$a['classification_id'] = $this->db->lastInsertId();
$this->log($a['classification_id']); // Output: "0"
...
The output should be a valid ID from 1+, but the insert failed so the lastInsertId() returned 0.
I checked what $sql compiled to, it came to this:
INSERT INTO property_classification (a1,a2,a3,b1,b2,b3,b4,b5,b6,b7,b8,c1,c2,c3,d1,d2) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?);
I also output $data with the code: implode(",",$data); and it gave me this output:
1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
Which was perfect because the input was "a1a2".
The only problem now is I don't understand why the query is failing all the time, because I put the two bits together like so:
INSERT INTO property_classification (a1,a2,a3,b1,b2,b3,b4,b5,b6,b7,b8,c1,c2,c3,d1,d2) VALUES(1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
Then I executed that query in MySQL Query Browser and it worked.
So why is it failing through PDO?
DBO class
function query($sql, $data) {
try {
$this->query = $this->db->prepare($sql);
if (!is_null($data) && is_array($data))
$this->query->execute($data);
else
$this->query->execute();
} catch (PDOException $e) {
array_push($this->log, $e->getMessage());
}
}
Since you're actually passing an associative array to the PDO, you can bind to named parameters. The use of ? or positional placeholders require a standard indexed array. If you're against using named params, just replace $data[$classification] = with $data[] =
Try the below.
public function add($a) {
$sql = "INSERT INTO property_classification
(a1,a2,a3,b1,b2,b3,b4,b5,b6,b7,b8,c1,c2,c3,d1,d2)
VALUES(:a1,:a2,:a3,:b1,:b2,:b3,:b4,:b5,:b6,:b7,:b8,:c1,:c2,:c3,:d1,:d2);";
// creating the classification_id
// e.g. "a1a2a3" => ["a1","a2","a3"]
$classifications = str_split($a['classifications'], 2);
$data = array();
// compile $data array
foreach (self::$classification_fields as $classification)
$data[$classification] = in_array($classification, $classifications) ? 1 : 0;
$this->db->query($sql, $data);
$a['classification_id'] = $this->db->lastInsertId();
$this->log($a['classification_id']); // Output: "0"
I am reading a 'kind' of csv file and exploding it and storing it in array.
The file I am reading has this structure
Id,Log,Res,File
mydb('Test1','log1','Pass','lo1.txt').
mydb('Test2','log2','Pass','lo2.txt').
mydb('Test3','log3','Pass','lo3.txt').
Now what I am trying to do is :
reading the last record in my database, get the Name, lets say in this case 'Test1' and then I am searching through my file and where I can find the position of 'Test1' and get the next lines in the file, extract the ID,s and add it to database.
I am getting the position of desired string in the file, but I am not sure how to get the next lines too.
Here's my code so far.
<?php
mysql_connect("localhost", "root", "") or die(mysql_error());
mysql_select_db("testing") or die(mysql_error());
$result = mysql_query("select ID from table_1 order by S_no DESC limit 1") or die(mysql_error());
$row = mysql_fetch_array( $result );
$a = $row['ID'];
echo 'Present Top Row is '.$a.'<br>';
$addresses = explode("\n", file_get_contents('\\\\fil1\\logs\\tes.pl'));
foreach($addresses as $val)
{
$pos = strstr($val, $a);
if ($pos === false) {
} else {
echo "The string <br> '$a' <br>was found in the string <br>'$val' <br>";
echo " and exists at position <br>$pos<br>";
}
}
Hope I understand you correctly. If so you could just set a flag once the item is found and then explode each line then on a ' character. The second value in the resultant array should then contain the id as per your example.
$addresses = explode("\n", file_get_contents('\\\\fil1\\logs\\tes.pl'));
$bFound = false;
foreach($addresses as $val)
{
if (!$bFound) {
$pos = strstr($val, $a);
if ($pos === false) {
} else {
echo "The string <br> '$a' <br>was found in the string <br>'$val' <br>";
echo " and exists at position <br>$pos<br>";
$bFound = true;
}
}
else {
$aCols = explode("'", $val);
$sId = $aCols[1];
// Now add Id as stored in $sId to DB here
}
}
If id is supposed to be unique, the traffic between the php process and the MySQL server is not the limiting factor and you have to parse each line anyway, you could create a unique index and let MySQL decide whether it accepts the record or not.
Simple example:
$pdo = new PDO('mysql:host=localhost;dbname=test', 'localonly', 'localonly');
$pdo->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );
$pdo->exec('CREATE TEMPORARY TABLE foo (Id varchar(16),Log varchar(16),Res varchar(16),File varchar(16), unique key(Id))');
$stmt = $pdo->prepare('INSERT IGNORE INTO foo (Id,Log,Res,File) VALUES(?,?,?,?)');
$pattern = '/(?<=^mydb\().+(?=\)\.$)/';
$fp = 'dummy';
while ( false!==($row=dummy_fgets($fp)) ) {
if ( preg_match($pattern, $row, $m) ) {
$row = str_getcsv($m[0], ',', "'");
$stmt->execute($row);
echo $row[0], ' -> ', $stmt->rowCount(), "\n";
}
}
function dummy_fgets($dummy) {
static $data = array(
"mydb('Test1','log1','Pass','lo1.txt').\n",
"mydb('Test2','log2','Pass','lo2.txt').\n",
"mydb('Test3','log3','Pass','lo3.txt').\n",
// start again ...like you process the same file again + 1 new record
"mydb('Test1','log1','Pass','lo1.txt').\n",
"mydb('Test2','log2','Pass','lo2.txt').\n",
"mydb('Test3','log3','Pass','lo3.txt').\n",
"mydb('Test4','log3','Pass','lo3.txt').\n"
);
$rv = current($data);
next($data);
return $rv;
}
prints
Test1 -> 1
Test2 -> 1
Test3 -> 1
Test1 -> 0
Test2 -> 0
Test3 -> 0
Test4 -> 1