I'm having a problem inserting my data in database with Codeigniter. I have this testing seed function
$latitude = rand(45.500000 * 1000000, 46.400000 * 1000000) / 1000000;
$longitude = rand(13.600000 * 1000000, 15.500000 * 1000000) / 1000000;
$data = array(
'unique_id' => '12319334',
'latitude' => $latitude,
'longitude' => $longitude,
'coordinates' => "ST_GeomFromText('POINT($latitude $longitude)')",
);
$locationId = $this->Locations->insert($data);
And this is insert function in model
function insert($data, $tableName = "")
{
if ($tableName == "") {
$tableName = $this->table;
}
$this->db->insert($tableName, $data);
return $this->db->insert_id();
}
And this is a query that happens
INSERT
INTO
`locations`(
`unique_id`,
`latitude`,
`longitude`,
`coordinates`
)
VALUES(
'ZTE1NGY2YT',
45.990292,
14.948462,
'ST_GeomFromText(\'POINT(45.582315 14.821478)\')'
)
Error i get is Cannot get geometry object from data you send to the GEOMETRY field
After some testing in phpmyadmin i figure out that query for inserting this kind of data should look like this
INSERT
INTO
`locations`(
`unique_id`,
`latitude`,
`longitude`,
`coordinates`
)
VALUES(
'ZTE1NGY2YT',
45.990292,
14.948462,
ST_GeomFromText('POINT(45.582315 14.821478)')
)
So somehow i need to get rid off single quotes (') in 'ST_GeomFromText(\'POINT(45.582315 14.821478)\')' line
Anybody got idea how to properly prepare data (without executing direct query, since there is a lot more data to store) so it can be process properly?
If you need any additional information please let me know and i will provide.
Thank you!
Actually this did it
$this->db->set('coordinates', "ST_GeomFromText('POINT($latitude $longitude)')", false);
So i made a small hack on my model (i know its not pretty but it works for now) and data is inserted..
function insert($data, $tableName = "")
{
foreach ($data as $key => $value) {
if ($key == 'coordinates') {
$this->db->set('coordinates', $value, false);
unset($data['coordinates']);
}
}
if ($tableName == "") {
$tableName = $this->table;
}
$this->db->insert($tableName, $data);
return $this->db->insert_id();
}
Unfortunately, CodeIgniter's database escape logic assumes that the pattern is field_name => rendered data. This means that you can't use things like NOW() or MD5(), or ST_GeomFromText. You can see the logic here:
foreach ($data as $key => $val)
{
$fields[] = $this->escape_identifiers($key);
$values[] = $this->escape($val);
}
The best option I've managed is to extend the DB driver and have the model call a custom function. The other option is to drop back into raw SQL and manually escape the values.
Related
I've been trying to put together functions in a more secure way that keeps us safe from injection or manipulating inserts by calling different columns to be updated. In your opinion, is this function safe at all, and if not what would you suggest is a better way to do it, and why.
This function is called when a user updates their profile, or specific parts of their profile, as you can see I've made an array with items which is all they can update in that table. Also, the user_id I am getting is from the secure encrypted JSON token that's attached to their session, they are not sending that. Thanks for your time.
function updateProfile( $vars, $user_id ) {
$db = new Database();
$update_string = '';
$varsCount = count($vars);
$end = ',';
$start = 1;
$safeArray = array( "gradYear", "emailAddress", "token", "iosToken", "country",
"birthYear", "userDescription" );
foreach($vars as $key => $value) {
if(in_array( $key, $safeArray )) {
if($start == $varsCount) {
$end = '';
}
$update_string .= $key . '=' . '"' . $value . '"' . $end;
}
$start++;
}
if($start > 0) {
$statement = "update users set " . $update_string . " where userId = '$user_id'";
$query = $db->updateQuery( $statement );
if($query) {
$response = array( "response" => 200 );
} else {
$response = array( "response" => 500, "title" => "An unknown error occured,
please try again");
}
}
As the comments above suggest, it's worth using query parameters to protect yourself from SQL injection.
You asked for an example of how anything malicious could be done. In fact, it doesn't even need to be malicious. Any innocent string that legitimately contains an apostrophe could break your SQL query. Malicious SQL injection takes advantage of that weakness.
The weakness is fixed by keeping dynamic values separate from your SQL query until after the query is parsed. We use query parameter placeholders in the SQL string, then use prepare() to parse it, and after that combine the values when you execute() the prepared query. That way it remains safe.
Here's how I would write your function. I'm assuming using PDO which supports named query parameters. I recommend using PDO instead of Mysqli.
function updateProfile( $vars, $userId ) {
$db = new Database();
$safeArray = [
"gradYear",
"emailAddress",
"token",
"iosToken",
"country",
"birthYear",
"userDescription",
];
// Filter $vars to include only keys that exist in $safeArray.
$data = array_intersect_keys($vars, array_flip($safeArray));
// This might result in an empty array if none of the $vars keys were valid.
if (count($data) == 0) {
trigger_error("Error: no valid columns named in: ".print_r($vars, true));
$response = ["response" => 400, "title" => "no valid fields found"];
return $response;
}
// Build list of update assignments for SET clause using query parameters.
// Remember to use back-ticks around column names, in case one conflicts with an SQL reserved keyword.
$updateAssignments = array_map(function($column) { return "`$column` = :$column"; }, array_keys($data));
$updateString = implode(",", $updateAssignments);
// Add parameter for WHERE clause to $data.
// This must be added after $data is used to build the update assignments.
$data["userIdWhere"] = $userId;
$sqlStatement = "update users set $updateString where userId = :userIdWhere";
$stmt = $db->prepare($sqlStatement);
if ($stmt === false) {
$err = $db->errorInfo();
trigger_error("Error: {$err[2]} preparing SQL query: $sqlStatement");
$response = ["response" => 500, "title" => "database error, please report it to the site administrator"];
return $response;
}
$ok = $stmt->execute($data);
if ($ok === false) {
$err = $stmt->errorInfo();
trigger_error("Error: {$err[2]} executing SQL query: $sqlStatement");
$response = ["response" => 500, "title" => "database error, please report it to the site administrator"];
return $response;
}
$response = ["response" => 200, "title" => "update successful"];
return $response;
}
In addition to the excellent Bill's answer, one little suggestion: always make your methods to do one thing at a time. If a method's job is to update a database, then it should only update a database and nothing else, the HTTP interaction included. Imagine this method could be used in non-AJAX context or without a web-server at all but from a command line utility. Those HTTP codes and JSON responses would look completely off the track. So have two classes: one to update the database and one to interact with the client. It will make your code much cleaner and reusable.
Also, never create a new connection to the database for the every query. Instead, have a ready made connection and use it for all database interactions.
function updateProfile($db, $vars, $userId )
{
$safeArray = array( "gradYear", "emailAddress", "token", "iosToken", "country",
"birthYear", "userDescription" );
// let's check if all columns are safe
if (array_diff(array_keys($vars), $safeArray)) {
throw new InvalidArgumentException("Unknown columns provided");
}
$updateAssignments = array_map(function($column) {
return "`$column` = :$column"; }, array_keys($vars)
);
$updateString = implode(",", $updateAssignments);
$vars["userIdWhere"] = $userId;
$sqlStatement = "update users set $updateString where userId = :userIdWhere";
$db->prepare($sqlStatement)->execute($vars);
}
See, it makes your code slim and readable. And, above all - reusable. You don't have to make your methods bloated. PHP is a very concise language, if used properly
Is there a way with MySQL to specify "use previous / inherit / no change / existing value"?
Rather than needing to pull the current data from the database and use it, or have a customized database function excluding editing those columns.
if(x > y) {
$role_id = 3;
} else {
$role_id = '#no-change'; // Is there a way to do this? (not proper SQL syntax)
}
$update_user = $this->db->update('users',
array(
'first_name' => filterName($post['first_name']),
'last_name' => filterName($post['last_name']),
'email' => filterEmail($post['email']),
'role_id' => $role_id,
), $user_id_to_edit, 'user_id');
In a case like this where the db function is using prepared statements (not shown) I can't use the column name as to reflect the current value.
Is there such a MySQL function / variable that will essentially "ignore" updating that column? (just leave the existing value)
UPDATE: Here's the Update function:
public function update($table, $data, $where_id, $column = 'user_id') {
// Check for $table or $data not set
if (( empty( $table ) || empty( $data )) || empty($data) ) {
return false;
}
// Initiate variable to append to
$placeholders ='';
// Parse data for column and placeholder names
foreach ($data as $key => $value) {
$placeholders .= sprintf('%s=:%s,', $key, $key);
}
// Trim excess commas
$placeholders = rtrim($placeholders, ',');
// Append where ID to $data
$data['where_id'] = $where_id;
// Prepary our query for binding
$stmt = $this->db->prepare("UPDATE {$table} SET {$placeholders} WHERE $column = :where_id");
// Execute the query
$stmt->execute($data);
// Check for successful insertion
if ( $stmt->rowCount() ) {
return true;
}
return false;
}
You could try this:
$data = array(
'first_name' => filterName($post['first_name']),
'last_name' => filterName($post['last_name']),
'email' => filterEmail($post['email']))
if(x > y) {
$data['role_id'] = 3;
}
$update_user = $this->db->update('users', $data, $user_id_to_edit, 'user_id');
That way, you can customize $data before hand if you like.
I should also mention, if you're concerned about redundancy, you can put your data sanitation inside a function. Something along the lines of:
function sanitize($data) {
if(x > y) {
$data['role_id'] = 3;
} else {
if(isset($data['role_id']) {
unset($data['role_id']);
}
}
return $data;
}
$data = array(
'first_name' => filterName($post['first_name']),
'last_name' => filterName($post['last_name']),
'email' => filterEmail($post['email']))
$update_user = $this->db->update('users', sanitize($data), $user_id_to_edit, 'user_id');
Edit: Something I should mention is that, if we're talking pure SQL, all you need to do is omit those columns from the query, so:
UPDATE table SET Col1=val1, Col2=val2, Col3=val3 WHERE id=val
But if for some reason you don't want to update Col3, just remove it from the query:
UPDATE table SET Col1=val1, Col2=val2 WHERE id=val
Since you have a function that just adds whatever you give it, you just need to sanitize the data ahead of time. That's probably the best way to do it.
I'm trying to build a script where I need to read a txt file and execute some process with the lines on the file. For example, I need to check if the ID exists, if the information has updated, if yes, then update the current table, if no, then insert a new row on another temporary table to be manually checked later.
These files may contain more than 20,30 thousand lines.
When I just read the file and print some dummie content from the lines, it takes up to 40-50ms. However, when I need to connect to the database to do all those verifications, it stops before the end due to the timeout.
This is what I'm doing so far:
$handle = fopen($path, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
$segment = explode('|', $buffer);
if ( strlen($segment[0]) > 6 ) {
$param = [':code' => intval($segment[0])];
$codeObj = Sql::exec("SELECT value FROM product WHERE code = :code", $param);
if ( !$codeObj ) {
$param = [
':code' => $segment[0],
':name' => $segment[1],
':value' => $segment[2],
];
Sql::exec("INSERT INTO product_tmp (code, name, value) VALUES (:code, :name, :value)", $param);
} else {
if ( $codeObj->value !== $segment[2] ) {
$param = [
':code' => $segment[0],
':value' => $segment[2],
];
Sql::exec("UPDATE product SET value = :value WHERE code = :code", $param);
}
}
}
}
fclose($handle);
}
And this is my Sql Class to connect with PDO and execute the query:
public static function exec($sql, $param = null) {
try {
$conn = new PDO('mysql:charset=utf8mb4;host= '....'); // I've just deleted the information to connect to the database (password, user, etc.)
$q = $conn->prepare($sql);
if ( isset($param) ) {
foreach ($param as $key => $value) {
$$key = $value;
$q->bindParam($key, $$key);
}
}
$q->execute();
$response = $q->fetchAll();
if ( count($response) ) return $response;
return false;
} catch(PDOException $e) {
return 'ERROR: ' . $e->getMessage();
}
}
As you can see, each query I do through Sql::exec(), is openning a new connection. I don't know if this may be the cause of such a delay on the process, because when I don't do any Sql query, the script run within ms.
Or what other part of the code may be causing this problem?
First of all, make your function like this,
to avoid multiple connects and also o get rid of useless code.
public static function getPDO() {
if (!static::$conn) {
static::$conn = new PDO('mysql:charset=utf8mb4;host= ....');
}
return static::$conn;
}
public static function exec($sql, $param = null) {
$q = static::getPDO()->prepare($sql);
$q->execute($param);
return $q;
}
then create unique index for the code field
then use a single INSERT ... ON DUPLICATE KEY UPDATE query instead of your thrree queries
you may also want to wrap your inserts in a transaction, it may speed up the inserts up to 70 times.
I want to query dynamically based on payload(json) from database.
Example:
$data = [{"key":"age","relation":">","value":"15"},{"operator":"OR"},{"key":"age","relation":"<=","value":"20"}]
I want to do query based on that payload.
Right now what I'm doing is:
$query = User::all();
$payload = json_decode($data, true);
foreach($payload as $value){
if ($value['key'] == 'age') {
$query = $query->where('birthday', $value['relation'], Carbon::now()->subYears($value['age'])->format('Y-m-d');)
}
if($value['key'] == 'gender'{
$query = $query->where('gender', $value['relation'], $value['gender']);
}
}
The problem is yes it can work, but I don't think this is best approach. I don't get any solution to use the "operator" key. Operator usage is to change where to orWhere.
Any solution or tips to make it call dynamically like this?. I want my column at DB neat and simple. I can only think this way.
Thanks!
Encountering this problem, I would go with Local Query Scopes. In this approach You create a model method named scopeJson() or whatever you feel better with to handl all conditions inside. I tried to handle most conditions here not only single where and orWhere. I assumed that your payload contains only one builder at a time.
public function scopeJson($query, $json)
{
$wheres = [
'between' => ['whereBetween', 'not' => 'whereNotBetween'],
'null' => ['whereNull', 'not' => 'whereNotNull'],
'or' => ['orWhere', 'not' => 'orWhereNot'],
'in' => ['whereIn', 'not' => 'whereNotIn'],
'and' => ['where', 'not' => 'orWhereNot'],
'raw' => 'whereRaw'
];
$builder = json_decode($json);
if (count($builder) > 0) {
$query->where(
$builder[0]->key,
$builder[0]->relation,
$builder[0]->value
);
// notBetween, notNull, notOr, notIn, notAnd
if (stripos($builder[1]->operator, 'not') !== false) {
$whereCondition = $wheres[strtolower(substr($builder[1]->operator, 3))]['not'];
} else {
$whereCondition = $wheres[strtolower($builder[1]->operator)];
}
if (count($builder[2]) == 3) {
if ($whereCondition == 'whereRaw') {
$query->$whereCondition(implode(" ", $builder[2]));
} else {
// where, whereNot
$query->$whereCondition(
$builder[2]->key,
$builder[2]->relation,
$builder[2]->value
);
}
} elseif (count($builder[2]) == 2) {
// whereBetween, whereNotBetween, where, whereNot
$query->$whereCondition(
$builder[2]->key,
$builder[2]->value
);
} elseif (count($builder[2]) == 1) {
// whereNull, whereNotNull, whereRaw
$query->$whereCondition(
$builder[2]->key ?? $builder[2]->value // PHP 7.0 Null Coalescing Operator
);
}
}
return $query;
}
If this method is defined within your User's model then you can use it this way:
$users = User::json($data)->get();
PS: Although it should work, I didn't test it.
You can do raw query like this.
$data = '[{"key":"age","relation":">","value":"15"},{"operator":"OR"},{"key":"age","relation":"<=","value":"20"}]';
$query = "SELECT * FROM tablename WHERE";
$payload = json_decode($data, true);
foreach ($payload as $value) {
if (isset($value['operator'])) {
$query .= " " . $value['operator'];
} else {
if ($value['key'] == 'age') {
$query .= " 'birthday' " . $value['relation'] . " " . Carbon::now()->subYears($value['value'])->format('Y-m-d');
}
if ($value['key'] == 'gender') {
$query .= " 'gender' " . $value['relation'] . " " . $value['gender'];
}
}
}
This results in a query like this :
SELECT * FROM tablename WHERE 'birthday' > 2001-07-02 OR 'birthday' <= 1996-07-02
Of course, you might use printf() for formatting and making this cleaner in some other way but this will get you started hopefully.
You can use a variable function name to add your orWhere logic:
$query = User::all();
$payload = json_decode($data, true);
$function = 'where';
foreach($payload as $value){
if(isset($value['operator'])){
$function = $value['operator'] == 'OR' ? 'orWhere' : 'where';
} else {
if ($value['key'] == 'age') {
$query = $query->$function('birthday', $value['relation'], Carbon::now()->subYears($value['age'])->format('Y-m-d');)
} else {
$query = $query->$function($value['key'], $value['relation'], $value['value']);
}
}
}
As long as your json data doesn't match your database (ie. the json has age but the database has birthday) you will not be able to avoid having that if/else statement. That custom logic will have to remain.
Ultimately this idea is its own limiter because the stored queries will have to represent the current state of the database. This means that the maintenance cost of these queries will be significant the moment data is stored in a different way -- if you changed the column birthday to date_of birth all of your stored queries will break. Avoid this.
Instead this goal is better achieved by storing the queries on the Model using Query Scopes and Relationships. If you still need a dynamic list of requests you can store keywords that are associated with the queries and loop through them.
Just getting into OOP, and changing from mysql queries to PDO. I am trying to create a class that will return the column names and meta data for a table. This is so I can output copy/paste data for all the tables I use. I have used such a tool, based on the mysql extension, for ages and it spits out variants such as complete SELECT/INSERT/UPDATE queries. Amongst other things I now want to add DECLARE listings for Stored Procedures - so getting meta data like type and length are essential. With about 150 tables across two schemas, such automation is essential.
With uncertainty about the reliability of getColumnMeta I hunted for code and found what looked good in a Sitepoint answer. I have attempted to wrap it in a class and mimic its original context but I am simply getting a number 1 when I try to echo or print_r the response. I have also had 'not an object' error messages while trying solutions.
This is the calling code
$db_host="localhost";
$db_username='root';
$db_pass='';
$db_name='mydatabase';
try{
$db= new PDO('mysql:host='.$db_host.';dbname='.$db_name,$db_username,$db_pass, array(PDO::ATTR_PERSISTENT=>false));
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_WARNING);
}
catch(PDOException $e){echo "Error: ".$e->getMessage()."<br />"; die(); }
include 'ColMetaData.php'; //the file containing the class for getting a column listing for each table
$coldat= new supplyColumnMeta($db);
$tablemet=$coldat->getColumnMeta('groups'); // a manual insertion of a table name for testing
echo $tablemet;
And this is the class that sits in the include file
class supplyColumnMeta{
public function __construct($db){
$this->db=$db;
}
/**
* Automatically get column metadata
*/
public function getColumnMeta($table)
{$this->tableName=$table;
// Clear any previous column/field info
$this->_fields = array();
$this->_fieldMeta = array();
$this->_primaryKey = NULL;
// Automatically retrieve column information if column info not specified
if(count($this->_fields) == 0 || count($this->_fieldMeta) == 0)
{
// Fetch all columns and store in $this->fields
$columns = $this->db->query("SHOW COLUMNS FROM " . $this->tableName, PDO::FETCH_ASSOC);
foreach($columns as $key => $col)
{
// Insert into fields array
$colname = $col['Field'];
$this->_fields[$colname] = $col;
if($col['Key'] == "PRI" && empty($this->_primaryKey)) {
$this->_primaryKey = $colname;
}
// Set field types
$colType = $this->parseColumnType($col['Type']);
$this->_fieldMeta[$colname] = $colType;
}
}
return true;
}
protected function parseColumnType($colType)
{
$colInfo = array();
$colParts = explode(" ", $colType);
if($fparen = strpos($colParts[0], "("))
{
$colInfo['type'] = substr($colParts[0], 0, $fparen);
$colInfo['pdoType'] = '';
$colInfo['length'] = str_replace(")", "", substr($colParts[0], $fparen+1));
$colInfo['attributes'] = isset($colParts[1]) ? $colParts[1] : NULL;
}
else
{
$colInfo['type'] = $colParts[0];
}
// PDO Bind types
$pdoType = '';
foreach($this->_pdoBindTypes as $pKey => $pType)
{
if(strpos(' '.strtolower($colInfo['type']).' ', $pKey)) {
$colInfo['pdoType'] = $pType;
break;
} else {
$colInfo['pdoType'] = PDO::PARAM_STR;
}
}
return $colInfo;
}
/**
* Will attempt to bind columns with datatypes based on parts of the column type name
* Any part of the name below will be picked up and converted unless otherwise sepcified
* Example: 'VARCHAR' columns have 'CHAR' in them, so 'char' => PDO::PARAM_STR will convert
* all columns of that type to be bound as PDO::PARAM_STR
* If there is no specification for a column type, column will be bound as PDO::PARAM_STR
*/
protected $_pdoBindTypes = array(
'char' => PDO::PARAM_STR,
'int' => PDO::PARAM_INT,
'bool' => PDO::PARAM_BOOL,
'date' => PDO::PARAM_STR,
'time' => PDO::PARAM_INT,
'text' => PDO::PARAM_STR,
'blob' => PDO::PARAM_LOB,
'binary' => PDO::PARAM_LOB
);
}
Here's the problem:
public function getColumnMeta($table)
{
$this->tableName=$table;
// Clear any previous column/field info
$this->_fields = array();
$this->_fieldMeta = array();
$this->_primaryKey = NULL;
// Automatically retrieve column information if column info not specified
if(count($this->_fields) == 0 || count($this->_fieldMeta) == 0)
{
// Fetch all columns and store in $this->fields
$columns = $this->db->query("SHOW COLUMNS FROM " . $this->tableName, PDO::FETCH_ASSOC);
foreach($columns as $key => $col)
{
// Insert into fields array
$colname = $col['Field'];
$this->_fields[$colname] = $col;
if($col['Key'] == "PRI" && empty($this->_primaryKey)) {
$this->_primaryKey = $colname;
}
// Set field types
$colType = $this->parseColumnType($col['Type']);
$this->_fieldMeta[$colname] = $colType;
}
}
return true;//<<--- not returning an object/array!
}
Your getColumnMeta method returns a boolean, true. The string representation of this value is, of course, 1. If you want this method to return all of the meta-data, change the return statement to something like:
return array(
'fields' => $this->_fields,
'meta' => $this->_fieldMeta,
'primary' => $this->_primaryKey
);
There are some other issues with your code, too, but seeing as this is not codereview.stackexchange, I'm not going to go into too much details. I will say this, though: Please, try to follow the coding standards that most major players adhere to: these standards can be found here: PHP-FIG.
Oh, and if you want to show the meta-data, don't echo them, but rather var_dump or print_r them, seeing as you return an array or an object.
Or at least echo json_encode($instance->getColumnMeta($table)); to get a correct string representation of the returned value(s).