I'm struggling with implementing PHP and MySQL Transactions. The script receives a SQL-statement along with some bindparameters through a blocking redisqueue. Everything is passed to a function 'do_transaction' which keeps track of the number of statements received.
I've debugged the PDO statement (after it has been processed) with PdoDebugger and the output is correct:
UPDATE bla SET processed = 1, severity_ou1 = 'low',
severity_ou2 = 'low', severity_ou3 = 'low', severity_ou4 = 'low',
severity_ou5 = 'low', saved = '1', hname = '1', sname = '1', if = '1',
v = '1', translated = 'blablabla.', filtered = 1, repeated = '1',
excessed = '1', eventfilterid = '212', building = '1', floor = '1'
WHERE id = '121614624'
global $batchcount;
$batchcount = 1;
while(true){
$redis = new Redis();
$redis->connect('xxx', xxx);
$sqlbatch = $redis->blpop('xxx:xxx:sqlfiltermatch', 0);
// blpop returns array: 0 has key, 1 has data.
if(is_array($sqlbatch)){
if(isJson($sqlbatch[1])){
$batchstatements = array();
$batchstatements[] = json_decode($sqlbatch[1], true);
// Get statement and bindparams.
$sqlstatement = $batchstatements[0]['statement'];
$bindparams = $batchstatements[0]['bindparams'];
// Replace empty bindparams.
foreach($bindparams as $column => $value){
if(is_null($value)){ $bindparams[$column] = '1'; }
if(empty($value)){ $bindparams[$column] = '1'; }
}
}
$batchcount++;
do_transaction($sqlstatement, $bindparams, $batchcount);
}
}
function do_transaction($sqlstatement, $bindparams){
global $batchcount;
if($batchcount >= 4){
try {
// Setup DB
$db = new PDO('mysql:host=xxx;dbname=xxx;charset=utf8', 'xxx', 'xxx', array(PDO::ATTR_PERSISTENT => true, PDO::ATTR_AUTOCOMMIT => FALSE, PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING));
echo $db->getAttribute(PDO::ATTR_AUTOCOMMIT)."\n\n";
$db->beginTransaction();
$stmt = $db->prepare($sqlstatement);
// Setup bindparams.
foreach($bindparams as $column => $value){
$stmt->bindParam(":$column", $value);
}
$stmt->execute() or die(print_r($stmt->errorInfo(), true));
echo PdoDebugger::show($sqlstatement, $bindparams)."\n";
$db->commit();
} catch(PDOExecption $e){
//$db->rollback();
print_r("ERROR"); exit;
}
$batchcount = 0;
}
$batchcount++;
}
I've made sure that AUTOCOMMIT = FALSE. Where in "do_transaction" does it go wrong?
There is no point in using transactions this way.
So, just leave them alone.
function do_query($db, $sqlstatement, $bindparams){
$stmt = $db->prepare($sqlstatement);
$stmt->execute($bindparams);
return $stmt;
}
is all the code you actually need.
Use it this way
$db = new PDO('mysql:host=xxx;dbname=xxx;charset=utf8', 'xxx', 'xxx',
array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION)
);
while(true){
// whatever redis code goes here
do_query($db, $sqlstatement, $bindparams);
}
Nota bene: If you want to make your inserts faster, you should ask a question titled "How to make inserts faster", no "My transactions do not work".
But your idea of inserting transactions is wrong too.
A single transaction (in terms of the business logic) have to be written to database as soon as possible, without interfering with other transactions. Means you should never couple different business logic transactions within single database transacion. Because error in single business logic transaction will ruin whole batch. So - just write them separately.
Related
I've been trying to put together functions in a more secure way that keeps us safe from injection or manipulating inserts by calling different columns to be updated. In your opinion, is this function safe at all, and if not what would you suggest is a better way to do it, and why.
This function is called when a user updates their profile, or specific parts of their profile, as you can see I've made an array with items which is all they can update in that table. Also, the user_id I am getting is from the secure encrypted JSON token that's attached to their session, they are not sending that. Thanks for your time.
function updateProfile( $vars, $user_id ) {
$db = new Database();
$update_string = '';
$varsCount = count($vars);
$end = ',';
$start = 1;
$safeArray = array( "gradYear", "emailAddress", "token", "iosToken", "country",
"birthYear", "userDescription" );
foreach($vars as $key => $value) {
if(in_array( $key, $safeArray )) {
if($start == $varsCount) {
$end = '';
}
$update_string .= $key . '=' . '"' . $value . '"' . $end;
}
$start++;
}
if($start > 0) {
$statement = "update users set " . $update_string . " where userId = '$user_id'";
$query = $db->updateQuery( $statement );
if($query) {
$response = array( "response" => 200 );
} else {
$response = array( "response" => 500, "title" => "An unknown error occured,
please try again");
}
}
As the comments above suggest, it's worth using query parameters to protect yourself from SQL injection.
You asked for an example of how anything malicious could be done. In fact, it doesn't even need to be malicious. Any innocent string that legitimately contains an apostrophe could break your SQL query. Malicious SQL injection takes advantage of that weakness.
The weakness is fixed by keeping dynamic values separate from your SQL query until after the query is parsed. We use query parameter placeholders in the SQL string, then use prepare() to parse it, and after that combine the values when you execute() the prepared query. That way it remains safe.
Here's how I would write your function. I'm assuming using PDO which supports named query parameters. I recommend using PDO instead of Mysqli.
function updateProfile( $vars, $userId ) {
$db = new Database();
$safeArray = [
"gradYear",
"emailAddress",
"token",
"iosToken",
"country",
"birthYear",
"userDescription",
];
// Filter $vars to include only keys that exist in $safeArray.
$data = array_intersect_keys($vars, array_flip($safeArray));
// This might result in an empty array if none of the $vars keys were valid.
if (count($data) == 0) {
trigger_error("Error: no valid columns named in: ".print_r($vars, true));
$response = ["response" => 400, "title" => "no valid fields found"];
return $response;
}
// Build list of update assignments for SET clause using query parameters.
// Remember to use back-ticks around column names, in case one conflicts with an SQL reserved keyword.
$updateAssignments = array_map(function($column) { return "`$column` = :$column"; }, array_keys($data));
$updateString = implode(",", $updateAssignments);
// Add parameter for WHERE clause to $data.
// This must be added after $data is used to build the update assignments.
$data["userIdWhere"] = $userId;
$sqlStatement = "update users set $updateString where userId = :userIdWhere";
$stmt = $db->prepare($sqlStatement);
if ($stmt === false) {
$err = $db->errorInfo();
trigger_error("Error: {$err[2]} preparing SQL query: $sqlStatement");
$response = ["response" => 500, "title" => "database error, please report it to the site administrator"];
return $response;
}
$ok = $stmt->execute($data);
if ($ok === false) {
$err = $stmt->errorInfo();
trigger_error("Error: {$err[2]} executing SQL query: $sqlStatement");
$response = ["response" => 500, "title" => "database error, please report it to the site administrator"];
return $response;
}
$response = ["response" => 200, "title" => "update successful"];
return $response;
}
In addition to the excellent Bill's answer, one little suggestion: always make your methods to do one thing at a time. If a method's job is to update a database, then it should only update a database and nothing else, the HTTP interaction included. Imagine this method could be used in non-AJAX context or without a web-server at all but from a command line utility. Those HTTP codes and JSON responses would look completely off the track. So have two classes: one to update the database and one to interact with the client. It will make your code much cleaner and reusable.
Also, never create a new connection to the database for the every query. Instead, have a ready made connection and use it for all database interactions.
function updateProfile($db, $vars, $userId )
{
$safeArray = array( "gradYear", "emailAddress", "token", "iosToken", "country",
"birthYear", "userDescription" );
// let's check if all columns are safe
if (array_diff(array_keys($vars), $safeArray)) {
throw new InvalidArgumentException("Unknown columns provided");
}
$updateAssignments = array_map(function($column) {
return "`$column` = :$column"; }, array_keys($vars)
);
$updateString = implode(",", $updateAssignments);
$vars["userIdWhere"] = $userId;
$sqlStatement = "update users set $updateString where userId = :userIdWhere";
$db->prepare($sqlStatement)->execute($vars);
}
See, it makes your code slim and readable. And, above all - reusable. You don't have to make your methods bloated. PHP is a very concise language, if used properly
I'm trying to build a script where I need to read a txt file and execute some process with the lines on the file. For example, I need to check if the ID exists, if the information has updated, if yes, then update the current table, if no, then insert a new row on another temporary table to be manually checked later.
These files may contain more than 20,30 thousand lines.
When I just read the file and print some dummie content from the lines, it takes up to 40-50ms. However, when I need to connect to the database to do all those verifications, it stops before the end due to the timeout.
This is what I'm doing so far:
$handle = fopen($path, "r") or die("Couldn't get handle");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
$segment = explode('|', $buffer);
if ( strlen($segment[0]) > 6 ) {
$param = [':code' => intval($segment[0])];
$codeObj = Sql::exec("SELECT value FROM product WHERE code = :code", $param);
if ( !$codeObj ) {
$param = [
':code' => $segment[0],
':name' => $segment[1],
':value' => $segment[2],
];
Sql::exec("INSERT INTO product_tmp (code, name, value) VALUES (:code, :name, :value)", $param);
} else {
if ( $codeObj->value !== $segment[2] ) {
$param = [
':code' => $segment[0],
':value' => $segment[2],
];
Sql::exec("UPDATE product SET value = :value WHERE code = :code", $param);
}
}
}
}
fclose($handle);
}
And this is my Sql Class to connect with PDO and execute the query:
public static function exec($sql, $param = null) {
try {
$conn = new PDO('mysql:charset=utf8mb4;host= '....'); // I've just deleted the information to connect to the database (password, user, etc.)
$q = $conn->prepare($sql);
if ( isset($param) ) {
foreach ($param as $key => $value) {
$$key = $value;
$q->bindParam($key, $$key);
}
}
$q->execute();
$response = $q->fetchAll();
if ( count($response) ) return $response;
return false;
} catch(PDOException $e) {
return 'ERROR: ' . $e->getMessage();
}
}
As you can see, each query I do through Sql::exec(), is openning a new connection. I don't know if this may be the cause of such a delay on the process, because when I don't do any Sql query, the script run within ms.
Or what other part of the code may be causing this problem?
First of all, make your function like this,
to avoid multiple connects and also o get rid of useless code.
public static function getPDO() {
if (!static::$conn) {
static::$conn = new PDO('mysql:charset=utf8mb4;host= ....');
}
return static::$conn;
}
public static function exec($sql, $param = null) {
$q = static::getPDO()->prepare($sql);
$q->execute($param);
return $q;
}
then create unique index for the code field
then use a single INSERT ... ON DUPLICATE KEY UPDATE query instead of your thrree queries
you may also want to wrap your inserts in a transaction, it may speed up the inserts up to 70 times.
When I am doing a simple insert via the PHP mysqli API with prepared statements on Windows within a PHP process, the defined AUTO_INCREMENT column is increased by 2 instead of 1:
INSERT INTO `table` (`name`) VALUES (?)
It get increased by 1 when doing multiple inserts (one by one in separate transactions) within one PHP process.
It always get increased by 1 when I use the same SQL query via phpmyadmin.
There are no other INSERT or UPDATE statements before or after the mentioned INSERT. Only a SHOW and some SELECT statements before.
I cannot find the cause for this problem. What can be the causes for such a behaviour?
Here the main code parts:
<?php
class DB
{
private function __construct($host, $username, $password, $schema, $port, $socket)
{
if(is_null(self::$DB))
{
self::$DB = new \MySQLi((string) $host, (string) $username, (string) $password, (string) $schema, (int) $port, (string) $socket);
self::$DB->set_charset('utf8');
}
}
// [...]
public function __destruct()
{
if(!is_null(self::$DB))
self::$DB->close();
}
// [...]
public static function connect($host = '', $username = '', $password = '', $schema = '', $port = 0, $socket = '')
{
if(is_null(self::$instance))
{
$MD = new \MySQLi_driver();
$MD->report_mode = MYSQLI_REPORT_ERROR | MYSQLI_REPORT_STRICT;
// [...]
self::$instance = new self($host, $username, $password, $schema, $port, $socket);
}
return self::$instance;
}
// [...]
public static function __callStatic($name, $args)
{
self::connect();
switch(true)
{
case in_array($name, array('insert', 'select', 'update', 'delete', 'show', 'describe', 'explain')):
$query = isset($args[0]) ? (string) $args[0] : '';
$vals = isset($args[1]) ? $args[1] : array();
$select = isset($args[2]) ? trim((string) $args[2]) : 'array';
$empty = isset($args[3]) ? $args[3] : array();
$types = isset($args[4]) ? trim((string) $args[4]) : '';
return self::dml(in_array($name, array('show', 'describe', 'explain')) ? 'select' : $name, $query, $vals, $select, $empty, $types);
break;
}
//[...]
}
// [...]
public static function dml($type, $query, $vals = array(), $select = 'array', $empty = array(), $types = '')
{
// [...]
if(!empty($vals) || mb_strpos($query,'?') !== false)
{
if(!$stmt = self::$DB->prepare($query))
throw new DBException('Failed to prepare statement '.htmlspecialchars($query).PHP_EOL.self::$DB->error.' ('.self::$DB->sqlstate.').');
$args = array();
if(empty($types))
{
foreach($vals as &$val)
{
$t = gettype($val);
if($t == 'string' || $t == 'NULL')
$types.= 's';
elseif($t == 'integer' || $t == 'boolean')
$types.= 'i';
elseif($t == 'double' || $t == 'float')
$types.= 'd';
else
throw new DBException('Its not possible to automatically assign a value of type '.$t.'. Please specify the corresponding types manually.');
$args[] = $val;
}
}
else
{
foreach($vals as &$val)
$args[] = $val;
}
array_unshift($args, $types);
$RC = new \ReflectionClass($stmt);
$RM = $RC->getMethod('bind_param');
if(!$RM->invokeArgs($stmt, $args))
throw new DBException('Failed to bind params.'.PHP_EOL.self::$DB->error.' ('.self::$DB->sqlstate.').');
if(!$stmt->execute())
throw new DBException('Failed to execute Statement.'.PHP_EOL.self::$DB->error.' ('.self::$DB->sqlstate.').');
if($type == 'select')
{
// [...]
}
else
{
$return = $type == 'insert' && self::$DB->insert_id > 0 ? self::$DB->insert_id : self::$DB->affected_rows;
$stmt->close();
return $return;
}
}
else
{
// [...]
}
}
}
?>
AND:
echo DB::insert("INSERT INTO `table` (`name`) VALUES (?)", ["test"]);
The Query-Log shows 2 inserts. This causes the additional increment. But, when i put an echo into the corresponding DB::dml()-method, its only outputted once, thus called once. The mysql-query-log:
151026 12:54:49 3 Connect XXX#localhost on XXX
3 Query SET NAMES utf8
3 Query SELECT DATABASE() AS `schema`
3 Query SHOW GRANTS FOR CURRENT_USER
3 Prepare INSERT INTO `table` (`name`) VALUES (?)
3 Execute INSERT INTO `table` (`name`) VALUES ('testinsert22')
3 Close stmt
3 Quit
4 Connect XXX#localhost on XXX
4 Query SET NAMES utf8
4 Query SELECT DATABASE() AS `schema`
4 Query SHOW GRANTS FOR CURRENT_USER
4 Prepare INSERT INTO `table` (`name`) VALUES (?)
4 Execute INSERT INTO `table` (`name`) VALUES ('testinsert22')
4 Close stmt
4 Quit
Ok. I finally got it.
The problem is: I redirect all requests to my index.php and do all the rest with PHP. I tested some classes behaviour with simple outputs. One of the tests lead to the problem above.
I checked the apache access log and noticed repeated favicon.ico requests. These requests - automatically produced by major browsers (see also How to prevent favicon.ico requests?) - also got to the index.php - and looks like some redirect to the browser?
normally this is all handled in my application framework, but i disabled it while i tested.
<?php
include_once '../includes/db_connect.php';
fetch_evt_values($conn, 7475, 2, 16);
function fetch_evt_values($conn, $p_frm_id, $p_evt_id, $p_usr_id) {
$p_rec_id = 0;
$l_rslt_msg = '';
$l_result = array(
'data' => array(),
'msg' => '0000'
);
$sql = 'BEGIN PHPEVT.EV_MOD.FETCH_EVT_VALUES(';
//$sql .= ':c_load_id,';
$sql .= ':c_frm_id,';
$sql .= ':c_evt_id,';
$sql .= ':c_rec_id,';
$sql .= ':c_usr_id,';
$sql .= ':c_rslt';
$sql .= '); END;';
if ($stmt = oci_parse($conn,$sql)) {
$l_results = oci_new_cursor($conn);
//oci_bind_by_name($stmt,':c_load_id',$p_load_id);
oci_bind_by_name($stmt,':c_frm_id',$p_frm_id);
oci_bind_by_name($stmt,':c_evt_id',$p_evt_id);
oci_bind_by_name($stmt,':c_rec_id',$p_rec_id);
oci_bind_by_name($stmt,':c_usr_id',$p_usr_id);
oci_bind_by_name($stmt,':c_rslt',$l_results,-1,OCI_B_CURSOR);
if(oci_execute($stmt)){ //Execute the prepared query.
oci_execute($l_results);
while($r = oci_fetch_array($l_results,OCI_ASSOC)) {
$l_evt_values = explode('|', $r['EVENT_VALUES']);
foreach($l_evt_values as $l_evt_value) {
list($l_ID, $l_value) = explode('#', $l_evt_value);
$l_values[] = array('ID' => $l_ID, 'VALUE' => $l_value);
}
$l_result['data'][] = array(
'LOAD_ID' => $r['LOAD_ID'],
'REC_ID' => $r['REC_ID'],
'TRAIT' => $l_values,
'G_MSG' => $r['G_MSG']
);
$l_rslt_msg = $r['G_MSG'];
}
} else {
//echo 'cannot get user';
$l_rslt_msg = '0005'; //PHP_MEMBER.FETCH_USER return error code
}
} else {
//echo 'connect fail';
$l_rslt_msg = '0006'; //Could not connect to database.
}
oci_close($conn);
echo json_encode($l_result);
}
?>
So on a webpage, when a user requests an event, a database call is made using this code to retrieve some values in the format :
"62#20000|65#15710|66#6|67#6|68#0|69#0|".
The PHP then breaks it apart by |, splits the ID#Value, puts everything into an array, then returns it as a JSON which is then parsed into a table. The latter works perfectly fine. But when this tries to fetch more than about 600 records or so, I get a 500 Internal Server Error, and I've figured it's something in this PHP that's handling the call.
I'm not convinced it's the database entirely, as a call for 3500 records with no further processing other than the JSON being returned is generally done in 5s or less.
Why would this code be failing at 500+ records? I've tried AJAX timeout of 0.
Just getting into OOP, and changing from mysql queries to PDO. I am trying to create a class that will return the column names and meta data for a table. This is so I can output copy/paste data for all the tables I use. I have used such a tool, based on the mysql extension, for ages and it spits out variants such as complete SELECT/INSERT/UPDATE queries. Amongst other things I now want to add DECLARE listings for Stored Procedures - so getting meta data like type and length are essential. With about 150 tables across two schemas, such automation is essential.
With uncertainty about the reliability of getColumnMeta I hunted for code and found what looked good in a Sitepoint answer. I have attempted to wrap it in a class and mimic its original context but I am simply getting a number 1 when I try to echo or print_r the response. I have also had 'not an object' error messages while trying solutions.
This is the calling code
$db_host="localhost";
$db_username='root';
$db_pass='';
$db_name='mydatabase';
try{
$db= new PDO('mysql:host='.$db_host.';dbname='.$db_name,$db_username,$db_pass, array(PDO::ATTR_PERSISTENT=>false));
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_WARNING);
}
catch(PDOException $e){echo "Error: ".$e->getMessage()."<br />"; die(); }
include 'ColMetaData.php'; //the file containing the class for getting a column listing for each table
$coldat= new supplyColumnMeta($db);
$tablemet=$coldat->getColumnMeta('groups'); // a manual insertion of a table name for testing
echo $tablemet;
And this is the class that sits in the include file
class supplyColumnMeta{
public function __construct($db){
$this->db=$db;
}
/**
* Automatically get column metadata
*/
public function getColumnMeta($table)
{$this->tableName=$table;
// Clear any previous column/field info
$this->_fields = array();
$this->_fieldMeta = array();
$this->_primaryKey = NULL;
// Automatically retrieve column information if column info not specified
if(count($this->_fields) == 0 || count($this->_fieldMeta) == 0)
{
// Fetch all columns and store in $this->fields
$columns = $this->db->query("SHOW COLUMNS FROM " . $this->tableName, PDO::FETCH_ASSOC);
foreach($columns as $key => $col)
{
// Insert into fields array
$colname = $col['Field'];
$this->_fields[$colname] = $col;
if($col['Key'] == "PRI" && empty($this->_primaryKey)) {
$this->_primaryKey = $colname;
}
// Set field types
$colType = $this->parseColumnType($col['Type']);
$this->_fieldMeta[$colname] = $colType;
}
}
return true;
}
protected function parseColumnType($colType)
{
$colInfo = array();
$colParts = explode(" ", $colType);
if($fparen = strpos($colParts[0], "("))
{
$colInfo['type'] = substr($colParts[0], 0, $fparen);
$colInfo['pdoType'] = '';
$colInfo['length'] = str_replace(")", "", substr($colParts[0], $fparen+1));
$colInfo['attributes'] = isset($colParts[1]) ? $colParts[1] : NULL;
}
else
{
$colInfo['type'] = $colParts[0];
}
// PDO Bind types
$pdoType = '';
foreach($this->_pdoBindTypes as $pKey => $pType)
{
if(strpos(' '.strtolower($colInfo['type']).' ', $pKey)) {
$colInfo['pdoType'] = $pType;
break;
} else {
$colInfo['pdoType'] = PDO::PARAM_STR;
}
}
return $colInfo;
}
/**
* Will attempt to bind columns with datatypes based on parts of the column type name
* Any part of the name below will be picked up and converted unless otherwise sepcified
* Example: 'VARCHAR' columns have 'CHAR' in them, so 'char' => PDO::PARAM_STR will convert
* all columns of that type to be bound as PDO::PARAM_STR
* If there is no specification for a column type, column will be bound as PDO::PARAM_STR
*/
protected $_pdoBindTypes = array(
'char' => PDO::PARAM_STR,
'int' => PDO::PARAM_INT,
'bool' => PDO::PARAM_BOOL,
'date' => PDO::PARAM_STR,
'time' => PDO::PARAM_INT,
'text' => PDO::PARAM_STR,
'blob' => PDO::PARAM_LOB,
'binary' => PDO::PARAM_LOB
);
}
Here's the problem:
public function getColumnMeta($table)
{
$this->tableName=$table;
// Clear any previous column/field info
$this->_fields = array();
$this->_fieldMeta = array();
$this->_primaryKey = NULL;
// Automatically retrieve column information if column info not specified
if(count($this->_fields) == 0 || count($this->_fieldMeta) == 0)
{
// Fetch all columns and store in $this->fields
$columns = $this->db->query("SHOW COLUMNS FROM " . $this->tableName, PDO::FETCH_ASSOC);
foreach($columns as $key => $col)
{
// Insert into fields array
$colname = $col['Field'];
$this->_fields[$colname] = $col;
if($col['Key'] == "PRI" && empty($this->_primaryKey)) {
$this->_primaryKey = $colname;
}
// Set field types
$colType = $this->parseColumnType($col['Type']);
$this->_fieldMeta[$colname] = $colType;
}
}
return true;//<<--- not returning an object/array!
}
Your getColumnMeta method returns a boolean, true. The string representation of this value is, of course, 1. If you want this method to return all of the meta-data, change the return statement to something like:
return array(
'fields' => $this->_fields,
'meta' => $this->_fieldMeta,
'primary' => $this->_primaryKey
);
There are some other issues with your code, too, but seeing as this is not codereview.stackexchange, I'm not going to go into too much details. I will say this, though: Please, try to follow the coding standards that most major players adhere to: these standards can be found here: PHP-FIG.
Oh, and if you want to show the meta-data, don't echo them, but rather var_dump or print_r them, seeing as you return an array or an object.
Or at least echo json_encode($instance->getColumnMeta($table)); to get a correct string representation of the returned value(s).