PHP detect SQL injection attempt - php

My code is already safe, using parameters in SQL queries, but, I would like to detect if anyone attempts to inject something into a submit form.
I found Snort, but I would need something that would be at PHP script level, not the whole network.
This is for a site that contains personal information for students and thus, we will warn (or even take action against) anyone that even tries an attack.

I have created a very basic and simple PHP class for checking / detecting SQL injection attempts.
<?php
/**
* simpleSQLinjectionDetect Class
* #link https://github.com/bs4creations/simpleSQLinjectionDetect
* #version 1.1
*/
class simpleSQLinjectionDetect
{
protected $_method = array();
protected $_suspect = null;
public $_options = array(
'log' => true,
'unset' => true,
'exit' => true,
'errMsg' => 'Not allowed',
);
public function detect()
{
self::setMethod();
if(!empty($this->_method))
{
$result = self::parseQuery();
if ($result)
{
if ($this->_options['log']) {
self::logQuery();
}
if ($this->_options['unset']){
unset($_GET, $_POST);
}
if ($this->_options['exit']){
exit($this->_options['errMsg']);
}
}
}
}
private function setMethod()
{
if ($_SERVER['REQUEST_METHOD'] === 'GET') {
$this->_method = $_GET;
}
if ($_SERVER['REQUEST_METHOD'] === 'POST') {
$this->_method = $_POST;
}
}
private function parseQuery()
{
$operators = array(
'select * ',
'select ',
'union all ',
'union ',
' all ',
' where ',
' and 1 ',
' and ',
' or ',
' 1=1 ',
' 2=2 ',
' -- ',
);
foreach($this->_method as $key => $val)
{
$k = urldecode(strtolower($key));
$v = urldecode(strtolower($val));
foreach($operators as $operator)
{
if (preg_match("/".$operator."/i", $k)) {
$this->_suspect = "operator: '".$operator."', key: '".$k."'";
return true;
}
if (preg_match("/".$operator."/i", $v)) {
$this->_suspect = "operator: '".$operator."', val: '".$v."'";
return true;
}
}
}
}
private function logQuery()
{
$data = date('d-m-Y H:i:s') . ' - ';
$data .= $_SERVER['REMOTE_ADDR'] . ' - ';
$data .= 'Suspect: ['.$this->_suspect.'] ';
$data .= json_encode($_SERVER);
#file_put_contents('./logs/sql.injection.txt', $data . PHP_EOL, FILE_APPEND);
}
}
/* then call it in your app...
*********************************************/
$inj = new simpleSQLinjectionDetect();
$inj->detect();
You can check it on github also
This is a very simple and basic class. Any suggestions for improvements / updates are welcome :)

This is actually quite a hard topic. BillyK may have a semi-viable approach but it's better to let MySQL do the hard work for you; therefore:
1) Run the user-constructed (ie unsafe) query in a MySQL Transaction.
2) How many results does it give? (Check for both rows returned and rows affected)
3) Record any MySQL error warning logs.
4) Cancel / rollback the Transaction. So that nothing has changed on your database.
5) Re-run the query with the paramaterised variable (ie safe)
6) How many results does it give? (Check for both rows returned and rows affected)
7) Check if (6) gives a different number of results to (2) or if (5) gives any SQL error warnings. You can also use PHP array comparison features to check if the result sets are equal.
8) Any positives come up, such as differences in result counts, result set eqauality or SQL warnings, then record that query string into a save file for human review.
Concept Thoughts:
With a properly implemented system of Prepared Statements it is not possible for SQL injection to occur from user variables as data strings. Therefore, rather like people throwing water balloons at tanks; it's also pretty worthless to try and "detect" these infractions; they in themselves don't show you anything more than someone read some website that offers such methods.
Therefore, as long as you have built your PHP/SQL correctly then any number or any quality of SQL injecton attempts are just water off a ducks back, and you cumulatively waste more processing power and time and effort trying to detect and record them than you would simply to ignore them.

Related

How does PHP & MySQL handle simultaneous requests?

I must be missing something regarding how simultaneous requests are handled by PHP/Symfony, or perhaps how potentially simultaneous queries on the DB are handled...
This code seems to be doing the impossible - it randomly (about once a month) creates a duplicate of the new entity at the bottom. I conclude that it must happen when two clients make the same request twice, and both threads execute the SELECT query at the same time, picking up the entry where stop == NULL, and then they both (?) set the stoptime for that entry, and they both write a new entry.
Here's my logical outline as far as I understand things:
Get all entries with NULL for stoptime
Loop over those entries
Only proceed if the day of the entry (UTC) is different from current day (UTC)
Set stoptime for the open entry to 23:59:59 and flush to DB
Build a new entry with the starttime 00:00:00 on the next day
Assert that there are no other open entries in that position
Assert that there are no future entries in that position
Only then - flush the new entry to DB
Controller autocloseAndOpen
//if entry spans daybreak (midnight) close it and open a new entry at the beginning of next day
private function autocloseAndOpen($units) {
$now = new \DateTime("now", new \DateTimeZone("UTC"));
$repository = $this->em->getRepository('App\Entity\Poslog\Entry');
$query = $repository->createQueryBuilder('e')
->where('e.stop is NULL')
->getQuery();
$results = $query->getResult();
if (!isset($results[0])) {
return null; //there are no open entries at all
}
$em = $this->em;
$messages = "";
foreach ($results as $r) {
if ($r->getPosition()->getACRGroup() == $unit) { //only touch the user's own entries
$start = $r->getStart();
//Assert entry spanning datebreak
$startStr = $start->format("Y-m-d"); //Necessary for comparison, if $start->format("Y-m-d") is put in the comparison clause PHP will still compare the datetime object being formatted, not the output of the formatting.
$nowStr = $now->format("Y-m-d"); //Necessary for comparison, if $start->format("Y-m-d") is put in the comparison clause PHP will still compare the datetime object being formatted, not the output of the formatting.
if ($startStr < $nowStr) {
$stop = new \DateTimeImmutable($start->format("Y-m-d")."23:59:59", new \DateTimeZone("UTC"));
$r->setStop($stop);
$em->flush();
$txt = $unit->getName() . " had an entry in position (" . $r->getPosition()->getName() . ") spanning datebreak (UTC). Automatically closed at " . $stop->format("Y-m-d H:i:s") . "z.";
$messages .= "<p>" . $txt . "</p>";
//Open new entry
$newStartTime = $stop->modify('+1 second');
$entry = new Entry();
$entry->setStart( $newStartTime );
$entry->setOperator( $r->getOperator() );
$entry->setPosition( $r->getPosition() );
$entry->setStudent( $r->getStudent() );
$em->persist($entry);
//Assert that there are no future entries before autoopening a new entry
$futureE = $this->checkFutureEntries($r->getPosition(),true);
$openE = $this->checkOpenEntries($r->getPosition(), true);
if ($futureE !== 0 || $openE !== 0) {
$txt = "Tried to open a new entry for " . $r->getOperator()->getSignature() . " in the same position (" . $r->getPosition()->getName() . ") next day but there are conflicting entries.";
$messages .= "<p>" . $txt . "</p>";
} else {
$em->flush(); //store to DB
$txt = "A new entry was opened for " . $r->getOperator()->getSignature() . " in the same position (" . $r->getPosition()->getName() . ")";
$messages .= "<p>" . $txt . "</p>";
}
}
}
}
return $messages;
}
I'm even running an extra check here with the checkOpenEntries() to see whether there is at this point any entries with stoptime == NULL in that position. Initially, I thought that to be superflous because I thought that if one request is running and operating on the DB, the other request will not start until the first is finished.
private function checkOpenEntries($position,$checkRelatives = false) {
$positionsToCheck = array();
if ($checkRelatives == true) {
$positionsToCheck = $position->getRelatedPositions();
$positionsToCheck[] = $position;
} else {
$positionsToCheck = array($position);
}
//Get all open entries for position
$repository = $this->em->getRepository('App\Entity\Poslog\Entry');
$query = $repository->createQueryBuilder('e')
->where('e.stop is NULL and e.position IN (:positions)')
->setParameter('positions', $positionsToCheck)
->getQuery();
$results = $query->getResult();
if(!isset($results[0])) {
return 0; //tells caller that there are no open entries
} else {
if (count($results) === 1) {
return $results[0]; //if exactly one open entry, return that object to caller
} else {
$body = 'Found more than 1 open log entry for position ' . $position->getName() . ' in ' . $position->getACRGroup()->getName() . ' this should not be possible, there appears to be corrupt data in the database.';
$this->email($body);
$output['success'] = false;
$output['message'] = $body . ' An automatic email has been sent to ' . $this->globalParameters->get('poslog-email-to') . ' to notify of the problem, manual inspection is required.';
$output['logdata'] = null;
return $this->prepareResponse($output);
}
}
}
Do I need to start this function with some kind of "Lock database" method to achieve what I am trying to do?
I have tested all functions and when I simulate all kinds of states (entry with NULL for stoptime even when it shouldn't be able to be etc.) it all works out. And the majority of time it all works nicely, but that one day somewhere in the middle of the month, this thing happens...
You can never guarantee sequential order (or implicit exclusive access). Try that and you will dig yourself deeper and deeper.
As Matt and KIKO mentioned in the comments, you could use constraints and transactions and those should help immensely as your database will stay clean, but keep in mind your app needs to be able to catch errors produced by the database layer. Definitely worth trying first.
Another way of handling this is to enforce database/application level locking.
Database-level locking is more coarse and very unforgiving if you somewhere forget to release a lock (in long-running scripts).
MySQL docs:
If the connection for a client session terminates, whether normally or abnormally, the server implicitly releases all table locks held by the session (transactional and nontransactional). If the client reconnects, the locks are no longer in effect.
Locking the entire table is usually a bad idea altogether, but it is doable. This highly depends on the application.
Some ORMs, out-of-box, support object versioning and throw exceptions if the version has changed during the execution. In theory, your application would get an exception and upon retrying it would find that someone else already populated the field and that is no longer a candidate for an update.
Application-level locking is more fine-grained, but all points in the code need to honor the lock, otherwise, you are back to square #1. And if your app is distributed (say K8S, or just deployed across multiple servers), your locking mechanism has to be distributed as well (not local to an instance)

Is this way of handling POST and GET data a good practice?

After reading this article, it made me wonder if this is actually a good practice for registering new users. Everyone with some PHP studied can see how it works, but I just feel repeating myself if I have to handle all the post data manually. I know it's not 'difficult' nor too long to do at once, but I think it could be handled in a better way on the long run if you implemented it something similar to this code. For example, to add a single field more you have to change a lot of code, doing copy/paste in the article, but here it's only one field more in the array $ValidFields. What do you think?
function registerUser()
{
// Only place (apart of the mysql table, obviously) to add new fields of all the script.
$ValidFields = array ("name","lastname","email","password");
$tablename="users"; // If oop, this could be done in the __construct()
foreach ($_POST as $key => $value)
if(in_array($key,$ValidFields))
{
$key=mysql_real_escape_string($key);
if ($key=="password") $value=md5($value);
else $value=mysql_real_escape_string($value);
if (!$mysql) // If there is nothing inside
{
$mysql="INSERT INTO ".$tablename." (".$key;
$endmysql=") VALUES ('".$value."'";
}
else
{
$mysql.=", ".$key;
$endmysql.=", '".$value."'";
}
}
$mysql=$mysql.$endmysql.")";
return $mysql;
}
Tested adding this code after the function
$_POST['name']="testname";
$_POST['lastname']="testlastname";
$_POST['email']="teste'mail"; // Checking MySQL injection (;
$_POST['password']="testpassword";
$_POST['CakePHP']="is_a_lie"; // "Hello world" is too mainstream
echo registerUser();
And the returned string is, effectively:
INSERT INTO users (name, lastname, email, password) VALUES ('testname', 'testlastname', 'teste\'mail', 'testpassword')
NOTE! I know that I should not use mysql_, this is only an illustrative script. There are many statements in php5 (PDO, MYSQLi, etc) that everyone should use. I'm focusing on scalability and performance. A similar process could be reproduced for creating the HTML form. Also, it should work similar with a class.
I'm just wondering why, in so many years PHP has been developed and in the 1 year something I've been studying it and searching for information online, I haven't seen any similar and maybe more efficient way of handling POST or GET data.
I do not handle $_GET and $_POST at all. Instead I use parameter binding on my queries.
So my insert looks like this:
public function Insert( $table, array $bind )
{
$this->fetch = function( $sth, $obj ) { return $obj->lastID = $obj->lastInsertId(); };
$this->sql = array();
$this->bindings = array();
$columns = implode( ", ", array_keys( $bind ) );
$values = ':' . implode( ", :", array_keys( $bind ) );
foreach ( $bind as $column => $value )
$this->bindings[] = array( 'binding' => $column, 'value' => $value );
$this->sql['insert'] = "INSERT INTO " . $table . " (" . $columns . ") VALUES (" . $values . ")";
return $this;
}
And the execute looks like this:
public function Execute()
{
$sth = $this->prepare( implode( ' ', $this->sql ));
if( !$sth )
return false;
foreach ( $this->bindings as $bind )
if( $bind['binding'] ) {
if( isset( $bind['type'] ))
$sth->bindValue( ':' . $bind['binding'], $bind['value'], $bind['type'] );
else
$sth->bindValue( ':' . $bind['binding'], $bind['value'] );
}
if( $sth->execute() ) {
$lambda = $this->fetch;
return $lambda( $sth, $this );
}
else
return false;
}
This is absolutely not a good practice -- mysql_real_escape_string is ONLY safe for esaping data which is safely contained within single quotation marks. It cannot safely be used for any other purpose.
As a result, it is possible to inject malicious content into your query by setting a crafted POST key -- for instance, one could set
$_POST["name) SELECT CONCAT(name, CHAR(58), password) FROM users --"] = "";
to create one user in your database for every existing user, with a name indicating the password of the existing user. If your application displayed a list of users publicly, this would have the effect of making all users' passwords public. More nuanced attacks are also possible; this is just a simple example.

Error on specific search query

I have a website ongrounds.com there is a search bar on top when ever I search for word "best" it generates following error
Warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s) AS relevance FROM hotaru_posts WHERE (post_status = %s OR post_status = %s)' at line 1 in /home2/onground/public_html/libs/extensions/ezSQL/mysql/ez_sql_mysql.php on line 264
Notice: Trying to get property of non-object in /home2/onground/public_html/content/plugins/bookmarking/libs/BookmarkingFunctions.php on line 132
But when I search any other word except "best" the search plugin works fine and it shows results. I do not know why it is showing error on word "best". Please help.
SEARCH PLUGIN CODE:
class Search
{
/**
* Add permissions and register search widget
*/
public function install_plugin($h)
{
// Permissions
$site_perms = $h->getDefaultPermissions('all');
if (!isset($site_perms['can_search'])) {
$perms['options']['can_search'] = array('yes', 'no');
$perms['can_search']['default'] = 'yes';
$h->updateDefaultPermissions($perms);
}
// widget
$h->addWidget('search', 'search', ''); // plugin name, function name, optional arguments
}
/**
* Get search results
*/
public function theme_index_top($h)
{
// Get page title
if ($h->cage->get->keyExists('search')) {
$title = stripslashes(htmlentities($h->cage->get->sanitizeTags('search'),ENT_QUOTES,'UTF-8'));
$h->pageTitle = make_name($title);
$h->subPage = 'search';
$h->pageType = 'list';
$h->pageName = 'search';
}
}
/**
* Displays "Search!" wherever the plugin hook is.
*/
public function search_box($h)
{
$h->displayTemplate('search_box', 'search');
}
/**
* Displays "Search!" wherever the plugin hook is.
*/
public function widget_search($h)
{
$h->displayTemplate('search_box', 'search');
}
/**
* Use the search terms to build a filter
*/
public function bookmarking_functions_preparelist($h, $vars)
{
if ($h->cage->get->keyExists('search'))
{
$return = $vars['return']; // are we getting the count or the result set?
$orig_search_terms = stripslashes($h->cage->get->sanitizeTags('search'));
$search_terms = $orig_search_terms;
if ($search_terms)
{
// fetch select, orderby and filter...
$prepared_search = $this->prepareSearchFilter($h, $search_terms, $return);
extract($prepared_search);
$h->vars['orig_search'] = $orig_search_terms; // use this to re-fill the search box after a search
$h->vars['orig_search_terms'] = $orig_search_terms; // used in the breadcrumbs function
return true;
}
}
return false;
}
/**
* Prepare search filter
*/
public function prepareSearchFilter($h, $search, $return = 'posts')
{
$search_terms = strtolower($search);
$search_terms = explode(" ", $search_terms);
$search_terms = array_iunique($search_terms);
$search_terms_clean = '';
$full_text = true; // Do a full text (better) search if all terms are longer than 3 characters
foreach($search_terms as $search_term) {
if ($this->isStopword($search_term)) {
continue; // don't include this in $search_terms_clean
}
if (strlen(trim($search_term)) < 4) {
$full_text = false;
}
$search_term = trim($h->db->escape($search_term));
// if the urlencoded term contains a percent sign, we can't use a full text search
if (strpos(urlencode($search_term), '%') !== false) {
$full_text = false;
}
$search_terms_clean .= $search_term . " ";
}
// Undo the filter that limits results to either 'top', 'new' or archived (See submit.php -> sub_prepare_list())
if (isset($h->vars['filter']['post_status = %s'])) { unset($h->vars['filter']['post_status = %s']); }
if (isset($h->vars['filter']['post_archived = %s'])) { unset($h->vars['filter']['post_archived = %s']); }
// filter to top or new stories only:
$h->vars['filter']['(post_status = %s OR post_status = %s)'] = array('top', 'new');
$select = ($return == 'count') ? "count(*) AS number " : "*";
if ($full_text) {
$h->vars['select'] = array($select . ", MATCH(post_title, post_domain, post_url, post_content, post_tags) AGAINST (%s) AS relevance" => trim($search_terms_clean));
$h->vars['orderby'] = "relevance DESC";
$h->vars['filter']["MATCH (post_title, post_domain, post_url, post_content, post_tags) AGAINST (%s IN BOOLEAN MODE)"] = trim($search_terms_clean);
} else {
$h->vars['select'] = $select;
$h->vars['orderby'] = "post_date DESC";
$h->vars['filter_vars'] = array();
$where = $this->explodeSearch($h, 'post_title', $search_terms_clean) . " OR ";
$where .= $this->explodeSearch($h, 'post_url', $search_terms_clean) . " OR ";
$where .= $this->explodeSearch($h, 'post_content', $search_terms_clean);
$where = '(' . $where . ')';
$h->vars['filter'][$where] = $h->vars['filter_vars'];
}
$prepared_search = array('select' => $h->vars['select'], 'orderby' => $h->vars['orderby'], 'filter' => $h->vars['filter']);
return $prepared_search;
}
/** Explode search for short words
*
* #param string $column
* #param string $search_terms
* #return string (with " OR " stripped off the end)
*/
public function explodeSearch($h, $column, $search_terms)
{
$query = '';
foreach(explode(' ', trim($search_terms)) as $word){
if ($word) {
$query .= $column . " LIKE %s OR ";
$search_term = urlencode(" " . trim($h->db->escape($word)) . " ");
// escape all percent signs for use in LIKE query:
$search_term = str_replace('%', '\%', $search_term);
array_push($h->vars['filter_vars'], "%" . $search_term . "%");
}
}
return substr($query, 0, -4);
}
/**
* Is it a stopword?
*
*#return bool
*/
public function isStopword($word)
{
$word_array = array();
// list came from http://meta.wikimedia.org/wiki/MySQL_4.0.20_stop_word_list
$stopwordlist = "things ii iii a able about above according accordingly across actually after afterwards again against ain't all allow allows almost alone along already also although always am among amongst an and another any anybody anyhow anyone anything anyway anyways anywhere apart appear appreciate appropriate are aren't around as aside ask asking associated at available away awfully be became because become becomes becoming been before beforehand behind being believe below beside besides best better between beyond both brief but by c'mon c's came can can't cannot cant cause causes certain certainly changes clearly co com come comes concerning consequently consider considering contain containing contains corresponding could couldn't course currently definitely described despite did didn't different do does doesn't doing don't done down downwards during each edu eg eight either else elsewhere enough entirely especially et etc even ever every everybody everyone everything everywhere ex exactly example except far few fifth first five followed following follows for former formerly forth four from further furthermore get gets getting given gives go goes going gone got gotten greetings had hadn't happens hardly has hasn't have haven't having he he's help hence her here here's hereafter hereby herein hereupon hers herself hi him himself his hither hopefully how howbeit however i'd i'll i'm i've ie if ignored immediate in inasmuch inc indeed indicate indicated indicates inner insofar instead into inward is isn't it it'd it'll it's its itself just keep keeps kept know knows known last lately later latter latterly least less lest let let's like liked likely little look looking looks ltd mainly many may maybe me mean meanwhile merely might more moreover most mostly much must my myself name namely nd near nearly necessary need needs neither never nevertheless new next nine no nobody non none noone nor normally not nothing novel now nowhere obviously of off often oh ok okay old on once one ones only onto or other others otherwise ought our ours ourselves out outside over overall own part particular particularly per perhaps placed please plus possible presumably probably provides que quite qv rather rd re really reasonably regarding regardless regards relatively respectively right said same saw say saying says second secondly see seeing seem seemed seeming seems seen self selves sensible sent serious seriously seven several shall she should shouldn't since six so some somebody somehow someone something sometime sometimes somewhat somewhere soon sorry specified specify specifying still sub such sup sure t's take taken tell tends th than thank thanks thanx that that's thats the their theirs them themselves then thence there there's thereafter thereby therefore therein theres thereupon these they they'd they'll they're they've think third this thorough thoroughly those though three through throughout thru thus to together too took toward towards tried tries truly try trying twice two un under unfortunately unless unlikely until unto up upon us use used useful uses using usually value various very via viz vs want wants was wasn't way we we'd we'll we're we've welcome well went were weren't what what's whatever when whence whenever where where's whereafter whereas whereby wherein whereupon wherever whether which while whither who who's whoever whole whom whose why will willing wish with within without won't wonder would would wouldn't yes yet you you'd you'll you're you've your yours yourself yourselves zero";
$word_array = explode(' ', $stopwordlist);
if (array_search($word, $word_array) == true) {
return true;
} else {
return false;
}
}
/**
* Add RSS link to breadcrumbs
*/
public function breadcrumbs($h)
{
if ($h->subPage != 'search') { return false; }
$crumbs = "<a href='" . $h->url(array('search'=>urlencode($h->vars['orig_search_terms']))) . "'>\n";
$crumbs .= $h->vars['orig_search_terms'] . "</a>\n ";
return $crumbs . $h->rssBreadcrumbsLink('', array('search'=>urlencode($h->vars['orig_search_terms'])));
}
/**
* If a search feed, set it up
*/
public function post_rss_feed($h)
{
Thank you
It looks like this error will occur if the searched word ist a Fulltext-stopword.
I cant tell you why this results in an error, but maybe this knowledge leads you into successfull investigations.

Is it possible to call a PHP function from an SQL query?

Say there is a special PHP function foo($number) that returns double the value of its input. Further, say there is a table that contains the column "number." Is there a way to have the PHP function and SQL query to run together so that I would get the following:
Number | Double
================
1 | 2
5 | 10
While in this simple example column Double can easily be implemented within the SQL statement, answers should cover the more general case of calling any PHP function, as there are many operations that are difficult to impossible in SQL but easy in PHP.
No, since the query results come straight from MySQL. You can apply the function to the result set after you execute your query, either by using a loop or by applying your function to the results using array_map() function.
Depending on what you're actually trying to achieve it might be possible to decouple the data source and the consumer a bit, enough to put another component between them.
Let's start with
<?php
$result = getData($pdo); // a)
doSomething($result); // b)
function getData($mysql) {
return mysql_query('SELECT x FROM soTest', $mysql);
}
function doSomething($result) {
while ( false!==($row=mysql_fetch_assoc($result)) ) {
echo ' ', join(', ', $row), "\n";
}
echo "----\n";
}
There's very little you can do to alter a mysql result resource. And doSomething() does nothing more than iterating over the result set. It does nothing that is special to a mysql result set, yet it allows nothing else but this exact resource type by using mysql_fetch_xyz().
Let's try this again using PDO (PDO_MYSQL).
$result = getData($pdo);
doSomething($result);
function getData($pdo) {
return $pdo->query('SELECT x FROM soTest');
}
function doSomething(PDOStatement $result) {
while ( $row=$result->fetch(PDO::FETCH_ASSOC) ) {
echo ' ', join(', ', $row), "\n";
}
echo "----\n";
}
That didn't change much. Some names but essentially this is the same. But PDOStatement implements the Traversable interface, so you can use it directly with foreach.
$result = getData($pdo);
doSomething($result);
function getData($pdo) {
return $pdo->query('SELECT x FROM soTest', PDO::FETCH_ASSOC);
}
function doSomething($traversable) {
foreach( $traversable as $row ) {
echo ' ', join(', ', $row), "\n";
}
echo "----\n";
}
That's a game changer... We can pass any traversable/iterator to doSomething() and it still does more or less the same thing as before.
Now we can put something "between" getData() and doSomething(). This something takes an inner iterator (like getData() provides in the form of an PDOStatement) and behaves itself like an iterator (so DoSomething() can use it) returning all elements of its inner iterator but modifying some elements.
I chose to extend FilterIterator for this example for no particular reason. You need php 5.3+ to run this demo since it uses an anonymous function:
<?php
$pdo = initDemo();
echo "#1: passing \$result\n";
$result = getData($pdo); // a)
doSomething($result); // b)
echo "#2: passing ModifyIterator\n";
$result = getData($pdo); // exact same as a)
$result = new ModifyIterator($result, null, function($e) {
$e['y'] = '/' . ($e['x'] * 2) .'/';
return $e;
});
doSomething($result); // exact same as b)
function getData($pdo) {
return $pdo->query('SELECT x FROM soTest', PDO::FETCH_ASSOC);
}
function doSomething($traversable) {
foreach($traversable as $row) {
echo ' ', join(', ', $row), "\n";
}
echo "----\n";
}
class ModifyIterator extends FilterIterator {
protected $fnAccept, $fnModify;
public function __construct($it, $fnAccept, $fnModify) {
// todo: test parameters!
$this->fnAccept = $fnAccept;
$this->fnModify = $fnModify;
if ( !($it instanceof Iterator) ) {
$it = new IteratorIterator($it);
}
parent::__construct($it);
}
public function accept() {
return $this->fnAccept ? $this->fnAccept(parent::current()) : true;
}
public function current() {
if ( $this->fnModify ) {
$fn = $this->fnModify;
return $fn(parent::current());
}
return parent::current();
}
}
function initDemo() {
$pdo = new PDO('mysql:host=localhost;dbname=test', 'localonly', 'localonly');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->exec('CREATE TEMPORARY TABLE soTest (x int auto_increment, primary key(x))');
$pdo->exec('INSERT INTO soTest VALUES (),(),(),(),()');
return $pdo;
}
prints
#1: passing $result
1
2
3
4
5
----
#2: passing ModifyIterator
1, /2/
2, /4/
3, /6/
4, /8/
5, /10/
----
The important part is that ModifyIterator forces very little particular behaviour on the inner iterator (e.g. you can still use an unbuffered query without the need to transfer all the data into the php process' memory at once) and that both getData() and doSomething() are left unchanged.
One way would be to fetch the results into objects:
class NumberDoubler
{
public $number;
public function __construct()
{
$this->number *= 2;
}
}
$pdo = new PDO('mysql:host=localhost;dbname=db_name', 'uname', 'pword');
$stmnt = $pdo->query('SELECT number FROM db_table');
$result = $stmnt->fetchAll(PDO::FETCH_CLASS, 'NumberDoubler');
print_r($result);
The result will be an array of objects with '$number' doubled. Of course, iteration will still be done "behind the scenes", and the manual warns, "Using this method to fetch large result sets will result in a heavy demand on system and possibly network resources."
See also PDOStatement::setFetchMode().
You don't need to use PHP. You can just execute a regular SQL statement as follows.
SELECT number, number * 2 FROM tablename;
//Your query would prob be like so.
$query = mysql_query("SELECT (number * 2) as double,number FROM table");
echo '<table>';
while($row = mysql_fetch_assoc($query))
{
echo sprintf('<tr><td>%d</td><td>%d</td></tr>',$row['number'],$row['double']);
}
echo '</table>';
I think I understand your question.
It sounds as though you want to pull a number from a database and double it through a php function.
I would first learn to write the php function... Check this out for a tutorial.
Once you have that down, pull your number from the database. Here is a tutorial on how to do that.
This is all about learning. Me posting code for you to just copy doesn't help you learn and is basically a waste of my time. Best of luck.
It depends on what the function is doing. Basic stuff like arithmetic can be done with SQL directly. Otherwise, you can loop over the result array and run the function in the particular field, e.g.:
$rows = array();
foreach(($row = mysql_fetch_assoc($result))) {
$row['double'] = func($row['number']);
rows[] = $row;
}
No, it's impossible to use php function() and sql together
However, you can get results from SQL database and apply whatever PHP function on it

PHP PDO: how does re-preparing a statement affect performance

I'm writing a semi-simple database wrapper class and want to have a fetching method which would operate automagically: it should prepare each different statement only the first time around and just bind and execute the query on successive calls.
I guess the main question is: How does re-preparing the same MySql statement work, will PDO magically recognize the statement (so I don't have to) and cease the operation?
If not, I'm planning to achieve do this by generating a unique key for each different query and keep the prepared statements in a private array in the database object - under its unique key. I'm planning to obtain the array key in one of the following ways (none of which I like). In order of preference:
have the programmer pass an extra, always the same parameter when calling the method - something along the lines of basename(__FILE__, ".php") . __LINE__ (this method would work only if our method is called within a loop - which is the case most of the time this functionality is needed)
have the programmer pass a totally random string (most likely generated beforehand) as an extra parameter
use the passed query itself to generate the key - getting the hash of the query or something similar
achieve the same as the first bullet (above) by calling debug_backtrace
Has anyone similar experience? Although the system I'm working for does deserve some attention to optimization (it's quite large and growing by the week), perhaps I'm worrying about nothing and there is no performance benefit in doing what I'm doing?
MySQL (like most DBMS) will cache execution plans for prepared statements, so if user A creates a plan for:
SELECT * FROM some_table WHERE a_col=:v1 AND b_col=:v2
(where v1 and v2 are bind vars) then sends values to be interpolated by the DBMS, then user B sends the same query (but with different values for interpolation) the DBMS does not have to regenerate the plan. i.e. it's the DBMS which finds the matching plan - not PDO.
However this means that each operation on the database requires at least 2 round trips (1st to present the query, the second to present the bind vars) as opposed to a single round trip for a query with literal values, then this introduces additional network costs. There is also a small cost involved in dereferencing (and maintaining) the query/plan cache.
The key question is whether this cost is greater than the cost of generating the plan in the first place.
While (in my experience) there definitely seems to be a performance benefit using prepared statements with Oracle, I'm not convinced that the same is true for MySQL - however, a lot will depend on the structure of your database and the complexity of the query (or more specifically, how many different options the optimizer can find for resolving the query).
Try measuring it yourself (hint: you might want to set the slow query threshold to 0 and write some code to convert literal values back into anonymous representations for the queries written to the logs).
Believe me, I've done this before and after building a cache of prepared statements the performance gain was very noticeable - see this question: Preparing SQL Statements with PDO.
An this was the code I came up after, with cached prepared statements:
function DB($query)
{
static $db = null;
static $result = array();
if (is_null($db) === true)
{
$db = new PDO('sqlite:' . $query, null, null, array(PDO::ATTR_ERRMODE => PDO::ERRMODE_WARNING));
}
else if (is_a($db, 'PDO') === true)
{
$hash = md5($query);
if (empty($result[$hash]) === true)
{
$result[$hash] = $db->prepare($query);
}
if (is_a($result[$hash], 'PDOStatement') === true)
{
if ($result[$hash]->execute(array_slice(func_get_args(), 1)) === true)
{
if (stripos($query, 'INSERT') === 0)
{
return $db->lastInsertId();
}
else if (stripos($query, 'SELECT') === 0)
{
return $result[$hash]->fetchAll(PDO::FETCH_ASSOC);
}
else if ((stripos($query, 'UPDATE') === 0) || (stripos($query, 'DELETE') === 0))
{
return $result[$hash]->rowCount();
}
else if (stripos($query, 'REPLACE') === 0)
{
}
return true;
}
}
return false;
}
}
Since I don't need to worry about collisions in queries, I've ended up using md5() instead of sha1().
OK, since I've been bashing methods of keying the queries for the cache, other than simply using the query string itself, I've done a naive benchmark. The following compares using the plain query string vs first creating the md5 hash:
$ php -v
$ PHP 5.3.0-3 with Suhosin-Patch (cli) (built: Aug 26 2009 08:01:52)
$ ...
$ php benchmark.php
$ PHP hashing: 0.19465494155884 [microtime]
$ MD5 hashing: 0.57781004905701 [microtime]
$ 799994
The code:
<?php
error_reporting(E_ALL);
$queries = array("SELECT",
"INSERT",
"UPDATE",
"DELETE",
);
$query_length = 256;
$num_queries = 256;
$iter = 10000;
for ($i = 0; $i < $num_queries; $i++) {
$q = implode('',
array_map("chr",
array_map("rand",
array_fill(0, $query_length, ord("a")),
array_fill(0, $query_length, ord("z")))));
$queries[] = $q;
}
echo count($queries), "\n";
$cache = array();
$side_effect1 = 0;
$t = microtime(true);
for ($i = 0; $i < $iter; $i++) {
foreach ($queries as $q) {
if (!isset($cache[$q])) {
$cache[$q] = $q;
}
else {
$side_effect1++;
}
}
}
echo microtime(true) - $t, "\n";
$cache = array();
$side_effect2 = 0;
$t = microtime(true);
for ($i = 0; $i < $iter; $i++) {
foreach ($queries as $q) {
$md5 = md5($q);
if (!isset($cache[$md5])) {
$cache[$md5] = $q;
}
else {
$side_effect2++;
}
}
}
echo microtime(true) - $t, "\n";
echo $side_effect1 + $side_effect2, "\n";
To my knowledge PDO does not reuse already prepared statements as it does not analyse the query by itself so it does not know if it is the same query.
If you want to create a cache of prepared queries, the simplest way imho would be to md5-hash the query string and generate a lookup table.
OTOH: How many queries are you executing (per minute)? If less than a few hundred then you only complicate the code, the performance gain will be minor.
Using a MD5 hash as a key you could eventually get two queries that result in the same MD5 hash. The probability is not high, but it could happen. Don't do it. Lossful hashing algorithms like MD5 is just ment as a way to tell if two objects are different with high certainty, but are not a safe means of identifying something.

Categories