I want to make use of generators to handle over chunks of data from a database query while using the minimum ammount of memory at the same time (one of the advantage for generators).
The problem I've found while using generators is associated with this piece of code:
if (!empty($this->attributes["queries"])) {
if (!empty($query)) {
if (!empty($this->attributes["queries"][$query])) {
$this->attributes["queries"][$query]->output();
}
} else {
foreach ($this->attributes["queries"] as $query) {
$query->output();
}
}
}
Optimizations aside, what this code does is:
Checks if we have any allocated SQL query
Checks if a specific SQL query was given to execute (ignore any other)
In case a specific SQL query was given and it exists, execute it with output()
If no specific SQL query was given, traverse all SQL queries and execute them with output()
The problem is this works flawlessly for any kind of SQL query EXCEPT the most used one: SELECT.
The problem lies within this piece of code:
if (!empty($query)) {
if (!empty($this->attributes["queries"][$query])) {
if ($this->attributes["queries"][$query]->attributes["type"] === "SELECT") {
if ($output = $this->attributes["link"]->prepare($this->attributes["queries"][$query]->attributes["query"], [\PDO::ATTR_CURSOR => \PDO::CURSOR_FWDONLY])) {
if ($output->execute(!empty($this->attributes["queries"][$query]->attributes["parameters"]) ? $this->attributes["queries"][$query]->attributes["parameters"] : null)) {
while ($row = $output->fetch(\PDO::FETCH_ASSOC)) {
yield $row;
}
}
}
}
}
}
This method (called input to differentiate it from the output method with executes any other function) follows the same logic.
Checks if the specified SQL query exists
Prepares the SQL query specified as a query/parameters pair
Execute the SQL query and yield the results
I'm a bit lost here so I would like some light on this subject. Seems like yielding any data row makes any other execution branch unusable (I can perform as many SELECT queries as I want, but no transactional operation works (CREATE, INSERT, UPDATE, etc. just isn't issued or executed but not commited).
Can you give me (or help me design) a piece of pseudo code to reflect the correct pattern for this? Even if I code with PHP 5.5 a lot there's still things I don't know or haven't experienced, so a bit of help would be greatly appreciated :)
EDIT: what I mean with the input() and output() methods is my scheme currently works but ONLY if I separate the SELECT query into its own input() method (so using the generator doesn't break the other SQL queries) and leave the output() method for any other SQL operation.
Related
So honestly, this is the first time I am working with PDO and Error Exceptions. I have gone thru manuals as well as different Q/As resolved here in past and came up with code that I am pretty satisfied with. But I really really need your opinion on this. I have built few functions that I normally use often in my projects.
Please note that right now I am doing a test project just intended to learn these 2 new things.
First of all, I am not a fan of OOP and have always preferred procedural programming type.
function _database_row($_table, $_id = 0, $_key = "id") {
global $Database;
if(is_object($Database)) {
$Query = $Database->prepare("SELECT * FROM {$_table} WHERE {$_key} = :id");
if($Query->execute(Array(":id" => $_id))) {
$Query_Data = $Query->fetchAll(PDO::FETCH_ASSOC);
if(count($Query_Data) >= 1) {
if(count($Query_Data) == 1) {
return $Query_Data[0];
}
return $Query_Data;
}
} else {
throw new Exception("Database Query Failure: ".$Query->errorInfo()[2]);
}
}
return false;
}
the above function is intended to fetch a row from $_table table with $_id (not necessarily an integer value).
Please note that $_id may (sometimes) be the only thing (for this function) that is fetched from $_REQUEST. By simply preparing the statement, am I totally secure from any SQL injection threat?
I couldn't find an alternative to mysql_num_rows() (I indeed found few PDO methods to use fetchColumn while using COUNT() in the query. But I didn't prefer it that way, so I want to know if I did it right?
also before people ask, let me explain that in above function I designed it so it returns the row directly whenever I am looking for a single one (it will always be the case when I use "id" as $_key because its PRIMARY auto_increment in database), while in rare cases I will also need multiple results :) this seems to be working fine, just need your opinions.
example of use:
_database_row("user", 14); // fetch me the user having id # 14
_database_row("products", 2); // fetch me the user having id # 14
_database_row("products", "enabled", "status"); // fetch me all products with status enabled
...sometimes during procedural programming, I wouldn't like those nasty "uncaught exception" errors, instead I will simply prefer a bool(false). So I did it like this:
function __database_row($_table, $_id = 0, $_key = "id") {
try {
return _database_row($_table, $_id, $_key);
} catch (Exception $e) {
return false;
}
}
(don't miss the use of another leading "_"). This also seems to be working perfectly fine, so what's your opinion here?
IMPORTANT:
what is the use of "PDOStatement::closeCursor" exactly? I did read the manual but I am quite confused as I can call my functions as many times as I want and still get the desired/expected results but never "closed the cursor"
now... ENOUGH WITH SELECTS AND FETCHINGS :) lets talk about INSERTS
so I made this function to add multiple products in a single script execution and quickly.
function _add_product($_name, $_price = 0) {
global $Database;
if(is_object($Database)) {
$Query = $Database->prepare("INSERT INTO products (name, price) VALUES (:name, :price)");
$Query->execute(Array(":name" => $_name, ":price" => $_price));
if($Query->rowCount() >= 1) {
return $Database->lastInsertId();
} else {
throw new Exception("Database Query Failure: ".$Query->errorInfo()[2]);
}
}
return false;
}
This also seems to be working perfectly fine, but can I really rely on the method I used to get the ID of latest insert?
Thank you all!
There is a lot going on here, so I will try to answer specific questions and address some issues.
I am not a fan of OOP
Note that just because code has objects doesn't mean that it is object oriented. You can use PDO in a purely procedural style, and the presence of -> does not make it OOP. I wouldn't be scared of using PDO for this reason. If it makes you feel any better, you could use the procedural style mysqli instead, but I personally prefer PDO.
By simply preparing the statement, am I totally secure from any SQL injection threat?
No.
Consider $pdo->prepare("SELECT * FROM t1 WHERE col1 = $_POST[rightFromUser]"). This is a prepared statement, but it is still vulnerable to injection. Injection vulnerability has more to do with the queries themselves. If the statement is properly parameterized (e.g. you were using ? instead of $_POST), you would know longer be vulnerable. Your query:
SELECT * FROM {$_table} WHERE {$_key} = :id
actually is vulnerable because it has variables in it that can be injected. Although the query is vulnerable, it doesn't necessarily mean that the code is. Perhaps you have a whitelist on the table and column names and they are checked before the function is called. However the query is not portable by itself. I would suggest avoiding variables in queries at all -- even for table/column names. It's just a suggestion, though.
I couldn't find an alternative to mysql_num_rows()
There isn't one. Looking at the count of fetched results, using SELECT COUNT or looking at the table stats (for some engines) are surefire way to get the column count for SELECT statements. Note that PDOStatement::rowCount does work for SELECT with MySQL. However, it is not guaranteed to work with any database in particular according to the documentation. I will say that I've never had a problem using it to get the selected row count with MySQL.
There are similar comments regarding PDO::lastInsertId. I've never had a problem with that and MySQL either.
let me explain that in above function I designed it so it returns the row directly whenever I am looking for a single one
I would advise against this because you have to know about this functionality when using the function. It can be convenient at times, but I think it would be easier to handle the result of the function transparently. That is to say, you should not have to inspect the return value to discover its type and figure out how to handle it.
I wouldn't like those nasty "uncaught exception" errors
Exception swallowing is bad. You should allow exceptions to propagate and appropriately handle them.
Generally exceptions should not occur unless something catastrophic happens (MySQL error, unable to connect to the database, etc.) These errors should be very rare in production unless something legitimately happens to the server. You can display an error page to users, but at least make sure the exceptions are logged. During development, you probably want the exceptions to be as loud as possible so you can figure out exactly what to debug.
I also think that names should be reasonably descriptive, so two functions named __database_row and _database_row are really confusing.
IMPORTANT: what is the use of "PDOStatement::closeCursor" exactly?
I doubt you will have to use this, so don't worry too much about it. Essentially it allows you to fetch from separate prepared statements in parallel. For example:
$stmt1 = $pdo->prepare($query1);
$stmt2 = $pdo->prepare($query2);
$stmt1->execute();
$stmt1->fetch();
// You may need to do this before $stmt2->execute()
$stmt1->closeCursor();
$stmt2->fetch();
I could be wrong, but I don't think you need to do this for MySQL (i.e. you could call execute on both statements without calling closeCursor.
can I really rely on the method I used to get the ID of latest insert?
PDO's documentation on this (above) seems to be more forgiving about it than it is about rowCount and SELECT. I would use it with confidence for MySQL, but you can always just SELECT LAST_INSERT_ID().
Am I doing it right?
This is a difficult question to answer because there are so many possible definitions of right. Apparently your code is working and you are using PDO, so in a way you are. I do have some criticisms:
global $Database;
This creates a reliance on the declaration of a global $Database variable earlier in the script. Instead you should pass the database as an argument to the function. If you are an OOP fan, you could also make the database a property of the class that had this function as a method. In general you should avoid global state since it makes code harder to reuse and harder to test.
the above function is intended to fetch a row from $_table table with $_id
Rather than create a generic function for querying like this, it is better to design your application in a way that will allow you to run queries that serve specific purposes. I don't really see why you would want to select all columns for a table for a given ID. This is not as useful as it seems. Instead, you probably want to get specific columns from these tables, perhaps joined with other tables, to serve specific functions.
Well, your main problem is not OOP, but SQL.
To tell you truth, SQL is by no means a silly key-value storage you are taking it for. So, it makes your first function, that can be used only on too limited SQL subset, is totally useless.
Moreover, you are making a gibberish out of almost natural English of SQL. Compare these 2 sentences
SELECT * FROM products WHERE status = ?
quite comprehensible - isn't it?
products! enabled! status!
HUH?
Not to mention that this function is prone to SQL injection. So, you just have to get rid of it. If you want a one-liner, you can make something like this
function db_row($sql, $data = array(), $mode = PDO::FETCH_ASSOC) {
global $Database;
$stmt = $Database->prepare($sql);
$stmt->execute($data);
return $stmt->fetch($mode);
}
to be called this way:
$row = db_row("SELECT * FROM products WHERE id = ?",[14]);
Note that PDO is intelligent enough to report you errors without any intervention. All you need is set it into exception mode.
Speaking of second function, can be reviewed.
function _add_product($_name, $_price = 0)
{
global $Database;
$sql = "INSERT INTO products (name, price) VALUES (?,?)";
$Database->prepare($sql)->execute(func_get_args());
return $Database->lastInsertId();
}
is all you actually need.
I couldn't find an alternative to mysql_num_rows()
You actually never needed it with mysql and would never need with PDO either
I'm running a loop to collect the data from an XML file, after the 2000+ lines are done there is some validation rules that run afterwards. If it is safe I'd like to store the MySQL data, in the before mentioned loops I am wanting to store all the MySQL queries to an array (that being almost 2000 queries) and run them in a loop after validation rules have finished.
Is this a bad idea? If so, what would be the best idea behind storing the query and running it later?
To give you an idea of what I'm talking about if you didn't understand:
foreach($xml->object as $control) {
// Stores relative xml data here
}
if(1=1) // Validation rules run
for(...) { // For Loop for my queries (2000 odd loop)
Mysql_Query(..)
}
To insert a lot of date into a MySQL database I would start a transaction (http://php.net/manual/en/mysqli.begin-transaction.php), and then create a prepared statement and just loop through all the items and validate them straight away and execute the prepared statement. Afterwards just mark the transaction as successful and end it. This is the fastest approach.
The use of prepared statements also prevents SQL Injection.
What you could do is use a prepared statement for a library that supports it (PDO or mysqli). In PDO:
$stmt = $pdo->prepare("INSERT INTO t1 VALUES (:f1, :f2, :f3)");
$stmt->bindParam(":f1", $f1);
$stmt->bindParam(":f2", $f2);
$stmt->bindParam(":f3", $f3);
foreach ($xml->obj as $control) {
// create $array
}
if ($valid) {
foreach ($array as $control) {
$f1 = $control['f1'];
$f2 = $control['f2'];
$f3 = $control['f3'];
$stmt->execute();
}
}
There's nothing wrong with this approach, except for the fact that it uses more RAM than would be used by progressively parsing and posting the XML data records to the database. If the server running the php belongs to you that's no problem. If you're on one of those cheap USD5 per month shared hosting plans, it's possible you'll run into memory or execution time limits.
You mentioned "safety." Are you concerned that you will have to back out all the database updates if any one of them fails? If that's the case, do two things:
Don't use the MyISAM access method for your table.
Use a transaction, and when you've posted all your data base changes, commit it.
This transactional approach is good, because if your posting process fails for any reason, you can roll back the transaction and get back to your starting point.
I wrote a database migration PHP script, that loops a list of SQL files and executes their content. It should help to automate the project setupd and updates. Now I'm getting errors like this:
mapCoursesToSport.sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'DELIMITER |
DROP PROCEDURE IF EXISTS mapCoursesToSport|
CREATE PROCEDURE mapCo' at line 1
I was getting the same error when I was passing to the script a file with a view definition, that also was using DELIMITERs. Then I found the workaround just to remove the delimiters from the SQL file. And it worked. But now it's not an option, since stored procedures really need delimiter definitions.
So, how can I define MySQL stored procedures from PHP? (Or maybe morre generally: How should this SDELIMITER be handeled?)
You can use native PHP function mysqli::query and mysqli::prepare alternatively to create stored procedure:
http://php.net/manual/en/mysqli.quickstart.stored-procedures.php
Using mysqli::multi_query also works but is a bit more tricky to handle since you might need to count the number of query to execute upfront before executing them one by one (provided the final query delimiter is optional, counting the queries can be tedious)
http://php.net/manual/en/mysqli.multi-query.php
<?php
// Using mysqli extension below in object-oriented mode
// after having executing queries
// with mysqli::multi_query
do {
$queryResult = $mysqli->use_result();
unset($results);
while ($result = $queryResult->fetch_array(MYSQLI_ASSOC)) {
$results[] = $result;
}
$queryResult->close();
if ($mysqli->more_results()) {
$mysqli->next_result();
}
$queryCount--;
} while ($queryCount > 0);
P.S.: The reason I'm not using mysqli:more_results nor mysqli:next_result as loop statement is that some query might be properly executed without returning any result. In such case, we don't want to break the loop before all queries have been executed.
Here's the code i am using withing a for loop which run 10 times:
$query = "UPDATE fblikes SET likes = '$var[$i]' WHERE link = '$web[$i]'";
if(mysql_query($query))
{
echo $query;
}
else
{
echo mysql_error();
}
The code runs, I do get ok! printed 10 times but nothing happens in the table. I also checked the 2 arrays i.e. $var and $web, they contain the correct values.
The query looks okay to me. Here's what i got (one of the 10 outputs) : UPDATE fblikes SET likes = '5' WHERE link = 'xxxxxxx.com/xxxx/iet.php';
I don't know what the problem exactly is, and to figure out you should print the value of $query, and show us what you get. More, please tell us the value of mysql_affected_rows() after the call to mysql_query().
However, your code implements some wrong patterns.
First of all, you are not escaping $var[$i] and $web[$i] with two potential effects:
You can produce wrong queries
You don't sanitize the input to the database, thus exposing your application to security issues
Moreover, you perform several similar queries that differ only on the inputs provided.
The solution, for both issues, is the use of prepared statements that will give you more control, security and performance. Consider abandoning mysql_* functions and switching to mysqli_* or PDO, and read about prepared statements.
I am trying to log the sql queries when a script is running. I am using zend framework and I already checked zend db profiler and this is not useful as this shows "?" for the values in a insert query..I need the actual values itself so that I can log it in a file. I use getAdapter()->update method for the update queries so I don' know if there is a way to get queries and log it. Please let me know if there is a way to log the queries.
regards
From http://framework.zend.com/manual/en/zend.db.profiler.html
The return value of getLastQueryProfile() and the individual elements of getQueryProfiles() are Zend_Db_Profiler_Query objects, which provide the ability to inspect the individual queries themselves:
getQuery() returns the SQL text of the query. The SQL text of a prepared statement with parameters is the text at the time the query was prepared, so it contains parameter placeholders, not the values used when the statement is executed.
getQueryParams() returns an array of parameter values used when executing a prepared query. This includes both bound parameters and arguments to the statement's execute() method. The keys of the array are the positional (1-based) or named (string) parameter indices.
When you use Zend_Db_Profiler_Firebug it will also show you the queries on the returned pages in the Firebug console along with any bound parameters.
I know you have got your answer though just for reference...
I have traversed hundred of pages, googled a lot but i have not found any exact solution.
Finally this worked for me. Irrespective where you are in either controller or model. This code worked for me every where. Just use this
//Before executing your query
$db = Zend_Db_Table_Abstract::getDefaultAdapter();
$db->getProfiler()->setEnabled(true);
$profiler = $db->getProfiler();
// Execute your any of database query here like select, update, insert
//The code below must be after query execution
$query = $profiler->getLastQueryProfile();
$params = $query->getQueryParams();
$querystr = $query->getQuery();
foreach ($params as $par) {
$querystr = preg_replace('/\\?/', "'" . $par . "'", $querystr, 1);
}
echo $querystr;
Finally this thing worked for me.
There are a few logs MySQL keeps itself.
Most notably:
The binary log (all queries)
Slow query log (queries that take longer than x time to execute)
See: http://dev.mysql.com/doc/refman/5.0/en/server-logs.html