I face a problem with a mysql query. The query runs but sometimes it makes php exceed the maximum memory. I have like 80 or 90.000 rows of coordinates, with engine speeds and other things. I have to create KML files to display routes individually. Where the engine speed is not null, the car is moving, if it is, the car is stopped. Half of the table's engine speeds contains 0s. When I'm iterating through the records, I also delete the records at the same time, after I created the routes array, but it runs very slowly and sometimes it runs out of memory. Can it be because of the high and massive data amount in the database or some logical error in my code? Here is the code:
public function getPositions($device_id) {
$db = connect_database(DB_HOST, DB_USER, DB_PASSWORD, DB_NAME, DB_PORT);
$sql = "SELECT * FROM coordinates_log WHERE imei=:imei ORDER BY device_time ASC";
$statement = $db->prepare($sql);
$statement->execute(array(':imei' => $device_id));
$positions = array();
$delete_sql = "DELETE FROM coordinates_log WHERE id=:id";
$delete_statement = $db->prepare($delete_sql);
$counter = 0;
$flag = 0;
while ($row = $statement->fetch(PDO::FETCH_ASSOC)) {
//here I flag the last started route
if ($row['vehicle_engine_speed'] <= 0) {
$flag = $counter;
}
$positions[] = $row;
$counter++;
}
if (!empty($positions)) {
$last_key = count($positions)-1;
//here I check if the route is completed yet, or he is on his way
if ($positions[$last_key]['vehicle_engine_speed'] != 0) {
for($i = $flag; $i<=$last_key; $i++){
unset($positions[$i]);
}
}
foreach ($positions as $position) {
$delete_statement->execute(array(':id' => $position['id']));
}
return $positions;
} else {
return FALSE;
}
}
The PDO subsystem in PHP offers two kinds of queries: buffered and unbuffered. Buffered queries are what you get if you don't specifically request unbuffered queries. Buffered queries consume more RAM in your PHP engine because PDO fetches the entire result set into RAM, then gives it back to your program a line at a time when you use $statement->fetch().
So, if your result sets are quite large and you can process them a row at a time, you will use less RAM with unbuffered mode. You process each row, then fetch the next one, without trying to hold them all in RAM at once.
Here's a writeup on unbuffered mode.
http://php.net/manual/en/mysqlinfo.concepts.buffering.php
Buffered mode is generally easier to use for programmers, because PDO reads the entire resultset from each query and implicitly closes the statement object. That leaves your connection available for the next sql statement, even if you have not yet processed all the information in your resultset. With unbuffered mode, if you want to run other mysql statements while you're processing your result set, you need another database connection to do that.
You should try unbuffered mode for your SELECT * FROM coordinates... result set.
Pro tip: If you avoid SELECT * and instead use SELECT col, col, col you probably can reduce the overhead of your queries, especially if you don't actually need all the columns.
Questions of the "Look at my code and tell me what's wrong with it" kind are off topic here. It is not only because the code is intended to be run, by the computers, not read by the humans, but because code itself is seldom relevant to the problem.
Before asking a question here, you have to profile your code, determining slowest parts, and memory consumption as well.
I could make some guesses though I hate it.
it could be query itself, if not optimized
it could be buffering issue, making whole resultset burden script's memory.
it could be select * issue, burdening your resulting array with lots of junk data
it could be slow writes due to innodb settings.
But guesswork doesn't make a good answer. You have to work out your question first.
I want to make use of generators to handle over chunks of data from a database query while using the minimum ammount of memory at the same time (one of the advantage for generators).
The problem I've found while using generators is associated with this piece of code:
if (!empty($this->attributes["queries"])) {
if (!empty($query)) {
if (!empty($this->attributes["queries"][$query])) {
$this->attributes["queries"][$query]->output();
}
} else {
foreach ($this->attributes["queries"] as $query) {
$query->output();
}
}
}
Optimizations aside, what this code does is:
Checks if we have any allocated SQL query
Checks if a specific SQL query was given to execute (ignore any other)
In case a specific SQL query was given and it exists, execute it with output()
If no specific SQL query was given, traverse all SQL queries and execute them with output()
The problem is this works flawlessly for any kind of SQL query EXCEPT the most used one: SELECT.
The problem lies within this piece of code:
if (!empty($query)) {
if (!empty($this->attributes["queries"][$query])) {
if ($this->attributes["queries"][$query]->attributes["type"] === "SELECT") {
if ($output = $this->attributes["link"]->prepare($this->attributes["queries"][$query]->attributes["query"], [\PDO::ATTR_CURSOR => \PDO::CURSOR_FWDONLY])) {
if ($output->execute(!empty($this->attributes["queries"][$query]->attributes["parameters"]) ? $this->attributes["queries"][$query]->attributes["parameters"] : null)) {
while ($row = $output->fetch(\PDO::FETCH_ASSOC)) {
yield $row;
}
}
}
}
}
}
This method (called input to differentiate it from the output method with executes any other function) follows the same logic.
Checks if the specified SQL query exists
Prepares the SQL query specified as a query/parameters pair
Execute the SQL query and yield the results
I'm a bit lost here so I would like some light on this subject. Seems like yielding any data row makes any other execution branch unusable (I can perform as many SELECT queries as I want, but no transactional operation works (CREATE, INSERT, UPDATE, etc. just isn't issued or executed but not commited).
Can you give me (or help me design) a piece of pseudo code to reflect the correct pattern for this? Even if I code with PHP 5.5 a lot there's still things I don't know or haven't experienced, so a bit of help would be greatly appreciated :)
EDIT: what I mean with the input() and output() methods is my scheme currently works but ONLY if I separate the SELECT query into its own input() method (so using the generator doesn't break the other SQL queries) and leave the output() method for any other SQL operation.
Here's the code i am using withing a for loop which run 10 times:
$query = "UPDATE fblikes SET likes = '$var[$i]' WHERE link = '$web[$i]'";
if(mysql_query($query))
{
echo $query;
}
else
{
echo mysql_error();
}
The code runs, I do get ok! printed 10 times but nothing happens in the table. I also checked the 2 arrays i.e. $var and $web, they contain the correct values.
The query looks okay to me. Here's what i got (one of the 10 outputs) : UPDATE fblikes SET likes = '5' WHERE link = 'xxxxxxx.com/xxxx/iet.php';
I don't know what the problem exactly is, and to figure out you should print the value of $query, and show us what you get. More, please tell us the value of mysql_affected_rows() after the call to mysql_query().
However, your code implements some wrong patterns.
First of all, you are not escaping $var[$i] and $web[$i] with two potential effects:
You can produce wrong queries
You don't sanitize the input to the database, thus exposing your application to security issues
Moreover, you perform several similar queries that differ only on the inputs provided.
The solution, for both issues, is the use of prepared statements that will give you more control, security and performance. Consider abandoning mysql_* functions and switching to mysqli_* or PDO, and read about prepared statements.
I am trying to log the sql queries when a script is running. I am using zend framework and I already checked zend db profiler and this is not useful as this shows "?" for the values in a insert query..I need the actual values itself so that I can log it in a file. I use getAdapter()->update method for the update queries so I don' know if there is a way to get queries and log it. Please let me know if there is a way to log the queries.
regards
From http://framework.zend.com/manual/en/zend.db.profiler.html
The return value of getLastQueryProfile() and the individual elements of getQueryProfiles() are Zend_Db_Profiler_Query objects, which provide the ability to inspect the individual queries themselves:
getQuery() returns the SQL text of the query. The SQL text of a prepared statement with parameters is the text at the time the query was prepared, so it contains parameter placeholders, not the values used when the statement is executed.
getQueryParams() returns an array of parameter values used when executing a prepared query. This includes both bound parameters and arguments to the statement's execute() method. The keys of the array are the positional (1-based) or named (string) parameter indices.
When you use Zend_Db_Profiler_Firebug it will also show you the queries on the returned pages in the Firebug console along with any bound parameters.
I know you have got your answer though just for reference...
I have traversed hundred of pages, googled a lot but i have not found any exact solution.
Finally this worked for me. Irrespective where you are in either controller or model. This code worked for me every where. Just use this
//Before executing your query
$db = Zend_Db_Table_Abstract::getDefaultAdapter();
$db->getProfiler()->setEnabled(true);
$profiler = $db->getProfiler();
// Execute your any of database query here like select, update, insert
//The code below must be after query execution
$query = $profiler->getLastQueryProfile();
$params = $query->getQueryParams();
$querystr = $query->getQuery();
foreach ($params as $par) {
$querystr = preg_replace('/\\?/', "'" . $par . "'", $querystr, 1);
}
echo $querystr;
Finally this thing worked for me.
There are a few logs MySQL keeps itself.
Most notably:
The binary log (all queries)
Slow query log (queries that take longer than x time to execute)
See: http://dev.mysql.com/doc/refman/5.0/en/server-logs.html
Nowadays, "Prepared statements" seem to be the only way anyone recommends sending queries to a database. I even see recommendations to use prepared statements for stored procs. However, do to the extra query prepared statements require - and the short time they last - I'm persuaded that they are only useful for a line of INSERT/UPDATE queries.
I'm hoping someone can correct me on this, but it just seems like a repeat of the whole "Tables are evil" CSS thing. Tables are only evil if used for layouts - not tabular data. Using DIV's for tabular data is a style violation of WC3.
Like wise, plain SQL (or that generated from AR's) seems to be much more useful for 80% of the queries used, which on most sites are a single SELECT not to be repeated again that page load (I'm speaking about scripting languages like PHP here). Why would I make my over-taxed DB prepare a statement that it is only to run once before being removed?
MySQL:
A prepared statement is specific to
the session in which it was created.
If you terminate a session without
deallocating a previously prepared
statement, the server deallocates it
automatically.
So at the end of your script PHP will auto-close the connection and you will lose the prepared statement only to have your script re-created it on the next load.
Am I missing something or is this just a way to decrease performance?
:UPDATE:
It dawned on me that I am assuming new connections for each script. I would assume that if a persistent connection is used then these problems would disappear. Is this correct?
:UPDATE2:
It seems that even if persistent connections are the solution - they are not a very good option for most of the web - especially if you use transactions. So I'm back to square one having nothing more than the benchmarks below to go on...
:UPDATE3:
Most people simply repeat the phrase "prepared statements protect against SQL injection" which doesn't full explain the problem. The provided "escape" method for each DB library also protects against SQL injection. But it is more than that:
When sending a query the normal way,
the client (script) converts the data
into strings that are then passed to
the DB server. The DB server then uses
CPU power to convert them back into
the proper binary datatype. The
database engine then parses the
statement and looks for syntax errors.
When using prepared statements... the
data are sent in a native binary form,
which saves the conversion-CPU-usage,
and makes the data transfer more
efficient. Obviously, this will also
reduce bandwidth usage if the client
is not co-located with the DB server.
...The variable types are predefined,
and hence MySQL take into account
these characters, and they do not need
to be escaped.
http://www.webdesignforums.net/showthread.php?t=18762
Thanks to OIS for finally setting me strait on this issue.
unlike the CSS tables debate, there are clear security implications with prepared statements.
if you use prepared statements as the ONLY way to put user-supplied data in to a query, then they are absolutely bullet-proof when it comes to SQL injection.
When you execute a sql statement on the database, the sql parser needs to analyse it beforehand, which is the exact same process as the preparation.
So, comparing executing sql statements directly to preparing and executing has no disadvantages, but some advantages:
First of all, as longneck already stated, passing user input into a prepared statement escapes the input automatically. It is as if the database has prepared filters for the values and lets in only those values that fit.
Secondly, if use prepared statements thoroughly, and you come in the situation where you need to execute it multiple times, you don't need to rewrite the code to prepare and execute, but you just execute it.
Thirdly: The code becomes more readable, if done properly:
$sql = 'SELECT u.id, u.user, u.email, sum(r.points)
FROM users u
LEFT JOIN reputation r on (u.id=r.user_id)
LEFT JOIN badge b on (u.id=b.user_id and badge=:badge)
WHERE group=:group';
$params = array(
':group' => $group,
':badge' => $_GET['badge']
);
$stmt = $pdo->prepare($sql);
$result = $stmt->execute($params);
Instead of
$sql = 'SELECT u.id, u.user, u.email, sum(r.points)
FROM users u
LEFT JOIN reputation r on (u.id=r.user_id)
LEFT JOIN badge b on (u.id=b.user_id and badge="'.mysql_real_escape_string($_GET['badge']).'")
WHERE group="'.mysql_real_escape_string($group).'"';
$result = mysql_query($sql);
Imagine you had to change the sql statement, which code would be your favourite? ;-)
Prepared Statements come in handy in several situations:
Great separation of query data from untrusted user data.
Performance increase when the same query is executed multiple times
Performance increase when binary data is being transmitted as the prepared statement can use the binary protocol, whereas a traditional query will end up doing encoding and such.
There is a performance hit under normal circumstances (not repeated, no binary data) as you now have to do two back and forths. The first to "prepare" the query, and the second to transmit the token along with the data to be inserted. Most people are willing to make this sacrifice for the security benefit.
With regards to persistent connections:
MySQL has one of the fastest connection build up times on the market. It's essentially free for most set ups, so you're not going to see too much of a change using persistent connections or not.
The answer has to do with security and abstraction. Everyone else has already mentioned security, but the real upside is that your input is completely abstracted from the query itself. This allows for a true database agnosticism when using an abstraction layer, whereas inlining the input is usually a database-dependent process. If you care anything for portability, prepared statements are the way to go.
In the real world, I rarely ever write DML queries. All of my INSERTS / UPDATES are automatically built by the abstraction layer and are executed by simply passing an input array. For all intents and purposes, there really is no "performance hit" for preparing queries and then executing them (save for connection latency in the initial PREPARE). But when using a UDS (Unix Domain Socket) connection, you're not going to notice (or even be able to benchmark) a difference. It's usually on the order of a few microseconds.
Given the security and abstraction upsides, I'd hardly call it wasteful.
The performance benefit doesn't come from less parsing - it comes from only having to calculate access paths once rather than repeatedly. This helps a lot when you're issuing thousands of queries.
Given mysql's very simple optimizer/planner this may be less of an issue than with a more mature database with much more sophisticated optimizers.
However, this performance benefit can actually turn into a detriment if you've got a sophisticated optimizer that is aware of data skews. In that case you can often be better off with getting a different access path for the same query using different literal values rather than reusing a preexisting path.
When using sql queries like SELECT x,y,z FROM foo WHERE c='mary had a little lamb' the server has to parse the sql statement including the data + you have to sanitize the "mary had..." part (a call to mysql_real_escape() or similar for each parameter).
Using prepared statements the server has to parse the statement, too, but without the the data and sends back only an identifier for the statement (a tiny tiny data packet). Then you send the actual data without first sanitizing it. I don't see the overhead here, though I freely admit I've never tested it. Have you? ;-)
edit: And using prepared statements can eliminate the need to convert each and every parameter (in/out) to strings. Probably even more so if your version of php uses mysqlnd (instead of the "old" libmysql client library). Haven't tested the performance aspect of that either.
I don't seem to be finding any good benefits to use persistent connections - or prepared statements for that mater. Look at these numbers - for 6000 select statements (which will never happen in a page request!) you can barely tell the difference. Most of my pages use less than 10 queries.
UPDATED I just revised my test to
include 4k SELECT and 4k INSERT
statements! Run it yourself and let me
know if there are any design errors.
Perhaps the difference would be greater if my MySQL server wasn't running on the same machine as Apache.
Persistent: TRUE
Prepare: TRUE
2.3399310112 seconds
Persistent: FALSE
Prepare: TRUE
2.3265211582184 seconds
Persistent: TRUE
Prepare: FALSE
2.3666892051697 seconds
Persistent: FALSE
Prepare: FALSE
2.3496441841125 seconds
Here is my test code:
$hostname = 'localhost';
$username = 'root';
$password = '';
$dbname = 'db_name';
$persistent = FALSE;
$prepare = FALSE;
try
{
// Force PDO to use exceptions for all errors
$attrs = array(PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION);
if($persistent)
{
// Make the connection persistent
$attrs[PDO::ATTR_PERSISTENT] = TRUE;
}
$db = new PDO("mysql:host=$hostname;dbname=$dbname", $username, $password, $attrs);
// What type of connection?
print 'Persistent: '.($db->getAttribute(PDO::ATTR_PERSISTENT) ? 'TRUE' : 'FALSE').'<br />';
print 'Prepare: '.($prepare ? 'TRUE' : 'FALSE').'<br />';
//Clean table from last run
$db->exec('TRUNCATE TABLE `pdo_insert`');
}
catch(PDOException $e)
{
echo $e->getMessage();
}
$start = microtime(TRUE);
$name = 'Jack';
$body = 'This is the text "body"';
if( $prepare ) {
// Select
$select = $db->prepare('SELECT * FROM pdo_insert WHERE id = :id');
$select->bindParam(':id', $x);
// Insert
$insert = $db->prepare('INSERT INTO pdo_insert (`name`, `body`, `author_id`)
VALUES (:name, :body, :author_id)');
$insert->bindParam(':name', $name);
$insert->bindParam(':body', $body);
$insert->bindParam(':author_id', $x);
$run = 0;
for($x=0;$x<4000;++$x)
{
if( $insert->execute() && $select->execute() )
{
$run++;
}
}
}
else
{
$run = 0;
for($x=0;$x<4000;++$x) {
// Insert
if( $db->query('INSERT INTO pdo_insert (`name`, `body`, `author_id`)
VALUES ('.$db->quote($name).', '. $db->quote($body).', '. $db->quote($x).')')
AND
// Select
$db->query('SELECT * FROM pdo_insert WHERE id = '. $db->quote($x)) )
{
$run++;
}
}
}
print (microtime(true) - $start).' seconds and '.($run * 2).' queries';
Cassy is right. If you don't prepare/compile it, the dbms would have to in any case before able to run it.
Also, the advantage is you could check the prepare result and if prepare fail your algo can branch off to treat an exception without wasting db resources to run the failing query.