I have a time dependent script I am working on and used microtime() to find the bottle neck. I determined the time increase is caused by doing a check on 300+ values to see if they exist in a database one at a time at 0.04 seconds a query.
The background of the script is it is a caching script. I need to see if it exists in the DB so I need a true/false (obtained by a rowCount) but i also need a way to relate a false to a value so I can update it. I know using a WHERE tag IN (:ARRAY) would work faster than the individual calls, but I cant think of a way to apply an association of true/false to value in this method.
My current code is below:
//loop through all our values!
//prepare out reusuable statement
$stmt = $db->prepare("SELECT * from cache WHERE value=?");
foreach($values as $tempVal)
{
//find if its in the database
try
{
$stmt->execute(array($tempVal));
$valCount = $stmt->rowCount();
} catch(PDOException $ex) {
echo "PDO error send this to us: " . $ex->getMessage();
}
//update flag
$addToUpdate = 1;
//if its in the database
if($valCount > 0)
{
//get the tag data
$valRes= $stmt->fetch();
//check if cache expired
$addToUpdate = 0;
}
//add to update list
if($addToUpdate)
{
//needs updating
$updateList[] = $tempVal;
//add to not in DB list to minimize queries
if($tagTCount == 0)
{
$notInDB[$tempVal] = $tempVal;
}
}
Any suggestions? I can explain more if anything is not clear.
Thank you,
Nick
So you just issue your query with the complete array, using the IN (?,?,?,?,?...) list:
// abstract, use a PDO wrapper of your choosing
$query = db("SELECT * FROM cache WHERE value IN (??)", $values);
Then iterate over the result list. Only matched $values will return. So build your first list from that:
foreach ($query as $row) {
$updateList[] = $row["value"];
}
To get the list of absent entries, just diff that against your original array:
$notInDB = array_diff($values, $updateList);
You could of course use a second NOT IN query. But doing that differentiation in PHP is simpler.
Related
I am using PDO to execute a query for which I am expecting ~500K results. This is my query:
SELECT Email FROM mytable WHERE flag = 1
When I run the query in Microsoft SQL Server management Studio I consistently get 544838 results. I wanted to write a small script in PHP that would fetch these results for me. My original implementation used fetchAll(), but this was exhausting the memory available to php, so I decided to fetch the results one at a time like so:
$q = <<<QUERY
SELECT Email FROM mytable WHERE flag = 1
QUERY;
$stmt = $conn->prepare($q);
$stmt->execute();
$c = 0;
while ($email = $stmt->fetch()[0]) {
echo $email." $c\n";
$c++;
}
but each time I run the query, I get a different number of results! Typical results are:
445664
445836
445979
The number of results seems to be short 100K +/- 200 ish. Any help would be greatly appreciated.
fetch() method fetches one row at a time from current result set. $stmt->fetch()[0] is the first column of the current row.
Your sql query has no ordering and can have some null or empty values (probably).
Since you are controlling this column value in while loop, if the current row's first value is null, it will exit from the loop.
Therefore, you should control only fetch(), not fetch()[0] or something like that.
Also, inside the while loop, use sqlsrv_get_field() to access the columns by index.
$c = 0;
while ($stmt->fetch()) { // You may want to control errors
$email = sqlsrv_get_field($stmt, 0); // get first column value
// $email can be false on errors
echo $email . " $c\n";
$c++;
}
sqlsrv_fetch
So, here's my issue: I have a function that queries sql and pulls it into a resource. In this function, I run:
if (odbc_num_rows($rs) === 0) {
return FALSE;
} else {
if ($type !== "Issues") {
while (odbc_fetch_row($rs)) {
$issues['tvi'][] = odbc_result($rs, 'TitleVolumeIssue_c_');
}
}
return $rs;
}
This does exactly what it is supposed to do. However, when I pass $rs into the next method for parsing into html, it seems as though $rs gets unset. Oddly, I can call odbc_num_rows($rs) and it gives me the correct number of rows, but dumping the var shows it is false and I can't loop through the resource to get any values.
How can I either free up that resource so that it can be used in the next function or how can I rewrite the IF condition so that I get the values without unsetting the resource?
Each time you fetch it moves the database cursor to the next row.
So odbc_fetch_row() keeps going to the next row, until the last one. After that it returns false.
Unfortunately you cannot move the row pointer back to the first record.
You will have to query your DB one more time.
Another option for this would be to store the odbc result in an array that you can loop through using a foreach. This preserves the data and you only have to query once vs multiple times (helps when dealing with a lot of data).
while($row = odbc_fetch_array($rs)) {
$resultsArray[] = $row;
}
Then loop through it like this...
foreach ($resultsArray as $key=>$value){
//Do what you need to do...
}
I had a similar problem in my question here and #Jeff Puckett was able to explain it pretty well.
So let me explain my problem, lets assume that I run query like so:
$myquery = sql_query("SELECT name FROM table WHERE name='example' LIMIT 0,1");
Now.. I want to store the retrieved name into a variable so I would do something like this:
while ($myrow = sql_fetch_assoc($myquery)) {
transfer_row($myrow);
print"Name: $row_name";
}
$stored_name = $row_name;
NOTE: transfer_row() is just a function I wrote that takes $myrow['name'] and stores it in $row_name, for easier reference
Now, all is fine at this stage, here is where it gets interesting. Note that at this stage I still have a name assigned to $row_name. Further down the page I run another query to retrieve some other information from the table, and one of the things I need to retrieve is a list of names again, so I would simply run this query:
$myquery = sql_query("SELECT name, year FROM table WHERE DESC LIMIT 0,10");
while ($myrow = sql_fetch_assoc($myquery)) {
transfer_row($myrow);
$year = $row_year;
$link = "/$year";
print "<li style=\"margin-bottom: 6px;\">$row_name\n";
}
Now, I want to write an if statement that executes something if the $row_name from this query matches the $row_name from the old query, this is why we stored the first $row_name inside the variable.
if ($row_name == $stored_name){
// execute code
}
However as most of you know, this WONT work, the reason is, it simply takes $stored_name again and puts the new $row_name into $stored_name, so therefore the value of the first $row_name is lost, now it is crucial for my application that I access the first $row_name and compare it AFTER the second query has been run, what can I do here people? if nothing can be done what is an alternative to achieving something like this.
Thanks.
EDIT, MY transfer_row() function:
function transfer_row($myrow) {
global $GLOBALS;
if(is_array($myrow)) {
foreach ($myrow as $key=>$value) {
$key=str_replace(":","",$key);
$GLOBALS["row_$key"] = $value;
}
}
}
Without you posting the code for the function transfer_row, we won't be able to give you an answer that exactly matches what you request, but I can give you an answer that will solve the problem at hand.
When matching to check if the names are the same, you can modify the if statement to the following.
if ($row_name == $myrow['name']){
// execute code
}
What I suggest you do though, but since I don't have the code to the transfer_row function, is to pass a second variable to that function. The second variable will be a prefix for the variable name, so you can have unique values stored and saved.
Refrain from using the transfor_row function in the second call so your comparison becomes:
if ($myrow['name'] == $row_name)
If you need to use this function, you could do an assignment before the second database call:
$stored_name = $row_name;
...
transfer_row($myrow);
In your first query you are selecting the name field WHERE name='example' , Why are you quering then? You already have what you want.
Your are querying like:
Hey? roll no 21 what is your roll no?
So perform the second query only and use the if condition as :
if ($row_name == 'example'){
// execute code
}
Does it make sense?
Update
//How about using prefix while storing the values in `$GLOBAL` ??
transfer_row($myrow, 'old_'); //for the first query
transfer_row($myrow, 'new_'); //for the second query
function transfer_row($myrow, $prefix) {
global $GLOBALS;
if(is_array($myrow)) {
foreach ($myrow as $key=>$value) {
$key=str_replace(":","",$key);
$GLOBALS["$prefix"."row_$key"] = $value;
}
}
}
//Now compare as
if ($new_row_name == $old_row_name){
// execute code
}
//You'll not need `$stored_name = $row_name;` any more
I am currently busy on a textbased RPG game, but I am stuck at one part right now.
In order to start a mission, the player does need some items, these are stored in a string: item:1x3-item:5x1 - (basicly item:IDxamount).I have already made a function that explodes the string into variables, but now the script needs to check if the player does have all the items listed.
I've tried to solve the issue with a foreach, but that returns positive or negative for every item, and I only need to know if the player has all items at once.
(don't mind the unsafe query)
$parseAmount is an array, containing all item ID's.
$uid is an variable containing userID
// check if player has all items
foreach($parseAmount as $itemID)
{
$check_query = mysql_query("SELECT * FROM `player_items` WHERE `player`='$uid' AND `item`=='$itemID' AND `value`>='$parseAmount[1]'");
if(mysql_num_rows($check_query)>=1)
{return true;}
else
{return false;}
}
If you want me to post the whole function, please let me know.
If I understood your question correctly you need something like:
foreach($parseAmount as $itemID) {
$sql = "SELECT COUNT(*) AS count
FROM player_items
WHERE player = '".mysql_real_escape_string($uid)."'
AND item = '".mysql_real_escape_string($itemID)."'
AND value >= ".intval($parseAmount[1]);
$row = mysql_fetch_array(mysql_query($sql));
if ($row['count'] == 0) {
return false;
}
}
return true;
You must not early return true. You know the result is true only after checking all the items. My code could be improved by selecting all the items at once, but it's up to you to build this.
Keep in mind my comment about the deprecation of the MySQL extension, using MySQLi and Prepared Statements it will look something like this (note that I never worked with MySQLi before and built it with help of the manual):
foreach($parseAmount as $itemID) {
$sql = "SELECT COUNT(*) AS count
FROM player_items
WHERE player = ?
AND item = ?
AND value >= ?"
$stmt = $mysqli->prepare($sql);
$stmt->bind_param("ssi", $uid, $itemID, $parseAmount[1]);
$stmt->execute();
$row = $stmt->get_result()->fetch_array();
if ($row['count'] == 0) {
return false;
}
}
return true;
Assuming that the mysqli object is already instantiatied (and connected) with the global variable $mysql, here is the code I am trying to work with.
class Listing {
private $mysql;
function getListingInfo($l_id = "", $category = "", $subcategory = "", $username = "", $status = "active") {
$condition = "`status` = '$status'";
if (!empty($l_id)) $condition .= "AND `L_ID` = '$l_id'";
if (!empty($category)) $condition .= "AND `category` = '$category'";
if (!empty($subcategory)) $condition .= "AND `subcategory` = '$subcategory'";
if (!empty($username)) $condition .= "AND `username` = '$username'";
$result = $this->mysql->query("SELECT * FROM listing WHERE $condition") or die('Error fetching values');
$this->listing = $result->fetch_array() or die('could not create object');
foreach ($this->listing as $key => $value) :
$info[$key] = stripslashes(html_entity_decode($value));
endforeach;
return $info;
}
}
there are several hundred listings in the db and when I call $result->fetch_array() it places in an array the first row in the db.
however when I try to call the object, I can't seem to access more than the first row.
for instance:
$listing_row = new Listing;
while ($listing = $listing_row->getListingInfo()) {
echo $listing[0];
}
this outputs an infinite loop of the same row in the db. Why does it not advance to the next row?
if I move the code:
$this->listing = $result->fetch_array() or die('could not create object');
foreach ($this->listing as $key => $value) :
$info[$key] = stripslashes(html_entity_decode($value));
endforeach;
if I move this outside the class, it works exactly as expected outputting a row at a time while looping through the while statement.
Is there a way to write this so that I can keep the fetch_array() call in the class and still loop through the records?
Your object is fundamentally flawed - it's re-running the query every time you call the getListingInfo() method. As well, mysql_fetch_array() does not fetch the entire result set, it only fetches the next row, so your method boils down to:
run query
fetch first row
process first row
return first row
Each call to the object creates a new query, a new result set, and therefore will never be able to fetch the 2nd, 3rd, etc... rows.
Unless your data set is "huge" (ie: bigger than you want to/can set the PHP memory_limit), there's no reason to NOT fetch the entire set and process it all in one go, as shown in Jacob's answer above.
as a side note, the use of stripslashes makes me wonder if your PHP installation has magic_quotes_gpc enabled. This functionality has been long deprecrated and will be removed from PHP whenever v6.0 comes out. If your code runs as is on such an installation, it may trash legitimate escaping in the data. As well, it's generally a poor idea to store encoded/escaped data in the database. The DB should contain a "virgin" copy of the data, and you then process it (escape, quote, etc...) as needed at the point you need the processed version.