I have quite an issue I can not seem to solve. I am trying to get a row count from a select statement.
I should start by saying I have tried most all methods resulting from google searches on this issue.
I am using the result set so I would prefer not to make a second query.
The query uses a prepared select statement which seems to be a main issue if I understand it correctly.
I decided to try a simple approach using PHP's native count() function. Which lead me here because I finally reached the end of the rope on this.
On to the details...within a class of mine, I make the query like this.
// Create database connection
$database = DatabaseFactory::getFactory()->getConnection();
// Set and execute database query
$sql = "SELECT * FROM `service_orders` WHERE `agency_id` = :agency_id $filter ORDER BY $sort $order $per_page";
$query = $database->prepare($sql);
$query->execute($query_array);
// If results
if ($query->rowCount() > 0) {
$results = $query->fetchAll();
self::$order_count = "Count: " . count($results);
return $results;
}
// Default, return false
return false;
Findings
If I perform count($results) like I did above, I get the total rows in the database (Let's say 50).
If I print_r($results), it shows the array with the proper number of entries (Let's say 10) that of course differs from the total rows in the database.
How can these two differ? It's as if the count($results) is misreading the result array.
More Findings
Within my actual php page, I call the class to retrieve the data like this.
$results = OrderModel::getServiceOrders();
echo count($results);
Strangely enough, if I then perform count($results) it gives me the correct reading of the result array (which in my example here would be 10).
I am perplexed by this as the count function is being performed on the exact same array. The only difference is one is called on the array within the class, and the other is called on the array returned from the class.
Does anyone have any ideas on how to solve this or why there is the discrepancy when using count() in this instance?
Thank you all in advance!
James
Additional Info
This is another mind numbing scenario. If I return the count along with the actual results, I can access it on the page with the correct value (10 rows). Yet, if I set it into a session variable, and access it that way, the count is the whole data set (50 rows). How is it even possible these two values are not the same?!
$results = $query->fetchAll();
Session::set("order_count", $total[0]); // Yields 50 (incorrect)
return [
"results"=> $results,
"count"=> $total[0], // Yields 10 (correct)
];
Related
How do I use get_compiled_select or count_all_results before running the query without getting the table name added twice? When I use $this->db->get('tblName') after either of those, I get the error:
Not unique table/alias: 'tblProgram'
SELECT * FROM (`tblProgram`, `tblProgram`) JOIN `tblPlots` ON `tblPlots`.`programID`=`tblProgram`.`pkProgramID` JOIN `tblTrees` ON `tblTrees`.`treePlotID`=`tblPlots`.`id` ORDER BY `tblTrees`.`id` ASC LIMIT 2000
If I don't use a table name in count_all_results or $this->db->get(), then I get an error that no table is used. How can I get it to set the table name just once?
public function get_download_tree_data($options=array(), $rand=""){
//join tables and order by tree id
$this->db->reset_query();
$this->db->join('tblPlots','tblPlots.programID=tblProgram.pkProgramID');
$this->db->join('tblTrees','tblTrees.treePlotID=tblPlots.id');
$this->db->order_by('tblTrees.id', 'ASC');
//get number of results to return
$allResults=$this->db->count_all_results('tblProgram', false);
//chunk data and write to CSV to avoid reaching memory limit
$offset=0;
$chunk=2000;
$treePath=$this->config->item('temp_path')."$rand/trees.csv";
$tree_handle=fopen($treePath,'a');
while (($offset<$allResults)) {
$this->db->limit($chunk, $offset);
$result=$this->db->get('tblProgram')->result_array();
foreach ($result as $row) {
fputcsv($tree_handle, $row);
}
$offset=$offset+$chunk;
}
fclose($tree_handle);
return array('resultCount'=>$allResults);
}
To count how many rows would be returned by a query, essentially all the work must be performed. That is, it is impractical to get the count, then perform the query; you may as well just do the query.
If your goal is to "paginate" by getting some of the rows, plus the total count, that is essentially two separate actions (that may be combined to look like one.)
If the goal is to estimate the number of rows, then SHOW TABLE STATUS or SELECT Rows FROM information_schema.TABLES WHERE ... gives you an estimate.
If you want to see if there are, say "at least 100 rows", then this may be practical:
SELECT 1 FROM ... WHERE ... ORDER BY ... LIMIT 99,1
and see if you get a row back. However, this may or may not be efficient, depending on the indexes and the WHERE and the ORDER BY. (Show us the query and I can elaborate.)
Using OFFSET for chunking is grossly inefficient. If there is not a usable index, then it is performing essentially the entire query for each chunk. If there is a usable index, the chunks are slower and slower. Here is a discussion of why OFFSET is not good for "pagination", plus an efficient workaround: Pagination . It talks about how to "remember where you left off " as an efficient technique for chunking. Fetch between 100 and 1000 rows per chunk.
The flaw in your code is that it aims to select a subset of some records and their total count in the same query. This is impossible in MySQL, so you cannot generate such a query, hence, you get the error as mentioned. The problem is that if you do a
select ... from t where ... limit 0, 2000
then you get maximum 2000 records, so, if the total records matching the criteria have a count that is greater than the limit, then you will not get accurately the count from above, so, in that case you need a
select count(1) from t where ...
This means that you need to build your actual query (the code below your count_all_results call), see whether the number of results reaches the limit. If the number of results does not reach the limit, then you do not need to perform a separate query in order to get the count, because you can compute $offset * $chunk + $recordCount. However, if you get as many records as they can be, then you will need to build another query, without the order_by call, since the count is independent of your sort and get the counts.
$this->db->count_all_results()
Counting the number of returned results with count_all_results()
It's useful to count the number of results returned—often bugs can arise if a section of code which expects to have at least one row is passed zero rows. Without handling the eventuality of a zero result, an application may become unpredictably unstable and may give away hints to a malicious user about the architecture of the app. Ensuring correct handling of zero results is what we're going to focus on here.
Permits you to determine the number of rows in a particular Active Record query. Queries will accept Query Builder restrictors such as where(), or_where(), like(), or_like(), etc. Example:
echo $this->db->count_all_results('my_table'); // Produces an integer, like 25
$this->db->like('title', 'match');
$this->db->from('my_table');
echo $this->db->count_all_results(); // Produces an integer, like 17
However, this method also resets any field values that you may have passed to select(). If you need to keep them, you can pass FALSE as the second parameter:
echo $this->db->count_all_results('my_table', FALSE);
get_compiled_select()
The method $this->db->get_compiled_select(); is introduced in codeigniter v3.0 and compiles active records query without actually executing it. But this is not a completely new method. In older versions of CI it is like $this->db->_compile_select(); but the method has been made protected in later versions making it impossible to call back.
// Note that the second parameter of the get_compiled_select method is FALSE
$sql = $this->db->select(array('field1','field2'))
->where('field3',5)
->get_compiled_select('mytable', FALSE);
// ...
// Do something crazy with the SQL code... like add it to a cron script for
// later execution or something...
// ...
$data = $this->db->get()->result_array();
// Would execute and return an array of results of the following query:
// SELECT field1, field1 from mytable where field3 = 5;
NOTE:- Double calls to get_compiled_select() while you’re using the Query Builder Caching functionality and NOT resetting your queries will results in the cache being merged twice. That in turn will i.e. if you’re caching a select() - select the same field twice.
Rick James got me on the right track. I ended up having to chunk the results using pagination AND a nested query. Using LIMIT on even 1 chunk of 2000 records was timing out. This is the code I ended up with, which uses get_compiled_select('tblProgram') and then get('tblTrees O1'). Since I didn't use FALSE as the second argument to get_compiled_select, the query was cleared before the get() was run.
//grab the data in chunks, write it to CSV chunk by chunk
$offset=0;
$chunk=2000;
$i=10; //counter for the progress bar
$this->db->limit($chunk);
$this->db->select('tblTrees.id');
//nesting the limited query and then joining the other field later improved performance significantly
$query1=' ('.$this->db->get_compiled_select('tblProgram').') AS O2';
$this->db->join($query1, 'O1.id=O2.id');
$result=$this->db->get('tblTrees O1')->result_array();
$allResults=count($result);
$putHeaders=0;
$treePath=$this->config->item('temp_path')."$rand/trees.csv";
$tree_handle=fopen($treePath,'a');
//while select limit returns the limit
while (count($result)===$chunk) {
$highestID=max(array_column($result, 'id'));
//update progres bar with estimate
if ($i<90) {
$this->set_runStatus($qcRunId, $status = "processing", $progress = $i);
$i=$i+1;
}
//only get the fields the first time
foreach ($result as $row) {
if ($offset===0 && $putHeaders===0){
fputcsv($tree_handle, array_keys($row));
$putHeaders=1;
}
fputcsv($tree_handle, $row);
}
//get the next chunk
$offset=$offset+$chunk;
$this->db->reset_query();
$this->make_query($options);
$this->db->order_by('tblTrees.id', 'ASC');
$this->db->where('tblTrees.id >', $highestID);
$this->db->limit($chunk);
$this->db->select('tblTrees.id');
$query1=' ('.$this->db->get_compiled_select('tblProgram').') AS O2';
$this->db->join($query1, 'O1.id=O2.id');
$result=$this->db->get('tblTrees O1')->result_array();
$allResults=$allResults+count($result);
}
//write out last chunk
foreach ($result as $row) {
fputcsv($tree_handle, $row);
}
fclose($tree_handle);
return array('resultCount'=>$allResults);
$out = odbc_exec($connection, $query);
$first = odbc_fetch_array($out);
works just fine
$out = odbc_exec($connection, $query);
$first = odbc_num_rows($out);
works just fine
$out = odbc_exec($connection, $query);
$first = odbc_fetch_array($out);
if(odbc_num_rows($out)) {
//This should execute true, but doesn't
}
To my knowledge, any variable you pass ($out) to a function are only read.
How can calling fetch_array cause num_rows to error?
Update
I've even copied the result straight after the odbc_exec into a 'temp' variable & used that on the fetch_array, yet still it throws. seriously, what the f.
From PHP Manual:
Note: Using odbc_num_rows() to determine the number of rows available
after a SELECT will return -1 with many drivers.
odbc_num_rows method is typically used to detect the number of rows modified by an INSERT, UPDATE, or DELETE call.
If you want the full query returned plus the length, loop through the results using odbc_fetch_array storing them into an array and then determine the number of tuples from the array length.
If you are only wanting to query the number of rows in a given view, then you are better off to write a query that returns a COUNT and fetch that as an array.
When I execute a PDO statement, internally a result set is stored, and I can use ->fetch() to get a row from the result.
If I wanted to convert the entire result to an array, I could do ->fetchAll().
With Laravel, in the Query Builder docs, I only see a way to get an array result from executing the query.
// example query, ~30,000 rows
$scores = DB::table("highscores")
->select("player_id", "score")
->orderBy("score", "desc")
->get();
var_dump($scores);
// array of 30,000 items...
// unbelievable ...
Is there any way to get a result set from Query Builder like PDO would return? Or am I forced to wait for Query Builder to build an entire array before it returns a value ?
Perhaps some sort of ->lazyGet(), or ->getCursor() ?
If this is the case, I can't help but see Query Builder is an extremely short-sighted tool. Imagine a query that selects 30,000 rows. With PDO I can step through row by row, one ->fetch() at a time, and handle the data with very little additional memory consumption.
Laravel Query Builder on the other hand? "Memory management, huh? It's fine, just load 30,000 rows into one big array!"
PS yes, I know I can use ->skip() and ->take() to offset and limit the result set. In most cases, this would work fine, as presenting a user with 30,000 rows is not even usable. If I want to generate large reports, I can see PHP running out of memory easily.
After #deczo pointed out an undocumented function ->chunk(), I dug around in the source code a bit. What I found is that ->chunk() is a convenience wrapper around multiplying my query into several queries queries but automatically populating the ->step($m)->take($n) parameters. If I wanted to build my own iterator, using ->chunk with my data set, I'd end up with 30,000 queries on my DB instead of 1.
This doesn't really help, too, because ->chunk() takes a callback which forces me to couple my looping logic at the time I'm building the query. Even if the function was defined somewhere else, the query is going to happen in the controller, which should have little interest in the intricacies of my View or Presenter.
Digging a little further, I found that all Query Builder queries inevitably pass through \Illuminate\Database\Connection#run.
// https://github.com/laravel/framework/blob/3d1b38557afe0d09326d0b5a9ff6b5705bc67d29/src/Illuminate/Database/Connection.php#L262-L284
/**
* Run a select statement against the database.
*
* #param string $query
* #param array $bindings
* #return array
*/
public function select($query, $bindings = array())
{
return $this->run($query, $bindings, function($me, $query, $bindings)
{
if ($me->pretending()) return array();
// For select statements, we'll simply execute the query and return an array
// of the database result set. Each element in the array will be a single
// row from the database table, and will either be an array or objects.
$statement = $me->getReadPdo()->prepare($query);
$statement->execute($me->prepareBindings($bindings));
return $statement->fetchAll($me->getFetchMode());
});
}
See that nasty $statement->fetchAll near the bottom ?
That means arrays for everyone, always and forever; your wishes and dreams abstracted away into an unusable tool Laravel Query Builder.
I can't express the valley of my depression right now.
One thing I will say though is that the Laravel source code was at least organized and formatted nicely. Now let's get some good code in there!
Use chunk:
DB::table('highscores')
->select(...)
->orderBy(...)
->chunk($rowsNumber, function ($portion) {
foreach ($portion as $row) { // do whatever you like }
});
Obviously returned result will be just the same as calling get, so:
$portion; // array of stdObjects
// and for Eloquent models:
Model::chunk(100, function ($portion) {
$portion; // Collection of Models
});
Here is a way to use the laravel query builder for making the query, but to then use the underlying pdo fetch to loop over the record set which I believe will solve your problem - running one query and looping the record set so you don't run out of memory on 30k records.
This approach will use all the config stuff you setup in laravel so you don't have to config pdo separately.
You could also abstract out a method to make this easy to use that takes in the query builder object, and returns the record set (executed pdo statement), which you would then while loop over as below.
$qb = DB::table("highscores")
->select("player_id", "score")
->orderBy("score", "desc");
$connection = $qb->getConnection();
$pdo = $connection->getPdo();
$query = $qb->toSql();
$bindings = $qb->getBindings();
$statement = $pdo->prepare($query);
$statement->execute($bindings);
while ($row = $statement->fetch($connection->getFetchMode()))
{
// do stuff with $row
}
I have this function in a linked file:
function getNumberOfHerps() {
$sql = "SELECT COUNT(ID)
FROM HERPES;";
return DBIface::connect()->query($sql);
}
I can call on this function from any other page in the website. What I want to do is to be able to use the result of the COUNT function in my php code on various pages, because I need to know how many herps there are in the database on several occasions in the code.
I tried this:
$result = getNumberOfHerps();
$numberOfHerps = $result['COUNT'];
But it caused a parse error on the second line, saying this:
Cannot use object of type PDOStatement as array.
Please tell me how I can use the result of a Count function in php code. Thanks :D
You are actually facing 2 separate items here that need your attention:
You can't access the COUNT(ID) returned from your query and
You are attempting to access data of a PDOStatement object as an array
So:
You can create an ALIAS for the value returned from the COUNT() function:
$sql = "SELECT COUNT(ID) as COUNT...";
This will provide to your result set a new column named COUNT that you can access your row count from.
Secondly, you need to perform a fetch operation on the PDOStatement in order to access it's data (this is what is causing your parse error). You have several options for accessing the result set in a PDOStatement:
list ($idCount) = $result->fetch();
// $idCount will now have the value of your COUNT column
or
$results = $result->fetch(PDO::FETCH_ASSOC);
// you can then access your COUNT as $results['COUNT'];
or even
$results = $result->fetch(PDO::FETCH_OBJ);
// you can then access your count as $results->COUNT
There are actually several additional FETCH_* styles available to use. The ones I've described though would likely be the most common. You can find more information about fetching PDO result sets at
http://us1.php.net/manual/en/pdostatement.fetch.php
I worked it out for myself, so I'll put my answer up for anyone else who finds this question:
I changed the function to this:
function getNumberOfHerps() {
$sql = "SELECT COUNT(1)
FROM HERPES";
return DBIface::connect()->query($sql)->fetchColumn(0);
}
So what the function now returns is the actual count value (the first column of the query). That means that in the webpage php file, you can access it in this way:
$numberOfHerps = getNumberOfHerps();
Much more simple.
I'm quite annoyed with CI's behavior lately, may be because i'm little ignorant towards small stuff, now to the question :D. i'm making an application where i'm trying to get one single row using active records, now as a good programmer(i think) i try this
when i select such as login
if($query->num_rows() == 1)
{
// do something
}
// now when i make an update, that also a single row i tend to use this command
if($query->affected_rows() == 1)
{
// perform some action
}
// now i use this code when i've to get all entries of one single user
if($query->num_rows() > 0)
{
// show them
}
now many time i get error like "Call to member function num_rows() on non object"
can anyone please explain what to use, when to use and how to use, so that code igniter never gets made a me
1.num_rows()
num_rows() are result helper functions, they are used against a query resultobject, note that we don't use it like $this->db->num_rows(), instead we use it like:
$query = $this-query('query');
OR
$query = $this->db->get('tableName');
if($query->num_rows()>0){ ....}
Now lets see affected_rowS().
2. affected_rows()
affected_rows is a query helper function, they are used like $this->db->affected_rows(). Note that $this->db->affected_rows() yields valid result only after a valid db write operation. i.e insert, update and delete. The database class has a small hack that allows it to return the correct number of affected rows after a delete operation because MySQL "DELETE FROM TABLE" returns 0 affected rows.
3.$query->result()
This function returns the query result as an array of objects, or an empty array on failure.
i.e if you have more than one result you have to loop the result object like :
foreach ($query->result() as $row){....}
4.$query->row()
This function returns a single result row as an object. If your query has more than one row, it returns only the first row.
I hope my explanation is readable enough.. Although you could find all of these in CI docs with relevant examples.