I have a database containing a large amount of information that a user can query using PHP to communicate with the MySQL database. I also want to limit their results to be at most 300 results, which I have successfully done. However, I also want to sort the results by dataset name. Is there a way to sort the end query, not the original table? From what I've read online, the ORDER BY statement in a query sorts the whole table and then does the query (or something similar) Sorting the query would be more efficient, as you would only be sorting say 15 records instead of 1500000. If it helps at all, I print the results to the page using the following syntax:
while ($row = mysql_fetch_array($result)) {
...
}
Thanks
If the result set is limited, save it into a php array then use the php sorting functions: http://php.net/manual/en/function.usort.php
$all = array();
while ($res = mysql_fetch_array($result)) {
$all[] = $res;
}
usort($all, function($a,$b) {
return $a['field'] < $b['field'];
});
Just make sure what you are sorting by has a decent index and the sort option should be very fast. The issue with sorting before/after a LIMIT is that if it were possible (which it is with a bit of GROUP BY logic), it would change the actual results you get as opposed to just the order they return in.
To create a sub table and only sort on that you could attempt something like:
Select * from ( QUERY limit 300) a order by COLUMN
Related
How do I use get_compiled_select or count_all_results before running the query without getting the table name added twice? When I use $this->db->get('tblName') after either of those, I get the error:
Not unique table/alias: 'tblProgram'
SELECT * FROM (`tblProgram`, `tblProgram`) JOIN `tblPlots` ON `tblPlots`.`programID`=`tblProgram`.`pkProgramID` JOIN `tblTrees` ON `tblTrees`.`treePlotID`=`tblPlots`.`id` ORDER BY `tblTrees`.`id` ASC LIMIT 2000
If I don't use a table name in count_all_results or $this->db->get(), then I get an error that no table is used. How can I get it to set the table name just once?
public function get_download_tree_data($options=array(), $rand=""){
//join tables and order by tree id
$this->db->reset_query();
$this->db->join('tblPlots','tblPlots.programID=tblProgram.pkProgramID');
$this->db->join('tblTrees','tblTrees.treePlotID=tblPlots.id');
$this->db->order_by('tblTrees.id', 'ASC');
//get number of results to return
$allResults=$this->db->count_all_results('tblProgram', false);
//chunk data and write to CSV to avoid reaching memory limit
$offset=0;
$chunk=2000;
$treePath=$this->config->item('temp_path')."$rand/trees.csv";
$tree_handle=fopen($treePath,'a');
while (($offset<$allResults)) {
$this->db->limit($chunk, $offset);
$result=$this->db->get('tblProgram')->result_array();
foreach ($result as $row) {
fputcsv($tree_handle, $row);
}
$offset=$offset+$chunk;
}
fclose($tree_handle);
return array('resultCount'=>$allResults);
}
To count how many rows would be returned by a query, essentially all the work must be performed. That is, it is impractical to get the count, then perform the query; you may as well just do the query.
If your goal is to "paginate" by getting some of the rows, plus the total count, that is essentially two separate actions (that may be combined to look like one.)
If the goal is to estimate the number of rows, then SHOW TABLE STATUS or SELECT Rows FROM information_schema.TABLES WHERE ... gives you an estimate.
If you want to see if there are, say "at least 100 rows", then this may be practical:
SELECT 1 FROM ... WHERE ... ORDER BY ... LIMIT 99,1
and see if you get a row back. However, this may or may not be efficient, depending on the indexes and the WHERE and the ORDER BY. (Show us the query and I can elaborate.)
Using OFFSET for chunking is grossly inefficient. If there is not a usable index, then it is performing essentially the entire query for each chunk. If there is a usable index, the chunks are slower and slower. Here is a discussion of why OFFSET is not good for "pagination", plus an efficient workaround: Pagination . It talks about how to "remember where you left off " as an efficient technique for chunking. Fetch between 100 and 1000 rows per chunk.
The flaw in your code is that it aims to select a subset of some records and their total count in the same query. This is impossible in MySQL, so you cannot generate such a query, hence, you get the error as mentioned. The problem is that if you do a
select ... from t where ... limit 0, 2000
then you get maximum 2000 records, so, if the total records matching the criteria have a count that is greater than the limit, then you will not get accurately the count from above, so, in that case you need a
select count(1) from t where ...
This means that you need to build your actual query (the code below your count_all_results call), see whether the number of results reaches the limit. If the number of results does not reach the limit, then you do not need to perform a separate query in order to get the count, because you can compute $offset * $chunk + $recordCount. However, if you get as many records as they can be, then you will need to build another query, without the order_by call, since the count is independent of your sort and get the counts.
$this->db->count_all_results()
Counting the number of returned results with count_all_results()
It's useful to count the number of results returned—often bugs can arise if a section of code which expects to have at least one row is passed zero rows. Without handling the eventuality of a zero result, an application may become unpredictably unstable and may give away hints to a malicious user about the architecture of the app. Ensuring correct handling of zero results is what we're going to focus on here.
Permits you to determine the number of rows in a particular Active Record query. Queries will accept Query Builder restrictors such as where(), or_where(), like(), or_like(), etc. Example:
echo $this->db->count_all_results('my_table'); // Produces an integer, like 25
$this->db->like('title', 'match');
$this->db->from('my_table');
echo $this->db->count_all_results(); // Produces an integer, like 17
However, this method also resets any field values that you may have passed to select(). If you need to keep them, you can pass FALSE as the second parameter:
echo $this->db->count_all_results('my_table', FALSE);
get_compiled_select()
The method $this->db->get_compiled_select(); is introduced in codeigniter v3.0 and compiles active records query without actually executing it. But this is not a completely new method. In older versions of CI it is like $this->db->_compile_select(); but the method has been made protected in later versions making it impossible to call back.
// Note that the second parameter of the get_compiled_select method is FALSE
$sql = $this->db->select(array('field1','field2'))
->where('field3',5)
->get_compiled_select('mytable', FALSE);
// ...
// Do something crazy with the SQL code... like add it to a cron script for
// later execution or something...
// ...
$data = $this->db->get()->result_array();
// Would execute and return an array of results of the following query:
// SELECT field1, field1 from mytable where field3 = 5;
NOTE:- Double calls to get_compiled_select() while you’re using the Query Builder Caching functionality and NOT resetting your queries will results in the cache being merged twice. That in turn will i.e. if you’re caching a select() - select the same field twice.
Rick James got me on the right track. I ended up having to chunk the results using pagination AND a nested query. Using LIMIT on even 1 chunk of 2000 records was timing out. This is the code I ended up with, which uses get_compiled_select('tblProgram') and then get('tblTrees O1'). Since I didn't use FALSE as the second argument to get_compiled_select, the query was cleared before the get() was run.
//grab the data in chunks, write it to CSV chunk by chunk
$offset=0;
$chunk=2000;
$i=10; //counter for the progress bar
$this->db->limit($chunk);
$this->db->select('tblTrees.id');
//nesting the limited query and then joining the other field later improved performance significantly
$query1=' ('.$this->db->get_compiled_select('tblProgram').') AS O2';
$this->db->join($query1, 'O1.id=O2.id');
$result=$this->db->get('tblTrees O1')->result_array();
$allResults=count($result);
$putHeaders=0;
$treePath=$this->config->item('temp_path')."$rand/trees.csv";
$tree_handle=fopen($treePath,'a');
//while select limit returns the limit
while (count($result)===$chunk) {
$highestID=max(array_column($result, 'id'));
//update progres bar with estimate
if ($i<90) {
$this->set_runStatus($qcRunId, $status = "processing", $progress = $i);
$i=$i+1;
}
//only get the fields the first time
foreach ($result as $row) {
if ($offset===0 && $putHeaders===0){
fputcsv($tree_handle, array_keys($row));
$putHeaders=1;
}
fputcsv($tree_handle, $row);
}
//get the next chunk
$offset=$offset+$chunk;
$this->db->reset_query();
$this->make_query($options);
$this->db->order_by('tblTrees.id', 'ASC');
$this->db->where('tblTrees.id >', $highestID);
$this->db->limit($chunk);
$this->db->select('tblTrees.id');
$query1=' ('.$this->db->get_compiled_select('tblProgram').') AS O2';
$this->db->join($query1, 'O1.id=O2.id');
$result=$this->db->get('tblTrees O1')->result_array();
$allResults=$allResults+count($result);
}
//write out last chunk
foreach ($result as $row) {
fputcsv($tree_handle, $row);
}
fclose($tree_handle);
return array('resultCount'=>$allResults);
I have quite an issue I can not seem to solve. I am trying to get a row count from a select statement.
I should start by saying I have tried most all methods resulting from google searches on this issue.
I am using the result set so I would prefer not to make a second query.
The query uses a prepared select statement which seems to be a main issue if I understand it correctly.
I decided to try a simple approach using PHP's native count() function. Which lead me here because I finally reached the end of the rope on this.
On to the details...within a class of mine, I make the query like this.
// Create database connection
$database = DatabaseFactory::getFactory()->getConnection();
// Set and execute database query
$sql = "SELECT * FROM `service_orders` WHERE `agency_id` = :agency_id $filter ORDER BY $sort $order $per_page";
$query = $database->prepare($sql);
$query->execute($query_array);
// If results
if ($query->rowCount() > 0) {
$results = $query->fetchAll();
self::$order_count = "Count: " . count($results);
return $results;
}
// Default, return false
return false;
Findings
If I perform count($results) like I did above, I get the total rows in the database (Let's say 50).
If I print_r($results), it shows the array with the proper number of entries (Let's say 10) that of course differs from the total rows in the database.
How can these two differ? It's as if the count($results) is misreading the result array.
More Findings
Within my actual php page, I call the class to retrieve the data like this.
$results = OrderModel::getServiceOrders();
echo count($results);
Strangely enough, if I then perform count($results) it gives me the correct reading of the result array (which in my example here would be 10).
I am perplexed by this as the count function is being performed on the exact same array. The only difference is one is called on the array within the class, and the other is called on the array returned from the class.
Does anyone have any ideas on how to solve this or why there is the discrepancy when using count() in this instance?
Thank you all in advance!
James
Additional Info
This is another mind numbing scenario. If I return the count along with the actual results, I can access it on the page with the correct value (10 rows). Yet, if I set it into a session variable, and access it that way, the count is the whole data set (50 rows). How is it even possible these two values are not the same?!
$results = $query->fetchAll();
Session::set("order_count", $total[0]); // Yields 50 (incorrect)
return [
"results"=> $results,
"count"=> $total[0], // Yields 10 (correct)
];
Is there any php function built in to make it easy to take a database of information and reverse the order in which it echo's while fetching it from a PHP script?
For instance a comment system, it would make it a lot easier if there was a function to reverse the order of the echo's to make the newest comment on the top.
yes...instead of echoing it out as you fetch row, you can first put the rows in an array and use rsort or arsort and then loop through and echo..or..you can do it in the mysql query (ideal) by adding ...order by columnname desc to your query
Check out http://php.net/manual/en/function.array-reverse.php
If you're returning an array from the database, this will do what you're looking for
The other option is simply to reverse the loop:
$count = count($aArray_Name);
for ( $i = $count; $i > 0; $i-- ) {
echo $aArray_Name[$i];
}
May or may not work depending on your situation. Would most definitely be better to just modify the query as mentioned by others.
I'm using an if clause to fetch the value of my mysql table data because my table schema is not normal.
now for getting these values, I wrote the below code:
$result = $db->sql_query("SELECT type, var, count from table_counter");
while ($row = $db->sql_fetchrow($result)) {
$type = $row['type'];
$var = $row['var'];
$count = intval($row['count']);
if(($type == "total") && ($var == "visits")) {
$totalVisits= $count;
}elseif(($type == "total") && ($var == "pageviews")) {
$totalPVisits= $count;
}
}
Is there any other way rather than using an if clause?!
SELECT `type`, `var`, SUM(`count`) AS `sum`
FROM table_counter
GROUP BY `type`,`var`;
There's no need for that intval() to convert the count field to integer. It's already defined as an integer in your MySQL table.
What you have here is some kind of EAV data structure and the if is a common way to fetch from it. There are other ways of pivoting them.
What are you aiming to do with the data? Just create the counts?
You could extract the data in more than one query.
Another option is to put the data into an array, then sort it using PHP array sort functions. It could then be split it into two separate arrays, if needed.
You will need to pivot the data. And that will require you to write some code to parse out what you need in php and mysql. This will can get heavy depending on how much data you want and how it's stored and how it needs to be retrieved. Based on your question text it looks you are storing very simple data, but for every type and set of data you are storing you will need to create logic for it.
Soooo, if you know exactly what the variables are going to be you could set up and array of the variable names you want set and loop through them.
$legal_values = array('visits', 'pageviews', etc...);
while ($row = $db->sql_fetchrow($result)) {
if (in_array($row['var'], $legal_values)) {
$$row['var'] = $intval($row['count'];
}
}
This is just a simple example and only works if all the types are count. Obviously if you want to filter it down you will need more logic. There isn't really a catch-all solution for this stuff, so it comes down to writing something that is simple so you understand it and you can easily extend it to handle more data that you might need later.
${$row['type'] . ucfirst($row['var'])} = $row['count'];
That reads total pageviews 107 to $totalPageviews = 107. Depending on where you want to use this, this may be a possibility.
I have two msyql tables, Badges and Events. I use a join to find all the events and return the badge info for that event (title & description) using the following code:
SELECT COUNT(Badges.badge_ID) AS
badge_count,title,Badges.description
FROM Badges JOIN Events ON
Badges.badge_id=Events.badge_id GROUP
BY title ASC
In addition to the counts, I need to know the value of the event with the most entries. I thought I'd do this in php with the max() function, but I had trouble getting that to work correctly. So, I decided I could get the same result by modifying the above query by using "ORDER BY badgecount DESC LIMIT 1," which returns an array of a single element, whose value is the highest count total of all the events.
While this solution works well for me, I'm curious if it is taking more resources to make 2 calls to the server (b/c I'm now using two queries) instead of working it out in php. If I did do it in php, how could I get the max value of a particular item in an associative array (it would be nice to be able to return the key and the value, if possible)?
EDIT:
OK, it's amazing what a few hours of rest will do for the mind. I opened up my code this morning, and made a simple modification to the code, which worked out for me. I simply created a variable on the count field and, if the new one was greater than the old one, changed it to the new value (see the "if" statement in the following code):
if ( $c > $highestCount ) {
$highestCount = $c; }
This might again lead to a "religious war", but I would go with the two queries version. To me it is cleaner to have data handling in the database as much as possible. In the long run, query caching, etc.. would even out the overhead caused by the extra query.
Anyway, to get the max in PHP, you simply need to iterate over your $results array:
getMax($results) {
if (count($results) == 0) {
return NULL;
}
$max = reset($results);
for($results as $elem) {
if ($max < $elem) { // need to do specific comparison here
$max = $elem;
}
}
return $max;
}