I have a query that running way too slow. the page takes a few minutes to load.
I'm doing a table join on tables with over 100,000 records. In my query, is it grabbing all the records or is it getting only the amount I need for the page? Do I need to put a limit in the query? If I do, won't that give the paginator the wrong record count?
$paymentsTable = new Donations_Model_Payments();
$select = $paymentsTable->select(Zend_Db_Table::SELECT_WITH_FROM_PART);
$select->setIntegrityCheck(false)
->from(array('p' => 'tbl_payments'), array('clientid', 'contactid', 'amount'))
->where('p.clientid = ?', $_SESSION['clientinfo']['id'])
->where('p.dt_added BETWEEN \''.$this->datesArr['dateStartUnix'].'\' AND \''.$this->datesArr['dateEndUnix'].'\'')
->join(array('c' => 'contacts'), 'c.id = p.contactid', array('fname', 'mname', 'lname'))
->group('p.id')
->order($sortby.' '.$dir)
;
$payments=$paymentsTable->fetchAll($select);
// paginator
$paginator = Zend_Paginator::factory($payments);
$paginator->setCurrentPageNumber($this->_getParam('page'), 1);
$paginator->setItemCountPerPage('100'); // items pre page
$this->view->paginator = $paginator;
$payments=$payments->toArray();
$this->view->payments=$payments;
Please see revised code below. You need to pass the $select to Zend_Paginator via the correct adapter. Otherwise you won't see the performance benefits.
$paymentsTable = new Donations_Model_Payments();
$select = $paymentsTable->select(Zend_Db_Table::SELECT_WITH_FROM_PART);
$select->setIntegrityCheck(false)
->joinLeft('contacts', 'tbl_payments.contactid = contacts.id')
->where('tbl_payments.clientid = 39')
->where(new Zend_Db_Expr('tbl_payments.dt_added BETWEEN "1262500129" AND "1265579129"'))
->group('tbl_payments.id')
->order('tbl_payments.dt_added DESC');
// paginator
$paginator = new Zend_Paginator(new Zend_Paginator_Adapter_DbTableSelect($select));
$paginator->setCurrentPageNumber($this->_getParam('page', 1));
$paginator->setItemCountPerPage('100'); // items pre page
$this->view->paginator = $paginator;
Please see revised code above!
In your code, you are :
first, selecting and fetching all records that match your condition
see the select ... from... and all that
and the call to fetchAll on the line just after
and, only the, you are using the paginator,
on the results returned by the fetchAll call.
With that, I'd say that, yes, all your 100,000 records are fetched from the DB, manipulated by PHP, passed to Zend_Paginator which has to work with them... only to discard almost all of them.
Using Zend_Paginator, you should be able to pass it an instance of Zend_Db_Select, and let it execute the query, specifying the required limit.
Maybe the example about DbSelect and DbTableSelect adapter might help you understand how this can be achieved (sorry, I don't have any working example).
I personally count the results via COUNT(*) and pass that to zend_paginator. I never understood why you'd deep link zend_paginator right into the database results. I can see the pluses and minuses, but really, its to far imho.
Bearing in mind that you only want 100 results, you're fetching 100'000+ and then zend_paginator is throwing them away. Realistically you want to just give it a count.
$items = Eurocreme_Model::load_by_type(array('type' => 'list', 'from' => $from, 'to' => MODEL_PER_PAGE, 'order' => 'd.id ASC'));
$count = Eurocreme_Model::load_by_type(array('type' => 'list', 'from' => 0, 'to' => COUNT_HIGH, 'count' => 1));
$paginator = Zend_Paginator::factory($count);
$paginator->setItemCountPerPage(MODEL_PER_PAGE);
$paginator->setCurrentPageNumber($page);
$this->view->paginator = $paginator;
$this->view->items = $items;
Related
I have some problems with the limit using active record.
I've created a dataProvider with a limit of 5:
$dataProvider = new ActiveDataProvider([
'query' => Devicesdb::find()
->joinWith('score')
->where('devices.devicetype = :deviceType', [':deviceType' => $device])
->orderBy(['score' => SORT_DESC])
->limit(5),
'totalCount' => 5,
]);
And this is the resultant query in debug panel:
SELECT `devicesdb`.*
FROM `devicesdb`
LEFT JOIN `devices`
ON `devicesdb`.`id` = `devices`.`id`
WHERE devices.devicetype = 'phone'
ORDER BY `score`
DESC LIMIT 20
The query is fine , and retourns me the data as I want, but I only want 5 items, not 20.
First of all totalCount is a property of Pagination, not ActiveDataProvider. You need to nest it inside of a pagination configurational array. But this is not how you achieve this.
Because you don't want the pagination to appear you can disable it by passing false and now limit will be taken from query (otherwise it's ignored and calculated differently, you can see this related question):
$dataProvider = new ActiveDataProvider([
'query' => Devicesdb::find()
->joinWith('score')
->where('devices.devicetype = :deviceType', [':deviceType' => $device])
->orderBy(['score' => SORT_DESC])
->limit(5),
'pagination' => false,
]);
]);
One more thing - you don't have to write params manually, see this question for explanation and better understanding. So where part of the query can be reduced to just:
->where(['devices.devicetype' => $device])
Also I recommend to refactor model name to just Device and use this to resolve duplicate names conflicts (if any) in SQL query:
->where(Device::tableName() . 'devicetype' => $device])
That way if this model related table name will changed in the future, you don't have to refactor your code.
I use ZendFramework Paginator and I have some code like this:
$defaultCount=1000;
$db = Zend_Db_Table::getDefaultAdapter();
$select = $db->select();
$select->from(array('u' => 'core_users'));
$select->join(array('ur' => 'core_users_roles'), 'u.uid = ur.uid');
$select->join(array('r' => 'core_roles'), 'r.rid = ur.rid');
$adapter=new Zend_Paginator_Adapter_DbSelect($select);
$adapter->setRowCount($db->select()->from('core_users',array(Zend_Paginator_Adapter_DbSelect::ROW_COUNT_COLUMN =>'uid')));
$paginator= new Zend_Paginator($adapter);
$paginator->setItemCountPerPage($defaultCount);
$paginator->setCurrentPageNumber($page);
but ,i can not get all of my data from DbSelect Adapter.when i remove $defaultCount , it always give me 20 total data (default ,I guess). should i use single table ?
$db->select()->from('core_users',array(Zend_Paginator_Adapter_DbSelect::ROW_COUNT_COLUMN =>'uid'))
give me wrong number. I use hard code ,works on me.
I'm using the mongoDB to store the log of user. In my real-time report, I need to count the distinct user of the table in a specific type. In the beginning, it runs fast, but it become slower when the table becomes bigger.
Here is the code I used:
$connection = new MongoClient();
$result = $collection->distinct('user', array('type' => $type, 'ctime' => array('$gte' => $start)));
$total = count($result);
$total is the total number of unique user
Can anyone suggest me how to improve the query to get the better performance?
Many thanks.
use $collection->ensureIndex(array('user' => 1)); to create index on user field.
I have a very complex setup on my tables and achieving this via any of the find() methods is not an option for me, since I would need to fix relationships between my tables and I don't have the time right now, so I'm looking for a simple fix here.
All I want to achieve is run a query like this:
SELECT MAX( id ) as max FROM MyTable WHERE another_field_id = $another_field_id
Then, I need to assign that single id to a variable for later use.
The way I have it now it returns something like [{{max: 16}}], I'm aware I may be able to do some PHP on this result set to get the single value I need, but I was hoping there was already a way to do this on CakePHP.
Assuming you have a model for your table and your are using CakePHP 2.x, do:
$result = $this->MyTable->field('id', array('1=1'), 'id DESC');
This will return a single value.
see Model::field()
This example is directly from the CakePHP documentation. it seems you can use the find method of a model to get count
$total = $this->Article->find('count');
$pending = $this->Article->find('count', array(
'conditions' => array('Article.status' => 'pending')
));
$authors = $this->Article->User->find('count');
$publishedAuthors = $this->Article->find('count', array(
'fields' => 'DISTINCT Article.user_id',
'conditions' => array('Article.status !=' => 'pending')
));
I'm writing a web app, with Codeigniter, that allows a user to enter query parameters and then spits out the results.
For pagination, I'm passing a limit and offset (pretty standard stuff) and can return results based on that, however, I'm having trouble passing back the TOTAL number or records that would have been returned without using the LIMIT and OFFSET parameters.
My question: Is it possible to pass the total row count that would have been returned by a previous query using Codeigniters AR syntax?
I've tried a few variations of the below, but have (at best) been able to return a count of ALL records in the items table (using count_all_results). I feel like I'm missing something here.
if (isset($brand)) {$this->db->where('brand', $brand);}
if (isset($color)) {$this->db->where('color', $color);}
$items = $this->db->get('items', $page, $offset)->result_array();
$rowcount = $this->db->count_all_results('items);
return array('items' => $items, 'row_count' => $rowcount);
Yes,
if (isset($brand)) {$this->db->where('brand', $brand);}
if (isset($color)) {$this->db->where('color', $color);}
$query = $this->db->get('items', $page, $offset);
$items = $query->result_array();
$rowcount = $query->num_rows();
return array('items' => $items, 'row_count' => $rowcount);