I'm trying to understand the Zend Paginator and would mostly like to make sure it doesn't break my scripts.
For example, I have the following snippet which successfully loads some contacts one at a time:
$offset = 1;
//returns a paginator instance using a dbSelect;
$contacts = $ContactsMapper->fetchAll($fetchObj);
$contacts->setCurrentPageNumber($offset);
$contacts->setItemCountPerPage(1);
$allContacts = count($contacts);
while($allContacts >= $offset) {
foreach($contacts as $contact) {
//do something
}
$offset++;
$contacts->setCurrentPageNumber($offset);
$contacts->setItemCountPerPage(1);
}
However I can have hundreds of thousands of contacts in the database and matched by the SELECT I send to the paginator. Can I be sure it only loads one at a time in this example? And how does it do it, does it run a customized query with limit and offset?
From the official documentation : Zend Paginator Usage
Note
Instead of selecting every matching row of a given query, the DbSelect
adapter retrieves only the smallest amount of data necessary for
displaying the current page. Because of this, a second query is
dynamically generated to determine the total number of matching rows.
If your using Zend\Paginator\Adapter\DbSelect it will apply limit and offset to the query you're passing it, and it will just fetch the wanted records. This is done in the getItems() function of DbSelect, you could see that these lines in the source code.
You could also read this from the documentation :
This adapter does not fetch all records from the database in order
to count them. Instead, the adapter manipulates the original query to
produce a corresponding COUNT query. Paginator then executes that
COUNT query to get the number of rows. This does require an extra round-trip to the database, but this is many times faster than
fetching an entire result set and using count(), especially with
large collections of data.
Related
i have a question. In my PHP Firebase query i have the problem that it seems to only get 20 documents of my database collection.
I am getting all documents data and then push each entry in a separate array to finally sort the entries.
While everything is working so far - i only seem to get 20 entries each time the code runs on my server.
This is my code for fetching the data:
$tracksCount = 0;
$tracksList = $firestore->collection('lists/'.$listId.'/tracks');
$tracksDocuments = $tracksList->documents();
$sortedTracks = [];
foreach ($tracksDocuments as $track) {
if ($track->exists()) {
$trackData = $track->data();
array_push($sortedTracks, $trackData);
}
}
array_multisort( array_column($sortedTracks, "index"), SORT_ASC, $sortedTracks);
foreach ($sortedTracks as $track) {
// pushing fetched data for output....
$tracksCount = $tracksCount + 1;
}
This code is indeed working, i am getting all results that are expected - but only for 20 documents. (If there are fewer documents in the collection, it is getting fewer documents aswell. But if more than 20 documents, it has the upper limit for 20)
I cannot find the problem. Maybe somebody can help?
In my case i was fetching firebase collections by simply calling rest request and i was only getting 20 collection objects.
I was able to get all collections by adding ?pageSize=1000 to the query URL as below.
https://firestore.googleapis.com/v1/projects/<project-name>/databases/(default)/documents/<collection-name>?pageSize=1000
There is no hard limit as such on the maximum number of documents you can request, that would also be not unreasonable.There is actually no documented limit on the number of documents that can be retrieved, although there likely is a physical limit which mostly will depend on the memory and bandwidth of your app.
There is a maximum depth of functions calls in the security rules for Cloud Firestore.
If you use the list method of the Firestore REST API, you can set the “pageSize” parameter in the method to specify the maximum number of documents to return, and then paginate this data to be displayed in a readable format while being able to scroll through page navigate and access these lists of documents.
Also these can be retrieved ID can be passed as an array input , which is something similar you are trying to workaround with.
Check for similar examples below:
How to get all documents where a specific field exists
Is there a workaround for firestore query in limit to 10
Select every document in firestore
Get firestore document with query
Have a formatResults callback function that adds a "custom calculated" field into the entities post returned from a model query in my Cakephp. I would like to sort by this field and use this on a paginate is this possible?
So far i cannot accomplish this because the paginate limits the records fetched and therefore only records less than the paginator limit get sorted and not all the resultset...
Current code:
$owners = $this->Owners->find('all');
$owners->formatResults(function (\Cake\Collection\CollectionInterface $owners) {
$owners = $owners->map(function ($entity) {
$entity->random = rand(0,1);
return $entity;
});
return $owners->sortBy(function ($item){
return $item->random;
},SORT_DESC);
});
This works as expected:
$owners->toArray();
This does not:
$owners = $this->paginate($owners);
$owners->toArray();
Mainly because its "callback processing" only the first 10 records, i would like to process the whole resultset.
After diggin around ive found a similar topic opened by a previous user on the this link, it seems that is not possible to use pagination sort in other than the fields in the database.
As a result, i would suggest:
1 - Either alter your model logic, to accommodate your requirements by creating virtual fields or alter database schema to include this data.
2 - If the data requires further or live processing and it cannot be added or calculated in the database, perhaps programming a component that will replicate the paginate functionality on a cakephp collection would be a good option.The downside of this approach is that all records will be returned from the database which may present performance issues on large resultsets.
I'm using CakePHP 2.1.3. I have a performance problem of looping a large set of array data returned from find('all'). I want to retrieve a query result row by row to eliminate this expensive array processing. I don't want the result set of array returning from find() or query(). What I'm trying to do is like below:
$db = $this->Model->getDataSource();
$sql = 'SELECT * FROM my_table';
if($result = $db->execute($sql){
$db->resultSet($result);
while($row = $db->fetchResult()){
// do something with $row
}
}
However, I don't want to write the raw query. Is there any Cake function that just builds the query according to the association set and executes it without returning the result set?
[Edit]
I'm currently implementing the above script in controller. My model has no associations and so I don't need to use recursive = -1. It is the whole table fetching for the purpose of CSV export.
The Cake's find() has an internal array processing and the returned result set has to be looped again explicitly. I want to optimize the code by avoiding the array processing of large data twice.
Related issue: https://github.com/cakephp/cakephp/issues/6426
At first be sure, that you only fetch the data you really yreally need. Ideally you get everything you need with $this->YourModel->recursive = -1
Often performance problems arise due to many connected data.
When you have checked this I think a loop would be the best solution where you fetch the desired data in chunks via an between condition. Although I am not sure if this will help you.
Why do you want to go through the whole table? Do you perform some maintenance like e.g. filling a new field or updating a counter? Maybe you can achieve the goal better than by trying to fetch a whole table.
I am relatively new to The Zend framework having only been working with it probably 1 month. I have a query that's running way too slow. The page takes a few minutes to load. In my query, is it grabbing all the records or is it getting only the amount I need for the page?Can I apply a limit in fetching the contents from the DB?
My code is as follows.
public function init() {
$db = Zend_Registry::get('db');
$sql = 'SELECT * FROM employee ';
$result = $db->fetchAll($sql);
$page=$this->_getParam('page',1);
$paginator = Zend_Paginator::factory($result);
$paginator->setItemCountPerPage(5);
$paginator->setCurrentPageNumber($page);
$this->view->paginator=$paginator;
}
I will be greatful for any help you can give me.
Because you are supplying a Db query to your Paginator factory, Paginator will in most cases use the DbSelect Paginator adapter. This adapter will make the smallest query possible to satisfy the demands on the Paginator.
To make sure the Paginator uses the correct adapter you can pass a string as the second arg.
$paginator = Zend_Paginator::factory($result, 'DbSelect');
Excerpt from Reference:
Note: Instead of selecting every matching row of a given query, the
DbSelect and DbTableSelect adapters retrieve only the smallest amount
of data necessary for displaying the current page.
Because of this, a second query is dynamically generated to determine
the total number of matching rows. However, it is possible to directly
supply a count or count query yourself. See the setRowCount() method
in the DbSelect adapter for more information.
if you use database profiling you can see where the limit and offset queries are generated.
Note:
In most instances Zend_Db will convert SQL select queries into Zend_Db_Select objects. If you find you are in one of the rare situations where this is not the case you will either need to use the array adapter for Paginator or refactor your sql as a Zend_Db_Select object. This should be very rare.
Also it is rather unusual to see a paginator instance in the init() method. The init() will be run on every request to that controller.
I am about to use Zend_Paginator class in my project. I found examples of the class on the internet. One of them is
$sql = 'SELECT * FROM table_name ';
$result = $db->fetchAll($sql);
$page=$this->_getParam('page',1);
$paginator = Zend_Paginator::factory($result);
$paginator->setItemCountPerPage(10));
$paginator->setCurrentPageNumber($page);
$this->view->paginator=$paginator;
on the first line, it actually select all the rows from table_name. What if I have a table with 50000 rows? That would be very inefficient.
Is there any other way to use Zend Paginator?
About this problem, you might be interested by this section of the manual : 39.2.2. The DbSelect and DbTableSelect adapter, which states (quoting, emphasis mine) :
... the database adapters require a
more detailed explanation. Contrary to
popular believe, these adapters do not
fetch all records from the database in
order to count them. Instead, the
adapters manipulates the original
query to produce the corresponding
COUNT query. Paginator then executes
that COUNT query to get the number of
rows. This does require an extra
round-trip to the database, but this
is many times faster than fetching an
entire result set and using count().
Especially with large collections of
data.
(There is more to read on that page -- and there is an example that should give you more information)
The idea is that you will not fetch all data yourself anymore, but you'll tell to Zend_Paginator which Adapter it must use to access your data.
This Adapter will be specific to "Data that is fetched via an SQL query", and will know how to paginate it directly on the database side -- which means fetching only what is required, and not all data like you initialy did.
I recommend passing a Zend_Db_Select object as Zend_Paginator::factory($select); rather than a passing a result rowset. Otherwise, you're selecting the entire result set and then doing the pagination. In your current solution, if you had a million rows, you'd select all of them before getting the chunk of rows defined by the current page.