Paginate and search associated query with dynamic AJAX loading in CakePHP 3 - php

Question

I am working on an application in CakePHP 3.7.
We have 3 database tables with the following hierarchy. These tables are associated correctly according to the Table classes:
regulations
groups
filters
(The associations are shown in a previous question: CakePHP 3 - association is not defined - even though it appears to be)
I can get all of the data from all three tables as follows:
$regulations = TableRegistry::getTableLocator()->get('Regulations');
$data = $regulations->find('all', ['contain' => ['Groups' => ['Filters']]]);
$data = $data->toArray();
The filters table contains >1300 records. I'm therefore trying to build a feature which loads the data progressively via AJAX calls as the user scrolls down the page similar to what's described here: https://makitweb.com/load-content-on-page-scroll-with-jquery-and-ajax/
The problem is that I need to be able to count the total number of rows returned. However, the data for this exists in 3 tables.
If I do debug($data->count()); it will output 8 because there are 8 rows in the regulations table. Ideally I need a count of the rows it's returning in the filters table (~1300 rows) which is where most of the data exists in terms of the initial load.
The problem is further complicated because this feature allows a user to perform a search. It might be the case that a given search term exists in all 3 tables, 1 - 2 of the tables, or not at all. I don't know whether the correct way to do this is to try and count the rows returned in each table, or the rows overall?
I've read How to paginate associated records? and Filtering By Associated Data in the Cake docs.
Additional information
The issue seems to come down to how to write a query using the ORM syntax Cake provides. The following (plain MySQL) query will actually do what I want. Assuming the user has searched for "Asbestos":
SELECT
regulations.label AS regulations_label,
groups.label AS groups_label,
filters.label AS filters_label
FROM
groups
JOIN regulations
ON groups.regulation_id = regulations.id
JOIN filters
ON filters.group_id = groups.id
WHERE regulations.label LIKE '%Asbestos%'
OR groups.label LIKE '%Asbestos%'
OR filters.label LIKE '%Asbestos%'
ORDER BY
regulations.id ASC,
groups_label ASC,
filters_label ASC
LIMIT 0,50
Let's say there are 203 rows returned. The LIMIT condition means I am getting the first 50. Some js on the frontend (which isn't really relevant in terms of how it works) will make an ajax call as the user scrolls to the bottom of the page to re-run this query with a different limit (e.g. LIMIT 51, 100 for the next set of results).
The problem seems to be two fold:
If I write the query using Cake's ORM syntax the output is a nested array. Conversely if I write it in plain SQL it's returning a table which has just 3 columns with the labels that I need to display. This structure is much easier to work with in terms of outputting the required data on the page.
The second - and perhaps more important issue - is that I can't see how you could write the LIMIT condition in the ORM syntax to make this work due to the nested structure described in 1. If I added $data->limit(50) for example, it only imposes this limit on the regulations table. So essentially the search works for any associated data on the first 50 rows in regulations. As opposed to the LIMIT condition I've written in MySQL which would take into consideration the entire result set which includes the columns from all 3 tables.
To further elaborate point 2, assume the tables contain the following numbers of rows:
regulations: 150
groups: 1000
filters: 5000
If I use $data->limit(50) it would only apply to 50 rows in the regulations table. I need to apply the LIMIT the result set after searching all rows in all 3 tables for a given term.

Answer 1

Creating that query using the query builder is pretty simple, if you want joins for non-1:1 associations, then use the *JoinWith() methods instead of contain(), in this case innerJoinWith(), something like:
$query = $GroupsTable
->find()
->select([
'regulations_label' => 'Regulations.label',
'groups_label' => 'Groups.label',
'filters_label' => 'Filters.label',
])
->innerJoinWith('Regulations')
->innerJoinWith('Filters')
->where([
'OR' => [
'Regulations.label LIKE' => '%Asbestos%',
'Groups.label LIKE' => '%Asbestos%',
'Filters.label LIKE' => '%Asbestos%',
],
])
->order([
'Regulations.id' => 'ASC',
'Groups.label' => 'ASC',
'Filters.label' => 'ASC',
]);
See also
Cookbook > Database Access & ORM > Query Builder > Using innerJoinWith
Cookbook > Database Access & ORM > Query Builder > Adding Joins