I'm trying to sort an array in to a three-deep array. This is my current query:
SELECT * FROM question
INNER JOIN category ON question.category_id = category.id
INNER JOIN difficulty ON question.difficulty_id = difficulty.id
Expected result is something like:
array(
'1' => array( // category id 1
'1' => array( // difficulty id 1
'1' => array('...'), // question id 1
'2' => array('...') // question id 2
),
'2' => array(
'3' => array('...'),
'4' => array('...')
)
)
)
I did have the following:
foreach($categories as $category) {
foreach($difficulties as $difficulty) {
foreach($questions as $question) {
if ($question['category_id'] == $category['id'] && $question['difficulty_id'] == $difficulty['id']) {
$feed[$category['id']][$difficulty['id']][$question['id']] = $question;
}
}
}
}
But there will be 10,000+ questions and performance will be bad so is there a way I can do this with one query and fewer loops?
Basically you could just return your query and order by the ids like so:
Category_ID Difficulty_ID Question_ID
0 0 0
0 0 1
1 0 2
1 3 3
1 3 4
2 0 5
2 1 6
Then parse everything in a while:
each time the category_ID changes add a new category with empty difficulty and reset previous difficulty
each time the difficulty changes add new difficulty to category with empty question
each time add the question to current difficulty.
To store this structure performantly in local storage:
define a unique delimiter (note: IE doesn't support control characters, this also means you can't store binary data without encoding it before, e.g. base64)
load each row of each table like this:
key: unique table prefix + id
value: columns (delimited with the delimiter defined before)
The easiest way to return a whole table at once is to define a second delimiter and then have some slightly ugly query in the form of:
SELECT id||delimiter||col1||delimiter||...||colN FROM ...
And then put it all together with a list aggregation using the second delimiter (group_concat() in mysql).
Sometimes you need maps (for N to M relations or also if you want to search questions by difficulty or category), but because each question only has one category and difficulty you are already done.
Alternative
If the data is not too big and doesn't change after login, then you can just use the application cache and echo your stuff in script tags.
Related
I have a table (tbl_operations) with rows of where the id column values may be comma-delimited. I want to get the count of each OpId for each month. I am trying to accomplish this through pure sql, but without success.
from this view
OpId
OpDate
3
2022-01-03
5,3
2022-01-15
4
2022-01-27
5
2022-02-01
7
2022-02-09
3,2
2022-01-16
to this
OpId
count
Month
2
1
01
3
3
01
4
1
01
5
1
01
5
1
02
7
1
02
I am stuck here. Can someone enlighten me on how to do this with sql? If not, maybe use php to display the result?
SELECT tbl_operations.OpId,
tbl_operations.OpDate ,
COUNT(tbl_operations.OpId) AS `count`
FROM tbl_operations
WHERE MONTH(OpDate)=1
GROUP BY tbl_operations.OpId
Here’s a quick example. The first part just creates an array of arrays which simulates what you’d get from the database.
The gist is that $counts is an array with a unique OpID for a keys. The values for those arrays are sub-arrays with keys of the month and values of how many times they are found.
Display should just be a simple loop again, however you might want to sort this.
$rows = [
['3', '2022-01-03'],
['5,3', '2022-01-15'],
['4', '2022-01-27'],
['5', '2022-02-01'],
['7', '2022-02-09'],
['3,2', '2022-01-16'],
];
$counts = [];
foreach($rows as $row){
$ids = explode(',', $row[0]);
$month = date('m', strtotime($row[1]));
foreach($ids as $id){
if(!array_key_exists($id, $counts)){
$counts[$id] = [];
}
if(!array_key_exists($month, $counts[$id])){
$counts[$id][$month] = 0;
}
$counts[$id][$month]++;
}
}
Demo here: https://3v4l.org/mVaBB
edit
From #mickmackusa, you can shorten the inner loop by using isset:
if(!isset($counts[$id][$month])){
$counts[$id][$month] = 0;
}
See their comment for a demo link
If you're going to query the data in PHP, you might as well return a better result to work with in the first place:
SQL
SELECT GROUP_CONCAT(OpId), MONTH(OpDate)
FROM tbl_operations
GROUP BY MONTH(OpDate)
PHP
// Result from MySQL query
$rows = [
['3,5,3,4,3,2', 1],
['5,7', 2]
];
And you can perform a count of those grouped results like this:
$results = [];
foreach ($rows as $row) {
$counts = array_count_values(explode(',', $row[0]));
$results[$row[1]] = $counts;
}
Result
Array
(
[1] => Array
(
[3] => 3
[5] => 1
[4] => 1
[2] => 1
)
[2] => Array
(
[5] => 1
[7] => 1
)
)
What you really want to do though is normalise your data, then you can do this easily in SQL alone.
If you are using at least MYSQL8 and you are not going to normalize your table design, then you can actually use the following CTE query to split, group, format, and sort your result set (no PHP processing).
This approach makes recursive calls on the denormalized table and progressively isolates the rightmost id from comma-delimited values and generates new rows for the individual id values. The recursion continues until there are no commas left.
This solution is built on top of the basic technique demonstrated here.
SQL: (Demo)
WITH RECURSIVE norm AS (
SELECT OpId,
OpDate
FROM tbl_operations
UNION ALL
SELECT REGEXP_REPLACE(OpId, '^[^,]*,', '') AS OpId,
OpDate
FROM norm
WHERE OpId LIKE '%,%'
)
SELECT Id,
Mo,
COUNT(*) AS Cnt
FROM (
SELECT REGEXP_REPLACE(norm.OpId, ',.*', '') AS Id,
MONTH(norm.OpDate) AS Mo
FROM norm
) formatted
GROUP BY formatted.Id,
formatted.Mo
Result Set:
Id
Mo
Cnt
2
1
1
3
1
3
4
1
1
5
1
1
5
2
1
7
2
1
That said, this is a lot of unnecessary voodoo mumbo jumbo for a task that is mega-easy once you've normalized your table --- just normalize it A.S.A.P.
I'm having trouble working in Redbean PHP with querying a table with an array of json objects in a single field, and producing a report on it.
I want to get a report with subtotals of all staff with notes by the category. I know this should be easy/obvious but I'm just not getting it properly.
I have a database, with:
table clients
with columns:(int) client_id, (string) client_name, (array of json) notes
notes is an array of json with
(int) note_id, (int) note_category_id, (int) staff_id, (string) description, (memo) content, (date) note_date
table staff with columns (int) sid, (string) sname
table categories with columns (int) cat_id, (string) cat_name
So in pseudocode (since I'm still trying to figure this all out)
I need to run a query like: (with parameters in brackets)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between [startdate] and [enddate],
categories.cat_name IN [chosencateg], staff.sname IN [pickednames])
Orderby sname Asc, cat_name Asc, note_date Desc ');
report output format:
Filters used: [picked filter choices if any]
-----------
[sname]
-- note category: [cat_name] 1
[note_date] 1 [description] 1 [content] 1
[note_date] 2 [description] 2 [content] 2
note category 1 subtotal
-- note category: [cat_name] 2
[note_date] 3 [description] 3 [content] 3
[note_date] 4 [description] 4 [content] 4
note category 2 subtotal
staff subtotal
[sname] 2 ...
I'm asking a fairly generic one because I'll have to work with a number of similar tables, and maybe seeing a query template will help my understanding.
Thanks for any help.
redbean is fantastic and - getAll is just scratching the surface and truly isn't working with redbean at all really... Read up on it here:
Here's a query template to get you started:
Query Template:
1)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between :startdate and :enddate,
categories.cat_name IN (:chosencateg), staff.sname IN (:pickednames))
Orderby sname Asc, cat_name Asc, note_date Desc ');
You could also simply use:
2)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between ? and ?,
categories.cat_name IN (?), staff.sname IN (?))
Orderby sname Asc, cat_name Asc, note_date Desc ');
The only difference is that query template 1 uses named parameters (so it's going to look to the array of params that you pass it to contain an associative array with parameters named in the same way as they are in the query). While template 2 requires simply an array of parameters with the indexes lined up with the order in which the ? marks appear in your query.
Anyway... the query should return an associative array of your columns representing rows. a var_dump would look something like this:
Array
(
[client_id] => 1,
[client_name] => "joe",
[noes] => "[
{note_id=1
,note_category_id=1
,staff_id=1
,description=blah blah
,content=blah blah blah blah
,content=some content for this note
,note_date=12/06/2018
}
]"
[sid] => 100,
[sname] => "some staff name"
[cat_id] => 100
[cat_name] => "some category name"
)
Notice how the notes field has just come out as a string (I know the json above is not properly formed json, I'm just trying to show an example).
I assume that what you want is to have that string converted into an array so you can work with it as if it were data and not a string. So the below should get you started with that:
Once you have it out of the database you should be able to access it like this:
$result = R::getAll($query,
['startdate'=> $mystartDate
,'enddate' => $myEndDate
,'chosencateg'=>$myChosenCategory
,'pickednames'=>$myPickedNames
]);
// this would output the json string to your screen
echo $result['notes'];
but it seems like you want to work with the json as if it were part of your data - so... you would need to decode it first.
// decode my notes field:
foreach(array_key($result) as $key) {
/* you are working with a multidimensional array in this loop
, use $key to access the row index. Each row index
will contain named column indexes that are column names from the database
*/
$result[$key]['decoded_notes'] = json_decode($result[$key]['notes'],true);
}
// I now have a new column in each row index, containing 'notes'
as another associative array
// the statement below now results in an array to string conversion error:
echo $result[someIndexNumber]['decoded_notes'];
So, I decided I would want this in MySQL (5.7) so as to use its capabilities. To do this I used string manipulation. MySQL 8 adds json_table functions which would have been nice to have.
I converted each array of JSON notes into lines of 'INSERT INTO temptable' to convert the array list into temptable rows,
one JSON object per row, adding the client_id to each object, then
EXECUTEing those statements.
SET #allnotes = (
SELECT json_arrayagg(REPLACE(`notes`,'{', CONCAT('{"id_client": ', id_client, ', '))) /* add id_client to each note object */
FROM clients
WHERE `notes` != '' AND `notes` != '[]' ); /* no empty note cases */
SET #allnotes = REPLACE(REPLACE(#allnotes ,'"[',''),']"','' ); /* flatten out outer array of each note */
SET #allnotes = REPLACE(REPLACE(#allnotes ,'{','("{'),'}','}")' ); /* INSERT INTO string formatting for the objects */
DROP TEMPORARY TABLE IF EXISTS jsonTemporary;
CREATE TEMPORARY TABLE IF NOT EXISTS jsonTemporary (anote json);
SET #allnotes = REPLACE(REPLACE(#allnotes,'[','INSERT INTO jsonTemporary (anote) VALUES '),']',';');
PREPARE astatement FROM #allnotes;
EXECUTE astatement;
/* totals */
SELECT concat(staff.last_name,", ",staff.first_name) AS sname,
categories.name AS cat_name,
count(anote->'$.id_client') AS cat_total,
FROM jsonTemporary
JOIN categories ON cast(anote->'$.note_category_id' as unsigned)=categories.id
JOIN clients ON clients.id_client=anote->'$.id_client'
JOIN staff ON staff.id=anote->'$.staff_id'
WHERE anote->'$.note_date' >= "2018-10-01" AND anote->'$.note_date' <= "2018-12-31"
GROUP BY sname, cat_name;
/* all notes */
SELECT concat(staff.last_name,", ",staff.first_name) AS sname,
categories.name AS cat_name,
anote->'$.note_date' AS n_date,
anote->'$.description' AS description,
anote->'$.content' AS content,
FROM jsonTemporary
JOIN categories ON cast(anote->'$.note_category_id' as unsigned)=categories.id
JOIN clients ON clients.id_client=anote->'$.id_client'
JOIN staff ON staff.id=anote->'$.staff_id'
WHERE anote->'$.note_date' >= "2018-10-01" AND anote->'$.note_date' <= "2018-12-31"
GROUP BY sname, cat_name;
DROP TEMPORARY TABLE IF EXISTS jsonTemporary;
I have two relational tables of which i both need data. The first (Main) table contains about 90k rows. The secondary table contains about 200k plus rows.
I need to add some data of the secondary table to the results i get from the first table currently i do this in two parts:
First: i select the required rows from the Main table this goes super fast.
$records = getData($conn, "
SELECT id
from Main
where contains_tags is not null
and contains_mediums is not null
and contains_techniques is not null
limit 100
");
-
Then i go over each record to add the extra data from the secondary table but this takes ages (1 min for 100 records 50min - 100 min for 5k records. In the end i need to run this query for about 5-10k rows of the Main table). Is there a way to speed this process up?
foreach ($records as $r => $record) {
$records[$r]['mediums'] = getData($conn, "SELECT medium from mediums where kleding_id = ".$record['id']."");
kleding_id = ".$record['id']."");
}
This is the desired output after both queries finish:
[1] => Array
(
[id] => 15
[mediums] => Array
(
[0] => Array
(
[medium] => wol
)
[1] => Array
(
[medium] => katoen
)
)
)
So my question is how to run this query structure efficiently in as little as time as possible.
the Main table look like this:
the Secondary table looks like this:
If anything is unclear let me know so i can clarify.
If you need improve performance
be sure you have a proper index table mediums column kleding_id
CREATE INDEX my_index ON mediums (kleding_id);
remember that limit (for not recent db version) normally work on the result ..a and not break after the first 100 is reached
You can do it in a single query. But it's for mysql Verison 5.7 & above.
SELECT id, GROUP_CONCAT(json_object('medium', mediums.medium))
FROM Main
LEFT JOIN mediums ON Main.id = mediums.kleding_id
WHERE contains_tags IS NOT NULL AND contains_mediums IS NOT NULL AND contains_techniques IS NOT NULL
GROUP BY Main.id
LIMIT 100
Also make sure you have a index on kleding_id.
There is no need to iterate over the first result and call the 2nd table multiple times. This will always be slow.
Try inner join to aggregate data from both tables in one sql statement.
SELECT
Main.id,
mediums.medium
from
Main
inner join mediums on
(
Main.id = mediums.kleding_id
)
where
contains_tags is not null
and contains_mediums is not null
and contains_techniques is not null limit 100;
Table 1 - User:
ID Name
1 Jonh
2 Mark
3 King
Table 2 - Book:
ID user_idstatus ...
1 1 1 ...
2 1 1 ...
3 1 1 ...
4 2 1 ...
5 1 0 ...
6 1 0 ...
Code:
$query = User::find();
$query->joinWith('books');
$query->select(['user.*', 'COUNT(book.id) AS booksCount']);
$query->andWhere(['book.status' => 1]); // Problem Here!
$query->groupBy(['user.id']);
$query->orderBy(['booksCount' => SORT_DESC]);
Problem:
The query is working properly, but it's not returning the user with id = 3.
If I remove the line $query->andWhere(['book.status' => 1]); it works fine and return all users.
What should I change to list all users, even those who do not have a related book with status = 1?
I found the answer:
$query = User::find();
$query->joinWith(['books' => function ($subquery) {
$subquery->onCondition(['book.status' => 1]);
}]);
$query->select(['user.*', 'COUNT(book.id) AS booksCount']);
$query->groupBy(['user.id']);
$query->orderBy(['booksCount' => SORT_DESC]);
Instead of using COUNT(book.id), if the status of the book is either 0 or 1, you can use SUM(book.status) to get the number of books the user has. Then you can remove your WHERE book.status = 1 clause, and it will return all the users with the number of books they have, even in user 3's case where they have 0 books.
The Problem
The real problem is in your where clause. Because WHERE is processed before grouping and user 3 doesn't have any rows where book.status = 1, then the user has no rows which are included in the base query. Therefor the user isn't present during/after the grouping.
If you want a pretty good idea of a catch-all case where you can count rows based on a condition, using COUNT(CASE WHEN book.status IS NULL THEN NULL ELSE NULLIF(0,book.status) END) will also give you the result you're looking for. Because COUNT() will not count rows where the expression is NULL, this would allow the book.status to be -1, 1, 2, and any other number as long as it isn't 0 (or NULL in user 3's case), and still be included in the count.
I have a parent category that holds all Cars names, denoted by parent_name in table "parent". For each of these parents, there could be any number of car models & they all go in table called "model". Each of these models can have any number of images & refereced via the model_id as the Foreign Key. My task is to show all the Parent Name only once (as in a group) and at the same time, list all the models under that Parent with just 1 corresponding image. The parent_name should not be shown more than once.
MY EXPERIMENTS:
I basically tried to write 2 queries. One was to left join "parent" table on "models" & use GROUP BY parent_id and then in the while loop, write another query to fetch only 1 image using by joining the models & images tables by using model_id field. But doing this lists only 1 Model, even though there are multiple models. So I tried to use GROUP BY parent_id, model_id. Using this does show all the models but at the same time, also repeats showing the parent_name & I need the parent_name to show only once throughout the page. You can say that I am trying to GROUP the model_name under the parent & show all the models under a single parent and I am showing only 1 image of the model. If somehow I can avoid showing the parent_name multiple times, the issue would be solved.
Following are my table schemas:
//Table parent
parent_id parent_name
1 Par1
2 Par2
//Table model
model_id parent_id model_name
1 1 Model1
2 2 Model2
3 1 Model3
4 1 Model4
5 2 Model5
//Table model_images
image_id model_id
1 1
2 1
3 1
4 2
5 3
6 3
DESIRED OUTPUT:
Par1 ---> This is the parent. Needs to be shown only once.
Model1 --> This is a model. List all models that belong to this parent.
image_id 1 -> Show only 1 image of the model (model may have multiple images but I need just one)
Model3 --> This is a model.
image_id 5 -> Show only 1 image of the model
Model4 --> This is a model.
No Image -> Note that no image exists for this model. So we show "No Image" text.
------------------------------------------------------------
Par2 ---> This is the parent. Needs to be shown only once.
Model2 --> This is a model.
image_id 4 -> Show only 1 image of the model
Model5 --> This is a model.
No Image -> Note that no image exists for this model. So we show "No Image" text.
I need the PHP & mySQL code to achieve the above. All help in resolving the issue is appreciated.
Thank you very much.
EDIT 1:
Sorry, I forgot to add this. I am non-object oriented programmer. So I would really be thankful if you can avoid object oriented code in your solution and show me the same in a non-oops way. Thanks.
You might do it in one query and than combine it to an associative array:
$query = ' SELECT *
FROM parent AS p
LEFT JOIN model AS m
ON p.id = m.parent_id
LEFT JOIN model_images AS m_i
ON m.model_id = m_i.model_id';
$array = array();
if($mysli->query($quer)){
while($row = $result->fetch_assoc()){
$array[$row['parent_name']][$row['model_id']] = $row;
}
}
You will than have an associative array with the parent name as the key of the array. You can then use a for loop to print the key only once (with $i = 0) but the rest value by value.
Is that clear enough?
EDIT: Your array than might look like this:
Array(
'Par 1' =>
Array(
[0] => Array(
'parent_id' => 1,
'parent_name' => 'Par 1',
'model_id' => 1,
'model_name' => 'Model 1',
'image_id',
),
[1] => Array(...)
),
'Par 2' => Array(...)
)
So to print out you need two loops. One for the parents (and there names) and one for their childs (models in this case).
foreach($array as $par_name => $models){
echo 'Parent name: '.$par_name.'<br />';
echo 'Model ID: '.$models[0]['model_id'].', Model Name: '.$models[0]['name']; // replace with your desired output
}
Now an idea of how it works? An sure as Artefacto said, you can use procedural functions if you don't like OOP functions.