I have two relational tables of which i both need data. The first (Main) table contains about 90k rows. The secondary table contains about 200k plus rows.
I need to add some data of the secondary table to the results i get from the first table currently i do this in two parts:
First: i select the required rows from the Main table this goes super fast.
$records = getData($conn, "
SELECT id
from Main
where contains_tags is not null
and contains_mediums is not null
and contains_techniques is not null
limit 100
");
-
Then i go over each record to add the extra data from the secondary table but this takes ages (1 min for 100 records 50min - 100 min for 5k records. In the end i need to run this query for about 5-10k rows of the Main table). Is there a way to speed this process up?
foreach ($records as $r => $record) {
$records[$r]['mediums'] = getData($conn, "SELECT medium from mediums where kleding_id = ".$record['id']."");
kleding_id = ".$record['id']."");
}
This is the desired output after both queries finish:
[1] => Array
(
[id] => 15
[mediums] => Array
(
[0] => Array
(
[medium] => wol
)
[1] => Array
(
[medium] => katoen
)
)
)
So my question is how to run this query structure efficiently in as little as time as possible.
the Main table look like this:
the Secondary table looks like this:
If anything is unclear let me know so i can clarify.
If you need improve performance
be sure you have a proper index table mediums column kleding_id
CREATE INDEX my_index ON mediums (kleding_id);
remember that limit (for not recent db version) normally work on the result ..a and not break after the first 100 is reached
You can do it in a single query. But it's for mysql Verison 5.7 & above.
SELECT id, GROUP_CONCAT(json_object('medium', mediums.medium))
FROM Main
LEFT JOIN mediums ON Main.id = mediums.kleding_id
WHERE contains_tags IS NOT NULL AND contains_mediums IS NOT NULL AND contains_techniques IS NOT NULL
GROUP BY Main.id
LIMIT 100
Also make sure you have a index on kleding_id.
There is no need to iterate over the first result and call the 2nd table multiple times. This will always be slow.
Try inner join to aggregate data from both tables in one sql statement.
SELECT
Main.id,
mediums.medium
from
Main
inner join mediums on
(
Main.id = mediums.kleding_id
)
where
contains_tags is not null
and contains_mediums is not null
and contains_techniques is not null limit 100;
Related
I'm having trouble working in Redbean PHP with querying a table with an array of json objects in a single field, and producing a report on it.
I want to get a report with subtotals of all staff with notes by the category. I know this should be easy/obvious but I'm just not getting it properly.
I have a database, with:
table clients
with columns:(int) client_id, (string) client_name, (array of json) notes
notes is an array of json with
(int) note_id, (int) note_category_id, (int) staff_id, (string) description, (memo) content, (date) note_date
table staff with columns (int) sid, (string) sname
table categories with columns (int) cat_id, (string) cat_name
So in pseudocode (since I'm still trying to figure this all out)
I need to run a query like: (with parameters in brackets)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between [startdate] and [enddate],
categories.cat_name IN [chosencateg], staff.sname IN [pickednames])
Orderby sname Asc, cat_name Asc, note_date Desc ');
report output format:
Filters used: [picked filter choices if any]
-----------
[sname]
-- note category: [cat_name] 1
[note_date] 1 [description] 1 [content] 1
[note_date] 2 [description] 2 [content] 2
note category 1 subtotal
-- note category: [cat_name] 2
[note_date] 3 [description] 3 [content] 3
[note_date] 4 [description] 4 [content] 4
note category 2 subtotal
staff subtotal
[sname] 2 ...
I'm asking a fairly generic one because I'll have to work with a number of similar tables, and maybe seeing a query template will help my understanding.
Thanks for any help.
redbean is fantastic and - getAll is just scratching the surface and truly isn't working with redbean at all really... Read up on it here:
Here's a query template to get you started:
Query Template:
1)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between :startdate and :enddate,
categories.cat_name IN (:chosencateg), staff.sname IN (:pickednames))
Orderby sname Asc, cat_name Asc, note_date Desc ');
You could also simply use:
2)
R::getAll('Select * from Join (staff, categories, clients)
On (staff.sid=clients.services.staff_id, categories.cat_id=clients.services.note_category_id)
Where (clients.services.note_date Between ? and ?,
categories.cat_name IN (?), staff.sname IN (?))
Orderby sname Asc, cat_name Asc, note_date Desc ');
The only difference is that query template 1 uses named parameters (so it's going to look to the array of params that you pass it to contain an associative array with parameters named in the same way as they are in the query). While template 2 requires simply an array of parameters with the indexes lined up with the order in which the ? marks appear in your query.
Anyway... the query should return an associative array of your columns representing rows. a var_dump would look something like this:
Array
(
[client_id] => 1,
[client_name] => "joe",
[noes] => "[
{note_id=1
,note_category_id=1
,staff_id=1
,description=blah blah
,content=blah blah blah blah
,content=some content for this note
,note_date=12/06/2018
}
]"
[sid] => 100,
[sname] => "some staff name"
[cat_id] => 100
[cat_name] => "some category name"
)
Notice how the notes field has just come out as a string (I know the json above is not properly formed json, I'm just trying to show an example).
I assume that what you want is to have that string converted into an array so you can work with it as if it were data and not a string. So the below should get you started with that:
Once you have it out of the database you should be able to access it like this:
$result = R::getAll($query,
['startdate'=> $mystartDate
,'enddate' => $myEndDate
,'chosencateg'=>$myChosenCategory
,'pickednames'=>$myPickedNames
]);
// this would output the json string to your screen
echo $result['notes'];
but it seems like you want to work with the json as if it were part of your data - so... you would need to decode it first.
// decode my notes field:
foreach(array_key($result) as $key) {
/* you are working with a multidimensional array in this loop
, use $key to access the row index. Each row index
will contain named column indexes that are column names from the database
*/
$result[$key]['decoded_notes'] = json_decode($result[$key]['notes'],true);
}
// I now have a new column in each row index, containing 'notes'
as another associative array
// the statement below now results in an array to string conversion error:
echo $result[someIndexNumber]['decoded_notes'];
So, I decided I would want this in MySQL (5.7) so as to use its capabilities. To do this I used string manipulation. MySQL 8 adds json_table functions which would have been nice to have.
I converted each array of JSON notes into lines of 'INSERT INTO temptable' to convert the array list into temptable rows,
one JSON object per row, adding the client_id to each object, then
EXECUTEing those statements.
SET #allnotes = (
SELECT json_arrayagg(REPLACE(`notes`,'{', CONCAT('{"id_client": ', id_client, ', '))) /* add id_client to each note object */
FROM clients
WHERE `notes` != '' AND `notes` != '[]' ); /* no empty note cases */
SET #allnotes = REPLACE(REPLACE(#allnotes ,'"[',''),']"','' ); /* flatten out outer array of each note */
SET #allnotes = REPLACE(REPLACE(#allnotes ,'{','("{'),'}','}")' ); /* INSERT INTO string formatting for the objects */
DROP TEMPORARY TABLE IF EXISTS jsonTemporary;
CREATE TEMPORARY TABLE IF NOT EXISTS jsonTemporary (anote json);
SET #allnotes = REPLACE(REPLACE(#allnotes,'[','INSERT INTO jsonTemporary (anote) VALUES '),']',';');
PREPARE astatement FROM #allnotes;
EXECUTE astatement;
/* totals */
SELECT concat(staff.last_name,", ",staff.first_name) AS sname,
categories.name AS cat_name,
count(anote->'$.id_client') AS cat_total,
FROM jsonTemporary
JOIN categories ON cast(anote->'$.note_category_id' as unsigned)=categories.id
JOIN clients ON clients.id_client=anote->'$.id_client'
JOIN staff ON staff.id=anote->'$.staff_id'
WHERE anote->'$.note_date' >= "2018-10-01" AND anote->'$.note_date' <= "2018-12-31"
GROUP BY sname, cat_name;
/* all notes */
SELECT concat(staff.last_name,", ",staff.first_name) AS sname,
categories.name AS cat_name,
anote->'$.note_date' AS n_date,
anote->'$.description' AS description,
anote->'$.content' AS content,
FROM jsonTemporary
JOIN categories ON cast(anote->'$.note_category_id' as unsigned)=categories.id
JOIN clients ON clients.id_client=anote->'$.id_client'
JOIN staff ON staff.id=anote->'$.staff_id'
WHERE anote->'$.note_date' >= "2018-10-01" AND anote->'$.note_date' <= "2018-12-31"
GROUP BY sname, cat_name;
DROP TEMPORARY TABLE IF EXISTS jsonTemporary;
I have the following code :
$foo = $bdd->prepare($qry);
$foo->execute();
$result = $foo->fetchAll();
In $qry, i have a SELECT with a JOIN between tables beatles b et status s (on the columns status_id), such that the result is the following :
Code :
b.id b.firstname b.lastname b.status_id s.status_id s.status
0 John Lennon 0 0 Mort
1 Paul McCartney 1 1 Vivant
2 Ringo Starr 1 1 Vivant
3 George Harrison 0 0 Mort
(The first line is the columns' names in the tables, it isn't inside the result of the query)
I want to select in php only the s.status of ringo starr, for instance. How can I do that ?
Thanks
If you want to retrieve the value of status for Ringo Starr, but need the values of the other fields within that page, then (assuming the array has come back indexed by id), you could access it with $result[2]['status']. Otherwise, change your select statement to just select the values of id and status.
As a side note, in your SELECT statement you'll need to give aliases potentially for the b.status_id and s.status_id; when you join, the column names are taken and the table names basically ignored in terms of what is returned - so something like b.status_id AS b_status_id, s.status_id AS s_status_id would make sure you got the correct values for both of these. You may have done this already of course - it just looked as though you might have not, judging from the column names you gave.
I am not sure if you want to just pick one column from the existing result or issue a query that will get you only what you want. But a better defined query sound the most sensible
SELECT
[ your joins etc ]
WHERE b.firstname = 'Ringo'
AND b.lastname = 'Starr'
Or if you want to just get the Ringo Starr entry from your existing result
foreach ( $resultats as $result ) {
if ( $result['firstname'] == 'Ringo' && $result['lastname'] == 'Starr' ) {
echo 'Got him';
}
}
add WHERE clause with condition that satisfy your needs
alter fields clause leaving only one you need
use fetchColumn() instead of fetchAll()
You're set.
I'm trying to sort an array in to a three-deep array. This is my current query:
SELECT * FROM question
INNER JOIN category ON question.category_id = category.id
INNER JOIN difficulty ON question.difficulty_id = difficulty.id
Expected result is something like:
array(
'1' => array( // category id 1
'1' => array( // difficulty id 1
'1' => array('...'), // question id 1
'2' => array('...') // question id 2
),
'2' => array(
'3' => array('...'),
'4' => array('...')
)
)
)
I did have the following:
foreach($categories as $category) {
foreach($difficulties as $difficulty) {
foreach($questions as $question) {
if ($question['category_id'] == $category['id'] && $question['difficulty_id'] == $difficulty['id']) {
$feed[$category['id']][$difficulty['id']][$question['id']] = $question;
}
}
}
}
But there will be 10,000+ questions and performance will be bad so is there a way I can do this with one query and fewer loops?
Basically you could just return your query and order by the ids like so:
Category_ID Difficulty_ID Question_ID
0 0 0
0 0 1
1 0 2
1 3 3
1 3 4
2 0 5
2 1 6
Then parse everything in a while:
each time the category_ID changes add a new category with empty difficulty and reset previous difficulty
each time the difficulty changes add new difficulty to category with empty question
each time add the question to current difficulty.
To store this structure performantly in local storage:
define a unique delimiter (note: IE doesn't support control characters, this also means you can't store binary data without encoding it before, e.g. base64)
load each row of each table like this:
key: unique table prefix + id
value: columns (delimited with the delimiter defined before)
The easiest way to return a whole table at once is to define a second delimiter and then have some slightly ugly query in the form of:
SELECT id||delimiter||col1||delimiter||...||colN FROM ...
And then put it all together with a list aggregation using the second delimiter (group_concat() in mysql).
Sometimes you need maps (for N to M relations or also if you want to search questions by difficulty or category), but because each question only has one category and difficulty you are already done.
Alternative
If the data is not too big and doesn't change after login, then you can just use the application cache and echo your stuff in script tags.
I have two tables: Workplans and Progress. Here is the Workplan table.
Workplan Table:
id
division_name
division_chief
Progress Table:
id
score
workplan_id foreign_key
Here workplan_id is foreign key of workplan table.
Now I want to get average from progress table grouping by the division_name. But I am really messed up with my query. What is I have done:
$report = DB::table('workplans')
->join('progress','workplans.id','=','progress.workplan_id')
->avg('progress.score')
->groupBy('workplans.division_name')
->get();
dd($report);
The SQL you want is something like this...
SELECT workplans.division_name, AVG(progress.score) as average_score
FROM `workplans`
LEFT JOIN `progress` ON progress.workplan_id = workplans.id
GROUP BY workplans.id
With the laravel query builder...
$report = DB::table ('workplans')
->join ('progress', 'progress.workplan_id', '=', 'workplans.id')
->select ('workplans.division_name', DB::raw ('AVG(progress.score) as average_score'))
->groupBy ('workplans.id')
->get ();
So for division 1 with scores 5 and 15, division 2 with scores 15 and 25 this returns...
Array
(
[0] => stdClass Object
(
[division_name] => Division 1
[average_score] => 10
)
[1] => stdClass Object
(
[division_name] => Division 2
[average_score] => 20
)
)
The problem with your query is that the avg aggregate function returns the average value for all columns on the table, not a reference back to the query so that you can continue to chain additional functions. The solution to this is to manually specify the average using the MySQL AVG function using the DB::raw method.
Try to do it via raw select:
$report = DB::table('workplans')
->join('progress','workplans.id','=','progress.activity_id')
->select(DB::raw('avg(progress.score) as avg_progress_score'))
->groupBy('workplans.division_name')
->get();
I'm not sure how to write it using Laravel's query builder, but a static MySQL query might be easier:
$report = DB::select("SELECT AVG(progress.score) AS 'avg' FROM workplans JOIN progress ON workplans.id=progress.activity_id GROUP BY workplans.division_name;");
I need help building a SQL query which searches for an user_id (let's say user_id = 5), but this user_id has multiple row entries, but I need to update each row differently though.
I have an array with data which I want to assign to that user_id. Here is a table example:
id user_id amount txn_id
1 5 10 foo_unique
2 5 5 bar_unique
3 7 15 xyz_unique
4 5 10 123_unique
My array would look something like this:
Array (
[0] => 14
[1] => 6
[2] => 9
)
As you see I have three array values and three rows for user_id 5, now I want to add each value of that array, to the corresponding user_id (5). Note that I have an UNIQUE column named txn_id
After updating the table, it would look like this:
id user_id amount txn_id
1 5 14 foo_unique
2 5 5 bar_unique
3 7 6 xyz_unique
4 5 9 123_unique
Any ideas how I could achieve this?
Thanks.
P.S: I cannot use SQL CASE on this issue.
If you want to update one row with a value, you have to be able to have a unique condition for that row.
Without adding some extra field, or condition to uniquely identify a row, you are out of luck.
You can use the following query:
UPDATE MyTable
SET
amount =
CASE id
WHEN 1 THEN 14
WHEN 2 THEN 6
WHEN 3 THEN 9
ELSE amount -- just in case user_id 5 has records not covered above.
END
WHERE
user_id = 5
The problem is, there is nothing on your array that says which array entry belongs to which database row. If you want to rely just on the orders (of the array indices, and the database ids,), you have to SELECT all rows for the user first, then loop and update a row at a time. Something like this on PHP (and PDO):
$values = array(14, 6, 9);
$dbh = new PDO('mysql:host=localhost;dbname=yourdbname', 'username', 'password');
$selectQuery = 'SELECT id FROM yourtable WHERE user_id = 5 ORDER BY id';
foreach($dbh->query($selectQuery) as $row) {
$id = $row['id'];
$sth = $dbh->prepare("UPDATE yourtable SET amount = :val WHERE id = :id");
$sth->execute(array(
'val' => array_unshift($values),
'id' => $id
));
}
(Code above assumes the number of values on your arrays matches the number of database rows for user_id = 5).
Looking at the problem you only need to have SQL statements.
First is to create two tables:
CREATE TABLE users
user_id int(11) PRIMARY KEY NOT NULL,
username varchar(25),
) Engine = InnoDB;
CREATE table userbalances (
balance_id int(11) PRIMARY KEY NOT NULL,
user_id int(11),
amount decimal(10,2),
CONSTRAINT userid_fk FOREIGN KEY userid_fk(user_id) REFERENCES users(userid)
) Engine = InnoDB;
Then, after you have INSERT(ed) all of the users and userbalances inside the table, all you need to do is create an UPDATE statement to update the amount:
UPDATE userbalances SET amount=? WHERE user_id=? AND balanceid=?
The update should be in a loop. And you should use mysql_query() function to update it with the assigned arrays that you have.
Note:
Both userid and balanceid should be in the WHERE clause of your UPDATE statement because you have multiple transactions inside the userbalances table. Otherwise, it's going to change all of the balances of all the transactions of that user.
Problem update:
If you don't use any SQL Cases on this problem, then your problem is just PHP. You need to find out what is the function for UPDATE(ing) - to update the amounts. As long as your tables are all prepopulated. There might be a function on the program you use to update it.