This question already has answers here:
Is there a way to fetch associative array grouped by the values of a specified column with PDO?
(8 answers)
Closed 1 year ago.
I want to send the overdue tasks that are assigned to a specific employee as an email summary. It is possible that multiple todos are assigned to the same employee. So multiple todos can be assigned to the same employee AND are overdue. That's where the problem starts...
So what I did, is grabbing all the overdue tasks from the database and then I grabbed the assigned employees to the tasks. I created an array that consists of the todoID, the employeeID and the employeeEmail. Now, is there a better way to do this and if not, how can I group the rows by email address?
The end result should be an array that shows every overdue todo that's assigned to one employee.
// Get all Todos that are not archived
$sql = "SELECT * FROM todo WHERE archiv = 0";
$abfrage = $db->prepare($sql);
$abfrage->execute();
$overdue_array = array();
// Now we get everything that's overdue from the Database
while ($row = $abfrage->fetch()) {
if ($row["status"] !== 3) {
if ($row["archiv"] !== 1) {
if ($row["durchfuehrung"]) {
if (strtotime($row["durchfuehrung"]) < strtotime(date("Y-m-d"))) {
// Here we now get the email from the assiged employee to the todo
$sql2 = "SELECT email FROM mitarbeiter WHERE mitarbeiterID = :mitarbeiterFID";
$abfrage2 = $db->prepare($sql2);
$abfrage2->bindParam("mitarbeiterFID", $row["mitarbeiterFID"]);
$abfrage2->execute();
while ($row2 = $abfrage2->fetch()) {
$overdue_array[] = array("todoID" => $row["todoID"], "mitarbeiterID" => $row["mitarbeiterFID"], "mitarbeiterEmail" => $row2["email"]);
}
}
}
}
}
The result is the following:
Let's pretty up your scripting with some best practices...
only add columns to your SELECT clause when you have a use for them
enjoy PDO's very handy fetching modes -- FETCH_GROUP is perfect for your case.
always endeavor to minimize trips to the database
always endeavor to minimize the number of loops that you use.
Recommended code (yes, it is just that simple):
$sql = "SELECT email, todoID, mitarbeiterFID
FROM todo
JOIN mitarbeiter ON mitarbeiterID = mitarbeiterFID
WHERE archiv = 0
AND status != 3
AND durchfuehrung < CURRENT_DATE";
foreach ($db->query($sql)->fetchAll(PDO::FETCH_GROUP) as $email => $rows) {
sendSummary($email, $rows, $company, $db);
}
For the record, I don't know where $company comes from.
You could strongly improve your request with a JOIN (as said #mickmackusa) and even structure your response with PDO Fetch Statements (like PDO FETCH_GROUP or PDO FETCH ASSOC). You could directly get the result you want with one (bigger but better) request.
Nevertheless, if you want to simply sort your array with PHP, the use of foreach can do the job.
foreach ($array as $todo) {
if (!isset($newArray[$todo["mitarbeiterEmail"]])) {
// Here we create a new array with the email as a key and put the two first key-values in it with array_slice
$newArray[$todo["mitarbeiterEmail"]] = [array_slice($todo, 0, 2)];
} else {
// Here we push another todo if the email is already declared as a key
array_push($newArray[$todo["mitarbeiterEmail"]], array_slice($todo, 0, 2));
}
}
// Check your new array
print_r($newArray);
You could also avoid foreach by combining array_multisort to sort by emails then array_reduce to remove duplicate elements keeping associated data. Many solutions can be proposed.
Related
My system is composed of 3 components.
ios - php - mysql
If ios users select multiple 'id's,
then, ios app posts these selected 'id' request to server,
and, the server finds the data based on selected 'id's in MySQL.
and finally, the server passes these data to ios app.
These are the PHP statements I tried.
<?php
$id_0 = $_POST['0'];
$id_1 = $_POST['1'];
$id_2 = $_POST['2'];
$id_3 = $_POST['3'];
...
$id_n = $_POST['n'];
$conn = mysqli_connect('address', 'user', 'password', 'database');
$sql = "SELECT * FROM firstname WHERE id = '$id_0'";
if ($result = mysqli_query($conn, $sql)) {
$resultArray = array();
$tempArray = array();
while($row = $result->fetch_object()) {
$tempArray = $row;
array_push($resultArray, $tempArray);
}
} else {
}
mysqli_close($conn);
?>
To get multiple 'id' data, I found that I need to use loop statement.
But the problem is, the number of selected id are variable.
I think that in this code, I need two loop statement.
To get multiple data based on multiple id(the number of id are variable)
To append these data into array
I don't know how to use loop statement when the range is not defined.
How can I use loop statement in PHP, and how can I rearrange these codes?
I'm sure this question is a duplicate but I can't find an exact source.
Your current method means you need to run a whole MySQL query for each iteration.
As the query results will never change what the iteration contains, therefore; you can simply rework it to use MySQL IN to load all of your variables at once:
Step 1) Take all the values and place them in a single array.
$new = []; //new array
foreach ($_POST as $key=>$id){
// Check it is a numeric key
// Check id value is valid to avoid importing other POST values
if((int)$key == $key && (int)$id == $id){
$new[] = $id;
}
}
The above code block is nessecary to stop using values from $_POST['button'] or other posted data that should not be included. This step can be removed if you can clarify your posted data, such as saving all ids to a $_POST['id'] array itself.
Step 2) Empty the array of null/void or repeated values.
$new = array_unique($new);
Step 3) Turn the array into a string, inside the SQL
$arrayString = implode(',',$new);
Step 4) Plug the string into the SQL IN clause:
$sql = "SELECT * FROM firstname WHERE id IN (" . $arrayString . ") ORDER BY id";
Simplified and reduced:
$new = []; //new array
foreach ($_POST as $key=>$id){
if((int)$key == $key && (int)$id == $id){
$new[] = $id;
}
}
$new = array_unique($new);
$sql = "SELECT * FROM firstname WHERE id IN (" .
implode(',',$new) . ") ORDER BY id";
The SQL query above will give you an array of arrays, each one a different row. You can the order them, sort them and output them in PHP as you wish.
See also: This Q&A.
BUT
As expressed by others, you really, REALLY should be using Prepared Statements with MySQLi.
See also here, here and here to see further how to do it and WHY you should do it.
Top Tips:
Numerical columns (typical of id columns) in MySQL do not need the ' quotes.
Until you're using Prepared Statements you can typecast the variables to integers ((int)$var) to limit risk.
It is better to specify the columns you need rather than to use the * catch-all.
Your $_POST data should be an array $_POST['ids'][...].
Eat five different pieces of fruit or veg' a day.
Never trust user input!
Suppose I have 2 tables:
Applicants:
- id
- full_name
- address
Educations:
- id
- applicant_id
- institute
- address
After I did an inner join, I want to loop through the data on a template. But first I want to convert all educational records for an applicant into an array and attach it to the applicant record.
applicant_a:
- id
- full_name
- address
- educations: [ OBJECTS HERE ]
What's the way to do so? Can I do it on the database side via SQL? Or do I have to do it on PHP side?
This is a simple draft of how I would do it in your case.
I'm not saying this is the best or even the only way to do it.
This specific is not tested, though I used that logic often before.
Note, that this is only about the logic here.... but this should give you what you want!
$applicants = array();
$old_applicant_id=null;
while ($row=$db->fetch()) {
// new applicant
if($row['applicant_id']!=$old_applicant_id) {
// save the education to the old one - if there is one
if(isset($applicant)) {
$applicant['education'] = $educations;
$applicants[] = $applicant;
}
// then (and in first round)
$applicant = array();
$applicant['fullName'] = $row['fullName'];
// repeat for other values of applicant
$educations = array(); // initialize educations
$education = array();
$education['id'] = $row['edu_id'];
// repeat for institute, etc
$educations[] = $education;
} else {
// already existing applicant, so only add education
$education = array();
$education['id'] = $row['edu_id'];
// repeat for institute, etc
$educations[] = $education;
}
// set old applicant
$old_applicant_id = $row['applicant_id'];
}
// finally you have to save the last one to the array
$applicant['education'] = $educations;
$applicants[] = $applicant;
Another way would be to have two seperate queries and merge them in two loops.
This question of mine was related. I was asking about the spead of the two versions. Might be interesting.
MySQL does not return arrays, so this kind of thing is usually best done in client code (in this case PHP).
But if you are just going to concatenate the educations into a string of some sort you may want to look into the GROUP_CONCAT aggregate function.
So I have a form that lets people add a list of grades, and using jQuery they are able to add up to 9 extra fields. (Meaning they can submit any number from 1 to 10 grades). What I want to know is how I can go about storing this in my database, as I do not want to convert them all into one string. My concern is that because I don't know how many grades the user is going to enter, I can't set a definitive array number to store (or can I?)
Sorry if this is not terribly well explained, I'm still relatively new to PHP and SQL!
A quick assumption if I am not mistaken would be: saving one user with multiple grades for multiple subjects can be achieved like this.
Firstly we get one user id from the form and put it in PHP:
$id = isset($_POST['id'])? $_POST['id']:'';
Then get multiple grades and subjects which would be sent as comma separated values:
//$id= explode(',',$_POST['id']);// For multiple users
$grade= explode(',',$_POST['grade']);
$subj= explode(',',$_POST['subject']);
$entry= explode(',',$_POST['entry']);
Now count the number of grades: $count= count($grade);
Use the count in a for loop to have insert in loop:
for ($i = 0; $i < $count; $i++) {
try {
$dbh->beginTransaction(); //$dbh is your PDO connection
$insertQ = "INSERT INTO `grades` (id, grade, subject, entry)
VALUES('$id', '$grade[$i]', '$subj[$i]','$entry[$i])";
$dbh->query($insertQ);
$dbh->commit();
} catch (Exception $e) {
$error = $e->getMessage();
}
}
Hope this may help.
Unfortunately I can't show you the code but I can give you an idea of what it looks like, what it does and what the problem I have...
<?php
include(db.php);
include(tools.php);
$c = new GetDB(); // Connection to DB
$t = new Tools(); // Classes to clean, prevent XSS and others
if(isset($_POST['var'])){
$nv = json_decode($_POST['var'])
foreach($nv as $k) {
$id = $t->clean($k->id);
// ... goes on for about 10 keys
// this might seems redundant or insufficient
$id = $c->real_escape_string($id);
// ... goes on for the rest of keys...
$q = $c->query("SELECT * FROM table WHERE id = '$id'");
$r = $q->fetch_row();
if ($r[1] > 0) {
// Item exist in DB then just UPDATE
$q1 = $c->query(UPDATE TABLE1);
$q4 = $c->query(UPDATE TABLE2);
if ($x == 1) {
$q2 = $c->query(SELECT);
$rq = $q2->fetch_row();
if ($rq[0] > 0) {
// Item already in table just update
$q3 = $c->query(UPDATE TABLE3);
} else {
// Item not in table then INSERT
$q3 = $c->query(INSERT TABLE3);
}
}
} else {
// Item not in DB then Insert
$q1 = $c->query(INSERT TABLE1);
$q4 = $c->query(INSERT TABLE2);
$q3 = $c->query(INSERT TABLE4);
if($x == 1) {
$q5 = $c->query(INSERT TABLE3);
}
}
}
}
As you can see is a very basic INSERT, UPDATE tables script, so before we release to full production we did some test to see that the script is working as it should, and the "result" where excellent...
So, we ran this code against 100 requests, everything when just fine... less than 1.7seconds for the 100 requests... but then we saw the amount of data that needed to be send/post it was a jaw drop for me... over 20K items it takes about 3 to 5min to send the post but the script always crash the "data" is an array in json
array (
[0] => array (
[id] => 1,
[val2] => 1,
[val3] => 1,
[val4] => 1,
[val5] => 1,
[val6] => 1,
[val7] => 1,
[val8] => 1,
[val8] => 1,
[val9] => 1,
[val10] => 1
),
[1] => array (
[id] => 2,
[val2] => 2,
[val3] => 2,
[val4] => 2,
[val5] => 2,
[val6] => 2,
[val7] => 2,
[val8] => 2,
[val8] => 2,
[val9] => 2,
[val10] => 2
),
//... about 10 to 20K depend on the day and time
)
but in json... any way, sending this information is not a problem, like I said it can take about 3 to 5mins the problem is the code that does the job receiving the data and do the queries... in a normal shared hosting we get a 503 error which by doing a debug it turn out to be a time out, so for our VPS we can increment the max_execution_time to whatever we need to, to process 10K+ it takes about 1hr in our VPS, but in a shared hosting we can't use max_execution_time... So I ask the other developer the one that is sending the information that instead of sending 10K+ in one blow to send a batch of 1K and let it rest for a second then send another batch..and so on ... so far I haven't got any answer... so I was thinking to do the "pause" on my end, say, after process 1K of items wait for a sec then continue but I don't see it as efficient as receiving the data in batches... how would you solve this?
Sorry, I don't have enough reputation to comment everywhere, yet, so I have to write this in an answer. I would recommend zedfoxus' method of batch processing above. In addition, I would highly recommend figuring out a way of processing those queries faster. Keep in mind that every single PHP function call, etc. gets multiplied by every row of data. Here are just a couple of the ways you might be able to get better performance:
Use prepared statements. This will allow MySQL to cache the memory operation for each successive query. This is really important.
If you use prepared statements, then you can drop the $c->real_escape_string() calls. I would also scratch my head to see what you can safely leave out of the $t->clean() method.
Next I would evaluate the performance of evaluating every single row individually. I'd have to benchmark it to be sure, but I think running a few PHP statements beforehand will be faster than making umpteen unnecessary MySQL SELECT and UPDATE calls. MySQL is much faster when inserting multiple rows at a time. If you expect multiple rows of your input to be changing the same row in the database, then you might want to consider the following:
a. Think about creating a temporary, precompiled array (depending on memory usage involved) that stores the unique rows of data. I would also consider doing the same for the secondary TABLE3. This would eliminate needless "update" queries, and make part b possible.
b. Consider a single query that selects every id from the database that's in the array. This will be the list of items to use an UPDATE query for. Update each of these rows, removing them from the temporary array as you go. Then, you can create a single, multi-row insert statement (prepared, of course), that does all of the inserts at a single time.
Take a look at optimizing your MySQL server parameters to better handle the load.
I don't know if this would speed up a prepared INSERT statement at all, but it might be worth a try. You can wrap the INSERT statement within a transaction as detailed in an answer here: MySQL multiple insert performance
I hope that helps, and if anyone else has some suggestions, just post them in the comments and I'll try to include them.
Here's a look at the original code with just a few suggestions for changes:
<?php
/* You can make sure that the connection type is persistent and
* I personally prefer using the PDO driver.
*/
include(db.php);
/* Definitely think twice about each tool that is included.
* Only include what you need to evaluate the submitted data.
*/
include(tools.php);
$c = new GetDB(); // Connection to DB
/* Take a look at optimizing the code in the Tools class.
* Avoid any and all kinds of loops–this code is going to be used in
* a loop and could easily turn into O(n^2) performance drain.
* Minimize the amount of string manipulation requests.
* Optimize regular expressions.
*/
$t = new Tools(); // Classes to clean, prevent XSS and others
if(isset($_POST['var'])){ // !empty() catches more cases than isset()
$nv = json_decode($_POST['var'])
/* LOOP LOGIC
* Definitely test my hypothesis yourself, but this is similar
* to what I would try first.
*/
//Row in database query
$inTableSQL = "SELECT id FROM TABLE1 WHERE id IN("; //keep adding to it
foreach ($nv as $k) {
/* I would personally use specific methods per data type.
* Here, I might use a type cast, plus valid int range check.
*/
$id = $t->cleanId($k->id); //I would include a type cast: (int)
// Similarly for other values
//etc.
// Then save validated data to the array(s)
$data[$id] = array($values...);
/* Now would also be a good time to add the id to the SELECT
* statement
*/
$inTableSQL .= "$id,";
}
$inTableSQL .= ");";
// Execute query here
// Then step through the query ids returned, perform UPDATEs,
// remove the array element once UPDATE is done (use prepared statements)
foreach (.....
/* Then, insert the remaining rows all at once...
* You'll have to step through the remaining array elements to
* prepare the statement.
*/
foreach(.....
} //end initial POST data if
/* Everything below here becomes irrelevant */
foreach($nv as $k) {
$id = $t->clean($k->id);
// ... goes on for about 10 keys
// this might seems redundant or insufficient
$id = $c->real_escape_string($id);
// ... goes on for the rest of keys...
$q = $c->query("SELECT * FROM table WHERE id = '$id'");
$r = $q->fetch_row();
if ($r[1] > 0) {
// Item exist in DB then just UPDATE
$q1 = $c->query(UPDATE TABLE1);
$q4 = $c->query(UPDATE TABLE2);
if ($x == 1) {
$q2 = $c->query(SELECT);
$rq = $q2->fetch_row();
if ($rq[0] > 0) {
// Item already in table just update
$q3 = $c->query(UPDATE TABLE3);
} else {
// Item not in table then INSERT
$q3 = $c->query(INSERT TABLE3);
}
}
} else {
// Item not in DB then Insert
$q1 = $c->query(INSERT TABLE1);
$q4 = $c->query(INSERT TABLE2);
$q3 = $c->query(INSERT TABLE4);
if($x == 1) {
$q5 = $c->query(INSERT TABLE3);
}
}
}
}
The key is to minimize queries. Often, where you are looping over data doing one or more queries per iteration, you can replace it with a constant number of queries. In your case, you'll want to rewrite it into something like this:
include(db.php);
include(tools.php);
$c = new GetDB(); // Connection to DB
$t = new Tools(); // Classes to clean, prevent XSS and others
if(isset($_POST['var'])){
$nv = json_decode($_POST['var'])
$table1_data = array();
$table2_data = array();
$table3_data = array();
$table4_data = array();
foreach($nv as $k) {
$id = $t->clean($k->id);
// ... goes on for about 10 keys
// this might seems redundant or insufficient
$id = $c->real_escape_string($id);
// ... goes on for the rest of keys...
$table1_data[] = array( ... );
$table2_data[] = array( ... );
$table4_data[] = array( ... );
if ($x == 1) {
$table3_data[] = array( ... );
}
}
$values = array_to_sql($table1_data);
$c->query("INSERT INTO TABLE1 (...) VALUES $values ON DUPLICATE KEY UPDATE ...");
$values = array_to_sql($table2_data);
$c->query("INSERT INTO TABLE2 (...) VALUES $values ON DUPLICATE KEY UPDATE ...");
$values = array_to_sql($table3_data);
$c->query("INSERT INTO TABLE3 (...) VALUES $values ON DUPLICATE KEY UPDATE ...");
$values = array_to_sql($table4_data);
$c->query("INSERT IGNORE INTO TABLE4 (...) VALUES $values");
}
While your original code executed between 3 and 5 queries per row of your input data, the above code only executes 4 queries in total.
I leave the implementation of array_to_sql to the reader, but hopefully this should explain the idea. TABLE4 is an INSERT IGNORE here since you didn't have an UPDATE in the "found" clause of your original loop.
I have a time dependent script I am working on and used microtime() to find the bottle neck. I determined the time increase is caused by doing a check on 300+ values to see if they exist in a database one at a time at 0.04 seconds a query.
The background of the script is it is a caching script. I need to see if it exists in the DB so I need a true/false (obtained by a rowCount) but i also need a way to relate a false to a value so I can update it. I know using a WHERE tag IN (:ARRAY) would work faster than the individual calls, but I cant think of a way to apply an association of true/false to value in this method.
My current code is below:
//loop through all our values!
//prepare out reusuable statement
$stmt = $db->prepare("SELECT * from cache WHERE value=?");
foreach($values as $tempVal)
{
//find if its in the database
try
{
$stmt->execute(array($tempVal));
$valCount = $stmt->rowCount();
} catch(PDOException $ex) {
echo "PDO error send this to us: " . $ex->getMessage();
}
//update flag
$addToUpdate = 1;
//if its in the database
if($valCount > 0)
{
//get the tag data
$valRes= $stmt->fetch();
//check if cache expired
$addToUpdate = 0;
}
//add to update list
if($addToUpdate)
{
//needs updating
$updateList[] = $tempVal;
//add to not in DB list to minimize queries
if($tagTCount == 0)
{
$notInDB[$tempVal] = $tempVal;
}
}
Any suggestions? I can explain more if anything is not clear.
Thank you,
Nick
So you just issue your query with the complete array, using the IN (?,?,?,?,?...) list:
// abstract, use a PDO wrapper of your choosing
$query = db("SELECT * FROM cache WHERE value IN (??)", $values);
Then iterate over the result list. Only matched $values will return. So build your first list from that:
foreach ($query as $row) {
$updateList[] = $row["value"];
}
To get the list of absent entries, just diff that against your original array:
$notInDB = array_diff($values, $updateList);
You could of course use a second NOT IN query. But doing that differentiation in PHP is simpler.