Every member has multiple records stored in the database. I need to sum a column for every user and get the highest / lowest value from another column. The table has over 100k records, one user might have more than 2k records in the table.
I tried this:
$query = $mysqli->query("SELECT DISTINCT `id` FROM `table`");
if($query){
$IDs = new SplFixedArray($query->num_rows);
$IDs = $query->fetch_all();
unset($query, $row);
set_time_limit(300);
foreach ($IDs as $key => $value) {
$query = $mysqli->query("SELECT SUM(price), dtime FROM `table` WHERE `id` = '".$value[0]."' ORDER BY dtime DESC");
if($query){
$row = $query->fetch_assoc();
print_r($row);
}
}
But is setting the time limit to 300 really the proper way doing this? I also tried a prepared statement, only assigning the ID in the loop, and several other things. All of which aren't working as I'd whish.
100k records really isn't that many, there should be no reason for this query to take longer than 5 minutes.
Instead of getting a distinct list of IDs and iterating through them, querying these values for each ID, it would probably be better to do everything all at once, then iterate over your results to do what you need to.
select
`id`,
sum(`price`) as `sum_price`,
min(`dtime`) as `min_dtime`,
max(`dtime`) as `max_dtime`
from
`table`
group by
`id`
(this is assuming that the "other field" that you need to get the min and max of is dtime)
I'm not strong on the PHP side though, but from a SQL perspective it's much, much faster to do things this way.
Related
Here's my usual way of counting rows...
$query = "SELECT * FROM users";
$stmt = $db->prepare($query);
$stmt->execute();
$count = $stmt->rowCount();
This will count all rows, even if I use a WHERE clause, it'll still count every row that meets that condition. However, let's say I have a table, we'll call it tokensEarned (that's my actual table name). I have the following data...
user_id = 1,2,4,5,8,8,2,4,3,7,6,2 (those are actual rows in my table - clearly, user 1 has 1 entry, 2 has three entries, etc.) In all, I have 12 entries. But I don't want my query to count 12. I want my query to count each user_id one time. In this example, my count should display 8.
Any help on this? I can further explain if you have any specific questions or clarification you need. I would appreciate it. Thank You.
The following query will yield the distinct user count:
$query = "SELECT COUNT(DISTINCT user_id) AS cnt FROM users";
$stmt = $db->prepare($query);
$stmt->execute();
$row = $stmt->fetch(PDO::FETCH_ASSOC);
echo "distinct user count: " . $row['cnt'];
It isn't possible to get all records and the distinct count in a single query.
Whether you use the query above or you return all the actual distinct rows really depends on whether you need the full records. If all you need are the counts, then it is wasteful to return the data in the records, and what I gave above is probably the best option. If you do need the data, then selecting all distinct rows might make more sense.
You can use distinct in mysql to select only unique fields in your table.
$query = "SELECT distinct user_id FROM users";
$stmt = $db->prepare($query);
$stmt->execute();
$count = $stmt->rowCount();
Change your query to the following, this way you only shows the unique user_id:
$query = "SELECT DISTINCT user_id FROM users";
I need to synchronize specific information between two databases (one mysql, the other a remote hosted SQL Server database) for thousands of rows. When I execute this php file it gets stuck/timeouts after several minutes I guess, so I wonder how I can fix this issue and maybe also optimize the way of "synchronizing" it.
What the code needs to do:
Basically I want to get for every row (= one account) in my database which gets updated - two specific pieces of information (= 2 SELECT queries) from another SQL Server database. Therefore I use a foreach loop which creates 2 SQL queries for each row and afterwards I update those information into 2 columns of this row. We talk about ~10k Rows which needs to run thru this foreach loop.
My idea which may help?
I have heard about things like PDO Transactions which should collect all those queries and sending them afterwards in a package of all SELECT queries, but I have no idea whether I use them correctly or whether they even help in such cases.
This is my current code, which is timing out after few minutes:
// DBH => MSSQL DB | DB => MySQL DB
$dbh->beginTransaction();
// Get all referral IDs which needs to be updated:
$listAccounts = "SELECT * FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();
foreach($ps_listAccounts as $row) {
$refid=$row['refId'];
// Refsinserted
$refsInserted = "SELECT count(username) as done FROM accounts WHERE referral='$refid'";
$ps_refsInserted = $dbh->prepare($refsInserted);
$ps_refsInserted->execute();
$row = $ps_refsInserted->fetch();
$refsInserted = $row['done'];
// Refscompleted
$refsCompleted = "SELECT count(username) as done FROM accounts WHERE referral='$refid' AND finished=1";
$ps_refsCompleted = $dbh->prepare($refsCompleted);
$ps_refsCompleted->execute();
$row2 = $ps_refsCompleted->fetch();
$refsCompleted = $row2['done'];
// Update fields for local order db
$updateGifting = "UPDATE Gifting SET refsInserted = :refsInserted, refsCompleted = :refsCompleted WHERE refId = :refId";
$ps_updateGifting = $db->prepare($updateGifting);
$ps_updateGifting->bindParam(':refsInserted', $refsInserted);
$ps_updateGifting->bindParam(':refsCompleted', $refsCompleted);
$ps_updateGifting->bindParam(':refId', $refid);
$ps_updateGifting->execute();
echo "$refid: $refsInserted Refs inserted / $refsCompleted Refs completed<br>";
}
$dbh->commit();
You can do all of that in one query with a correlated sub-query:
UPDATE Gifting
SET
refsInserted=(SELECT COUNT(USERNAME)
FROM accounts
WHERE referral=Gifting.refId),
refsCompleted=(SELECT COUNT(USERNAME)
FROM accounts
WHERE referral=Gifting.refId
AND finished=1)
A correlated sub-query is essentially using a sub-query (query within a query) that references the parent query. So notice that in each of the sub-queries I am referencing the Gifting.refId column in the where clause of each sub-query. While this isn't the best for performance because each of those sub-queries still has to run independent of the other queries, it would perform much better (and likely as good as you are going to get) than what you have there.
Edit:
And just for reference. I don't know if a transaction will help here at all. Typically they are used when you have several queries that depend on each other and to give you a way to rollback if one fails. For example, banking transactions. You don't want the balance to deduct some amount until a purchase has been inserted. And if the purchase fails inserting for some reason, you want to rollback the change to the balance. So when inserting a purchase, you start a transaction, run the update balance query and the insert purchase query and only if both go in correctly and have been validated do you commit to save.
Edit2:
If I were doing this, without doing an export/import this is what I would do. This makes a few assumptions though. First is that you are using a mssql 2008 or newer and second is that the referral id is always a number. I'm also using a temp table that I insert numbers into because you can insert multiple rows easily with a single query and then run a single update query to update the gifting table. This temp table follows the structure CREATE TABLE tempTable (refId int, done int, total int).
//get list of referral accounts
//if you are using one column, only query for one column
$listAccounts = "SELECT DISTINCT refId FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();
//loop over and get list of refIds from above.
$refIds = array();
foreach($ps_listAccounts as $row){
$refIds[] = $row['refId'];
}
if(count($refIds) > 0){
//implode into string for use in query below
$refIds = implode(',',$refIds);
//select out total count
$totalCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE referral IN ($refIds) GROUP BY referral";
$ps_totalCounts = $dbh->prepare($totalCount);
$ps_totalCounts->execute();
//add to array of counts
$counts = array();
//loop over total counts
foreach($ps_totalCounts as $row){
//if referral id not found, add it
if(!isset($counts[$row['referral']])){
$counts[$row['referral']] = array('total'=>0,'done'=>0);
}
//add to count
$counts[$row['referral']]['total'] += $row['cnt'];
}
$doneCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE finished=1 AND referral IN ($refIds) GROUP BY referral";
$ps_doneCounts = $dbh->prepare($doneCount);
$ps_doneCounts->execute();
//loop over total counts
foreach($ps_totalCounts as $row){
//if referral id not found, add it
if(!isset($counts[$row['referral']])){
$counts[$row['referral']] = array('total'=>0,'done'=>0);
}
//add to count
$counts[$row['referral']]['done'] += $row['cnt'];
}
//now loop over counts and generate insert queries to a temp table.
//I suggest using a temp table because you can insert multiple rows
//in one query and then the update is one query.
$sqlInsertList = array();
foreach($count as $refId=>$count){
$sqlInsertList[] = "({$refId}, {$count['done']}, {$count['total']})";
}
//clear out the temp table first so we are only inserting new rows
$truncSql = "TRUNCATE TABLE tempTable";
$ps_trunc = $db->prepare($truncSql);
$ps_trunc->execute();
//make insert sql with multiple insert rows
$insertSql = "INSERT INTO tempTable (refId, done, total) VALUES ".implode(',',$sqlInsertList);
//prepare sql for insert into mssql
$ps_insert = $db->prepare($insertSql);
$ps_insert->execute();
//sql to update existing rows
$updateSql = "UPDATE Gifting
SET refsInserted=(SELECT total FROM tempTable WHERE refId=Gifting.refId),
refsCompleted=(SELECT done FROM tempTable WHERE refId=Gifting.refId)
WHERE refId IN (SELECT refId FROM tempTable)
AND refsCompleted <= 100";
$ps_update = $db->prepare($updateSql);
$ps_update->execute();
} else {
echo "There were no reference ids found from \$dbh";
}
Hi with the follow code I request what dates are all in my database without duplicates.
Then I save it to an array. In the array I also need an other value.
The value I need is how much users are in one day in the database without duplicates.
For Example the array must later lookslike 23.07.2013 - 10, 24.07.2013 - 50 (users).
I search for several hours but I don't find a good mysql query.
$query = "SELECT id, user, timestamp FROM stat WHERE timestamp BETWEEN '$datum1' AND '$datum2' GROUP BY timestamp";
$result = mysql_query($query,$db);
while($row = mysql_fetch_assoc($result))
{
mysql_num_rows($result);
$dataset1[] = array(strtotime($row['timestamp']),$number_of_users_on_this_day);
}
Try:
$query = "SELECT id, user, COUNT(*) as count FROM stat WHERE timestamp BETWEEN '$datum1' AND '$datum2' GROUP BY timestamp";
This will return the number of entries in the value 'count'
if you want distinct data, in place of * use
COUNT(DISTINCT id)
with whatever field you want to be unique in place of 'id'
I have a table as below,
ID Name Age
----------------------
100 A 10
203 B 20
Now how do i select only row1 using MySQL SELECT command and then I've to increase +1 to it to select row2. In short I'll be using for loop to do certain operations.
Thanks.
Sounds like you've got a mix up. You want to select all the rows you want to iterate through in your for loop with your query, and then iterate through them one by one using php's mysql functions like mysql_fetch_row
You should not try to use tables in a linear fashion like this. Set your criteria, sorting as appropriate, and then to select the next row use your existing criteria and limit it to one row.
SELECT * FROM `table` ORDER BY `ID` LIMIT 1
SELECT * FROM `table` ORDER BY `ID` WHERE ID > 100 LIMIT 1
You'd probably be better off retrieving all rows that you need, then using this. Note the LIMIT is entirely optional.
$query = mysql_query(' SELECT ID, Name, Age FROM table_name WHERE condition LIMIT max_number_you_want '))
while ($row = mysql_fetch_assoc($query)
{
// Do stuff
// $row['ID'], $row['Name'], $row['Age']
}
Lots of small queries to the database will execute much slower than one decent-sized one.
You should get the result into an array (php.net : mysql_fetch_*).
And after you'll can loop on the array "to do certain operations"
Yep, this is a pretty common thing to do in PHP. Like the others who have posted, here is my version (using objects instead of arrays):
$result = mysql_query("SELECT * FROM table_name");
while ($row = mysql_fetch_object($result)) {
// Results are now in the $row variable.
// ex: $row->ID, $row->Name, $row->Age
}
I have a table with attributes like this:
likes(id, message, ip, time)
id is the primary key and is auto-incrementing. Whet I want to do is select 30 rows from this table based on an id that I provide.
I can build a random array of all IDs I want to grab from the table, now I just need to go about doing that. My usual SQL query would be:
SELECT message
FROM `likes`
WHERE id=$id
LIMIT 1
But I need to select 30 rows. There are two ways I can do this, I can either use a FOR loop with 30 queries, or combine it all into one query and grab them all at the same time. Obviously I would like to do the latter.
Can anyone help me out on the SQL query to use to grab more than one row from my database table at once? And perhaps an example of how I would parse these into an array for easy retrieval?
Greatly appreciated, thanks.
You could use the IN operator, so it would be something like this:
SELECT message FROM likes WHERE id IN (1, 2, 3, 4, 5, ...)
$ids = array(1,2,3,4,5,6,7,8,9); //or however many
$query = "SELECT * FROM `likes` WHERE `id` IN (".implode(', ', $ids).")";
$res = mysql_query($query);
$results = array();
while($tmp = mysql_fetch_array($res)) {
$results[$tmp['id'] = $tmp;
}
Will give you an array, $results, sorted by key ['id'], with each being an array on each answer key-- 'id', 'message', 'ip', 'time'. So for example:
for($results as $key => $val) {
echo "${val['ip']} posted {$var['message']} on {$val['time']} <br/>";
}