I am creating a running inventory system. I am looking for suggestions on how to speed up the processing time to populate my table that contains the daily inventories.
I have 4 database tables that I am using to do this:
1). Daily Usage table
2). Incoming Product table (product ordered to come in)
3). Physical Inventory Counts table (Monthly Physical Count totals held here)
4). Perpetual Running Inventory table (In a perfect world this is what inventories should be)
As of now I have each piece of data being updated in my perpetual table and it takes quite awhile for my code to run through, do the calculations, connect to the database and update the information.
My question is: How can I speed this up? Is there a way that I can upload all info from an array at once so I am only updating the database once instead of a couple hundred times?
Sorry for dumping 75 lines of code below but I wanted to make sure to show everything so it can easily be seen what is going on.
My logic behind the code:
I start off by pulling info from all 4 tables into arrays and then cycling through the arrays. I use a date function to start from 2 weeks ago instead of the very beginning of the table (Beginning of 2016). Once the date quota is met, I pull the previous days perpetual running inventory, subtract the current dates daily usage, and add the current days incomming to get the current days perpetual. Once I have the current days perpetual I update the perpetual table with that piece of data.
If I simply just stored the information into to the array, could I basically write over the whole table at the end? Im assuming this would be much faster if it is possible?
Heres the code:
<?php
include("connection.php");
/* $rowper below (Row Perpetual) */
$query= "SELECT * FROM perpetual";
$result = mysqli_query($link, $query);
/* $rowdaily below */
$query2= "SELECT * FROM dailyusage";
$result2 = mysqli_query($link, $query2);
/* $rowpo below (Row Purchase Order) */
$query3= "SELECT * FROM incomming";
$result3 = mysqli_query($link, $query3);
/* $rowinv below (Row Inventory) */
$query4= "SELECT * FROM inventory";
$result4 = mysqli_query($link, $query4);
$checkdate = mktime(0, 0, 0, date('n'), date('d')-14, date('y'));
$checkdate= date('Y-m-d', $checkdate);
$b=1;
while (($rowper=mysqli_fetch_array($result)) and ($rowdaily=mysqli_fetch_array($result2)) and ($rowpo=mysqli_fetch_array($result3)) and ($rowinv=mysqli_fetch_array($result4))){
$a=2;
if($b == 1){
while($a< mysqli_num_fields($result)) {
/* $it holds the item #s from the column */
$it[$a]=$rowper[$a];
$a++;
}
$a=2;
}
if($rowper[1]>= $checkdate){
if($b>2){
while($a< mysqli_num_fields($result)) {
$previnv[$a];
if($rowinv[$a]!=0){
$rowper[$a]=$rowinv[$a];
$previnv[$a]=$rowper[$a];
/* the 'p' is because column name was made by item # + p at end to make valid colum name */
$query="UPDATE perpetual SET ".$it[$a]."p ='".$rowper[$a]."' WHERE date='".$rowper[1]."' LIMIT 1";
mysqli_query($link, $query);
}else{
$rowper[$a]=$previnv[$a] - $rowdaily[$a] + $rowpo[$a];
$previnv[$a]=$rowper[$a];
$query="UPDATE perpetual SET ".$it[$a]."p ='".$rowper[$a]."' WHERE date='".$rowper[1]."' LIMIT 1";
mysqli_query($link, $query);
}
$a++;
}
}
}
$b++;
}
?>
Stored Procedures and User Defined Functions are what you are looking for. You can schedule them to run at certain times to perform operations on your tables.
Side note, remember the fundamentals behind ETL.
Extract - Transform - Load in that order.
Related
I want to get how many (possibly 0) times a particular number occurs in a particular column. I set the number in $contact_client_ID then do the SELECT query below.
$sql = "SELECT * FROM t_contacts WHERE contact_client_ID ='$contact_client_ID'";
$result=(mysqli_query($link, $sql));
$count_result= mysqli_num_rows($result);
echo "contact client ID $contact_client_ID xxxxx $count_result";
Instead of $count_result containing the number I want, it contains a result made up of the number I want and the contact_client_ID joined together and the result doesn't seem to be usable as a number in any following code.
So, if $contact_client_ID = 50 and there are 2 occurrences of it in the table, the output I get is:
contact client ID 50 xxxxx 250
I've looked at the manuals and examples all over the place (including here) and I can't see what I'm doing wrong.
There are multiple ways of doing this, more precisely you have the option, to either do the count in PHP, or make your SQL server do the counting and just return the number:
Do it in PHP:
What it requires is: - fetch all data; -make a counter variable; - loop trough the data, for each loop increase counter +1
For small tables you can use PHP, for bigger ones I advice doing it on the SQL, since for PHP to count it it must fetch all the data.
$counter=0;
$query = "SELECT * FROM t_contacts WHERE contact_client_ID ='$contact_client_ID";
$res = $con->query($query);
while ($row = $res->fetch_assoc()) {
$counter++;
}
Do it in SQL
Now the smarter way would be to do it on the SQL server ,as it would handle the load better;
I would say what /u/Adaleni wrote is pretty close to what I would use:
$sql = "SELECT count(contact_client_ID) as total FROM t_contacts WHERE contact_client_ID ='$contact_client_ID'";
$result=mysqli_query($link, $sql);
$count_result= mysqli_fetch_row(result);
echo "contact client ID $contact_client_ID xxxxx $count_result[0]['total']";
Lets just describe what he does:
we use COUNT() function in SQL, this makes the server count the number of occurances of contact_client_ID and then make (in the result) a new variable called "total"
we execute the query and get the result
We use mysqli_fetch_rowm this function gets the result row as an enumerated array
then we access that array (as we know its only 1 item, we accessed index 0) and we print the variable total which we made in step 1 - $count_result[0]['total']
Try this
/* Select queries return a resultset */
if ($result = $mysqli->query("SELECT * FROM t_contacts WHERE contact_client_ID ='$contact_client_ID")) {
printf("Select returned %d rows.\n", $result->num_rows);
I need to synchronize specific information between two databases (one mysql, the other a remote hosted SQL Server database) for thousands of rows. When I execute this php file it gets stuck/timeouts after several minutes I guess, so I wonder how I can fix this issue and maybe also optimize the way of "synchronizing" it.
What the code needs to do:
Basically I want to get for every row (= one account) in my database which gets updated - two specific pieces of information (= 2 SELECT queries) from another SQL Server database. Therefore I use a foreach loop which creates 2 SQL queries for each row and afterwards I update those information into 2 columns of this row. We talk about ~10k Rows which needs to run thru this foreach loop.
My idea which may help?
I have heard about things like PDO Transactions which should collect all those queries and sending them afterwards in a package of all SELECT queries, but I have no idea whether I use them correctly or whether they even help in such cases.
This is my current code, which is timing out after few minutes:
// DBH => MSSQL DB | DB => MySQL DB
$dbh->beginTransaction();
// Get all referral IDs which needs to be updated:
$listAccounts = "SELECT * FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();
foreach($ps_listAccounts as $row) {
$refid=$row['refId'];
// Refsinserted
$refsInserted = "SELECT count(username) as done FROM accounts WHERE referral='$refid'";
$ps_refsInserted = $dbh->prepare($refsInserted);
$ps_refsInserted->execute();
$row = $ps_refsInserted->fetch();
$refsInserted = $row['done'];
// Refscompleted
$refsCompleted = "SELECT count(username) as done FROM accounts WHERE referral='$refid' AND finished=1";
$ps_refsCompleted = $dbh->prepare($refsCompleted);
$ps_refsCompleted->execute();
$row2 = $ps_refsCompleted->fetch();
$refsCompleted = $row2['done'];
// Update fields for local order db
$updateGifting = "UPDATE Gifting SET refsInserted = :refsInserted, refsCompleted = :refsCompleted WHERE refId = :refId";
$ps_updateGifting = $db->prepare($updateGifting);
$ps_updateGifting->bindParam(':refsInserted', $refsInserted);
$ps_updateGifting->bindParam(':refsCompleted', $refsCompleted);
$ps_updateGifting->bindParam(':refId', $refid);
$ps_updateGifting->execute();
echo "$refid: $refsInserted Refs inserted / $refsCompleted Refs completed<br>";
}
$dbh->commit();
You can do all of that in one query with a correlated sub-query:
UPDATE Gifting
SET
refsInserted=(SELECT COUNT(USERNAME)
FROM accounts
WHERE referral=Gifting.refId),
refsCompleted=(SELECT COUNT(USERNAME)
FROM accounts
WHERE referral=Gifting.refId
AND finished=1)
A correlated sub-query is essentially using a sub-query (query within a query) that references the parent query. So notice that in each of the sub-queries I am referencing the Gifting.refId column in the where clause of each sub-query. While this isn't the best for performance because each of those sub-queries still has to run independent of the other queries, it would perform much better (and likely as good as you are going to get) than what you have there.
Edit:
And just for reference. I don't know if a transaction will help here at all. Typically they are used when you have several queries that depend on each other and to give you a way to rollback if one fails. For example, banking transactions. You don't want the balance to deduct some amount until a purchase has been inserted. And if the purchase fails inserting for some reason, you want to rollback the change to the balance. So when inserting a purchase, you start a transaction, run the update balance query and the insert purchase query and only if both go in correctly and have been validated do you commit to save.
Edit2:
If I were doing this, without doing an export/import this is what I would do. This makes a few assumptions though. First is that you are using a mssql 2008 or newer and second is that the referral id is always a number. I'm also using a temp table that I insert numbers into because you can insert multiple rows easily with a single query and then run a single update query to update the gifting table. This temp table follows the structure CREATE TABLE tempTable (refId int, done int, total int).
//get list of referral accounts
//if you are using one column, only query for one column
$listAccounts = "SELECT DISTINCT refId FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();
//loop over and get list of refIds from above.
$refIds = array();
foreach($ps_listAccounts as $row){
$refIds[] = $row['refId'];
}
if(count($refIds) > 0){
//implode into string for use in query below
$refIds = implode(',',$refIds);
//select out total count
$totalCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE referral IN ($refIds) GROUP BY referral";
$ps_totalCounts = $dbh->prepare($totalCount);
$ps_totalCounts->execute();
//add to array of counts
$counts = array();
//loop over total counts
foreach($ps_totalCounts as $row){
//if referral id not found, add it
if(!isset($counts[$row['referral']])){
$counts[$row['referral']] = array('total'=>0,'done'=>0);
}
//add to count
$counts[$row['referral']]['total'] += $row['cnt'];
}
$doneCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE finished=1 AND referral IN ($refIds) GROUP BY referral";
$ps_doneCounts = $dbh->prepare($doneCount);
$ps_doneCounts->execute();
//loop over total counts
foreach($ps_totalCounts as $row){
//if referral id not found, add it
if(!isset($counts[$row['referral']])){
$counts[$row['referral']] = array('total'=>0,'done'=>0);
}
//add to count
$counts[$row['referral']]['done'] += $row['cnt'];
}
//now loop over counts and generate insert queries to a temp table.
//I suggest using a temp table because you can insert multiple rows
//in one query and then the update is one query.
$sqlInsertList = array();
foreach($count as $refId=>$count){
$sqlInsertList[] = "({$refId}, {$count['done']}, {$count['total']})";
}
//clear out the temp table first so we are only inserting new rows
$truncSql = "TRUNCATE TABLE tempTable";
$ps_trunc = $db->prepare($truncSql);
$ps_trunc->execute();
//make insert sql with multiple insert rows
$insertSql = "INSERT INTO tempTable (refId, done, total) VALUES ".implode(',',$sqlInsertList);
//prepare sql for insert into mssql
$ps_insert = $db->prepare($insertSql);
$ps_insert->execute();
//sql to update existing rows
$updateSql = "UPDATE Gifting
SET refsInserted=(SELECT total FROM tempTable WHERE refId=Gifting.refId),
refsCompleted=(SELECT done FROM tempTable WHERE refId=Gifting.refId)
WHERE refId IN (SELECT refId FROM tempTable)
AND refsCompleted <= 100";
$ps_update = $db->prepare($updateSql);
$ps_update->execute();
} else {
echo "There were no reference ids found from \$dbh";
}
For a research project I am obtaining data from a local bus company's GPS system (through their API). I created a php cron job that runs every minute to obtain data like the vehicle, route ID, location, destination, etc. The data did not contain a unique "run number" for each bus route (a unique number so that I can track the progression of a single bus along its route), so I created my own that checks if the vehicle ID, destination, and relative time are similar, and assigns the unique "run ID" to it so that I can track the bus along its route. If no run ID exists, a random one is generated. (Any vehicle with the same "vid" and "pid" within 2 minutes of the last inserted row "timeadded" is on the same run, and this is important for my research)
Each time the cron runs (1 minute), approximately 80 rows are added into the database.
Initially the job would run quickly. However, with over 500,000 rows now, I've noticed the job can take upwards of 40 seconds. I believe it's because for each of the ~80 rows, it has to check the entire table ("vehicles") to see if the same run ID exists, essentially querying a large table and inserting a row 80 times. I want to get at least a week's worth of data (on day 4 now), at which point I can export the data, erase all rows, and start over. My question is: Is there any way I can refactor my PHP/SQL code to make the process run faster? It's been years since I've worked with SQL, so I'm sure there's a more ingenious way to insert all this data.
<?php
// Obtain data from XML
$xml = simplexml_load_file("url.xml");
foreach ($xml->vehicle as $vehicle) {
$vid = $vehicle->vid;
$tm = $vehicle->tmstmp;
$dat = substr($vehicle->tmstmp, 0, 8);
$tme = substr($vehicle->tmstmp, 9);
$lat = $vehicle->lat;
$lon = $vehicle->lon;
$hdg = $vehicle->hdg;
$pid = $vehicle->pid;
$rt = $vehicle->rt;
$des = $vehicle->des;
$pdist = $vehicle->pdist;
// Database connection and insert
mysql_connect("redacted", "redacted", "redacted") or die(mysql_error()); mysql_select_db("redacted") or die(mysql_error());
$sql_findsim = "SELECT vid, pid, timeadded, run, rt FROM vehicles WHERE vid=" . mysql_real_escape_string($vid). " AND pid=" . mysql_real_escape_string($pid). " AND rt=" . mysql_real_escape_string($rt). " AND timeadded > DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 2 MINUTE);";
$handle = mysql_query($sql_findsim);
$row = mysql_fetch_row($handle);
$runid = $row[3];
if($runid !== null) {
$run = $runid;
} else {
$run = substr(md5(rand()), 0, 30);
}
$sql = "INSERT INTO vehicles (vid, tmstmp, dat, tme, lat, lon, hdg, pid, rt, des, pdist, run) VALUES ($vid,'$tm','$dat','$tme','$lat','$lon',$hdg,$pid,'$rt','$des',$pdist,'$run')";
$result = mysql_query($sql);
mysql_close();
}
?>
Thanks for any help with refactoring this code to get it to run more quickly and efficiently.
Do you have any indexes on the table? A compound index on (vid,pid,rt,timeadded) will make the query faster, avoiding a full table scan.
create index fastmagic on vehicles (vid,pid,rt,timeadded)
Alternatively, you could skip the select all together and just to the insert without assigning the "run" random value. This will keep your cron job at "constant time" since all you're doing is appending new data.
After you've got your week of data go back and write "second pass" code to step through each row (select * from vehicle order by timeadded). For each row, do your "select" similar to how you've already done it - then "update" the row you are processing now.
If you go with the alternate, you'll probably want an autoincrement "id" integer column to make row identification clearer (if you don't already have one).
I would suggest that,
Create a table as vehicle_ids ( or some meaningful name ) these fields.
vid, pid, run, rt
instead of checking in vehicles table for vid, you can check the above table for id, if not insert ( make vid as auto increment ).
Normalize your table and also index your vehicle table
I have a table that has rows like CALL_ID, Timestamp, Date, First_Name, Last_Name, Status and so on..
What I need to do is draw graphs based on the data, filtered according to their dates.
That is for the Month-to-date graph, I need to show the number of rows matching status=approved for each day of the month uptil current date.
And, for the Year-to-date graph, I need to show the number of rows matching Status=approved for each month of the year uptil current month.
My idea of doing it is this -
For Month-to-date:
$temp = date("Y-m-");
while($i<=date("d"))
{
$query = "SELECT call_id FROM main WHERE status='approved' AND date='".$temp.$i."'";
$result = mysql_query($query, $link) or die("",mysql_error());
$count[i]= mysql_num_rows($result);
}
For Date-to-year:
while ($i <= date("m"))
{
$query = "SELECT call_id FROM main WHERE status='approved' AND date BETWEEN'".$today_y."-".$i."-01' AND '".$today_y."-".$i."-31";
$result = mysql_query($query, $link) or die("",mysql_error());
$count_y[i]= mysql_num_rows($result);
}
And then I was thinking of pushing these counts present in the array by a "while" loop into another temporary tables (one for MTD and one for YTD) and then using PHP libchart to draw the graph from those tables.
Is there a better way to do this? Or is this the only way?
You can accomplish this a lot easier just using mysql and getting rid of a lot of your php loops. Something like this for month-to-date:
$query="SELECT COUNT(call_id) as 'num_records', date FROM main WHERE status='approved' AND CONCAT(YEAR(date),MONTH(date)) = CONCAT(YEAR(NOW()),MONTH(NOW())) GROUP BY date
That query will give a row showing the date and the total records for that day, for every day month-to-date.
What I'm basically trying to do is get the the staff table, fetch the id of the names per job title and then hit another table (based on the id fetched) and get the data I'm interested in off.
My approach so far is make a query, go with a while loop to get all the ids of the job title im interested and for every id go with another loop ( connection-query) to subtract more data.
I think my approach is wrong cause im suspicious i could merge those two queries into one not sure how though.
//new db connection here... (1)
$query="SELECT * FROM staff WHERE jobtitle='$forEachJob'";
$result=mysql_query($query);
while ($row = mysql_fetch_assoc($result)) {
$idFetched = $row['id'];
//new db connection here... (2)
$nextQuery = "SELECT * FROM schedule WHERE name='$idFetched' ORDER BY Day asc";
$nextResult = mysql_query($nextQuery)
}
You want to do a JOIN in mysql:
SELECT * FROM staff
JOIN schedule ON schedule.name = staff.id
WHERE jobtitle = '$forEachJob'
ORDER BY Day ASC
By the way, avoid using SELECT * and look into a DB wrapper such as PDO to sanitize/prepare your queries.