How can I insert rows into my DB daily using Cron Jobs? - php

I have this Cron Job that I want to run once a day. It checks to see if it's a new day, and if it is, it inserts a new row into my database table. I use this to keep track of the daily sales for each item on my online store.
I also need it to grab the day from the last row for that product_id, increment it by 1 and then include it in my INSERT query. This is so that I can easily keep track of the current day for each product.
The MAX(time_stamp) is also illegal code, however I don't know what to replace it with. I use it to select the latest day for each item, so that the Cron Job doesn't select every single row.
I've never used Cron Jobs before, meaning that I'm a noob so please be easy on me :)
Thanks for the help! :)
Here's my code:
PHP:
// Checks if past a day
$days = DB::fetch("SELECT * FROM `daily_sales` WHERE MAX(time_stamp)<=CURRENT_DATE - INTERVAL 1 DAY;");
$numrows = count($days);
// If there is a new day
if ($numrows > 0) {
foreach ($days as $i => $day) {
// Adds a fresh day with 0 sales
$sales = 0;
$product_id = $day->product_id;
$day = // Get the day from the last row, and add 1
$time_stamp = time();
DB::query('INSERT INTO `daily_sales` (`product_id`, `sales`, `day`, `time_stamp`) VALUES (?, ?, ?, ?);', array($product_id, $sales, $day, $time_stamp));
}
}
daily_sales: (the DB table)
id | product_id | day | time_stamp | sales

you have to open cron job, add new job and define
how to open cron tab
crontab -e
add following to your cron tab
0 0 * * * php /full/path/to/script.php
OR
you can use mysql triggers http://www.mysqltutorial.org/mysql-triggers.aspx that will allow you to make "audit" table which will contain daly sales. So you can insert/update audit table by using "after insert".
hope this will help

Related

How to improve execution time for this SQL query?

I have a table with the following example:
ID
name
task
startDate
endDate
1
John
Day
2022-09-20
2022-09-21
2
Joe
Midday
2022-09-20
2022-09-21
3
John
Day
2022-09-22
2022-09-23
4
Sara
Night
2022-09-20
2022-09-21
5
Joe
Night
2022-09-24
2022-09-25
I would like to count the rows that have a name in a given name list AND task in a given task list AND within a date range.
I am using php and the code and sql query I am currently using is:
$taskList = list of all the tasks I want to search for
$nameList = list of all the names I want to search for
SELECT COUNT(ID) FROM table WHERE
`task` IN ('".implode("', '", $taskList)."')
AND `name` IN ('".implode("', '", $nameList)."')
AND (startDate >= '".$startDate."')
AND (startDate <= '".$endDate."')
It takes ~40 milliseconds per execution but I need to do this multiple times. Is there a way I can shorten the execution time?
Looks like your query is fast enough, but on some case it also slow.
Please make sure:
Add index group to your table columns that where query executed:
task, name, startDate
Consider don't use asterisk to select count query with the column name that not nullable values, try to use select count(primary_key_column)
Just Test The difference.
But, before start please doing escape the values before doing direct query to sql to prevent sql injection and or invalid data value.
/**
* #var PDO $pdo
*/
$taskList = array_map([$pdo, 'quote'], $taskList);
$nameList = array_map([$pdo, 'quote'], $nameList);
$startDate = $pdo->quote($startDate);
$endDate = $pdo->quote($endDate);
$quotedTaskList = implode(',', $taskList);
$quotedNameList = implode(',', $nameList);
$query = "SELECT COUNT(primary_key_or_non_nullable_column) FROM table WHERE
`task` IN ({$quotedTaskList}')
AND `name` IN ({$quotedNameList})
AND (`startDate` >= {$startDate})
AND (`startDate` <= {$endDate})
";

Processing millions of data records with PHP MySQL issue

I have run into a delayed processing time for a PHP program,
I have a MySQL record with over 1000 tables;
Each table is created once a new device is added, e.g assets_data_imeixx - to assets_data_imeixx1000th table
Each table contains about 45,000 rows of records inserted every 10 seconds,
Below is my PHP code to query the database and fetch all these records based on datetime.
Issue: The program executes without error but it takes about 1.3minutes to 4mins for very large records.
PHP Code:
$ms = mysqli connection string in config.php //$ms is OKAY
$user_id = '5';
$q = "SELECT * FROM `user_assets` WHERE `user`='".$user_id ."' ORDER BY `imei` ASC";
$r = mysqli_query($ms,$q);
$result = array(); //$result array to contain all data
while($row =mysqli_fetch_array($r)){
//fetch 7 days record
for ($i=1; $i < 7; $i++) {
$date = "-" . $i . " days";
$days_ago = date('Y-m-d', strtotime($date, strtotime('today')));
$sql1 = "SELECT * FROM assets_data_" . $row["imei"] . " WHERE dt_time LIKE '" . $days_ago . "%' LIMIT 1"; // its correct
//$result1 = $conn->query($sql1);
$result1 = mysqli_query($ms,$sql1);
$row2 = mysqli_fetch_array($result1);
echo $row['imei']." ".$row2['dt_server']."<br/>";
}
}
Above code fetches over 1000 devices from user_assets table, These IMEI each has its own table that contains over 45,000 records in each table of location data.
The for loop iterates over each IMEI table and records.
Above code runs without error but take so much time to complete, I want to find a solution to optimize and have code execute in a very short time max 5 seconds.
I need help and suggestions on optimizing and running this large scale of data and iteration.
(from Comment)
CREATE TABLE gs_object_data_863844052008346 (
dt_server datetime NOT NULL,
dt_tracker datetime NOT NULL,
lat double DEFAULT NULL,
lng double DEFAULT NULL,
altitude double DEFAULT NULL,
angle double DEFAULT NULL,
speed double...
(From Comment)
gs_object_data_072101424612
gs_object_data_072101425049
gs_object_data_072101425486
gs_object_data_072101445153
gs_object_data_111111111111111
gs_object_data_1234567894
gs_object_data_222222222222222
gs_object_data_2716325849
gs_object_data_2716345818
gs_object_data_30090515907
gs_object_data_3009072323
gs_object_data_3009073758
gs_object_data_352093088838221
gs_object_data_352093088839310
gs_object_data_352093088840045
gs_object_data_352121088128697
gs_object_data_352121088132681
gs_object_data_352621109438959
gs_object_data_352621109440203
gs_object_data_352625694095355
gs_object_data_352672102822186
gs_object_data_352672103490900
gs_object_data_352672103490975
gs_object_data_352672103490991
gs_object_data_352887074794052
gs_object_data_352887074794102
gs_object_data_352887074794193
gs_object_data_352887074794417
gs_object_data_352887074794425
gs_object_data_352887074794433
gs_object_data_352887074794441
gs_object_data_352887074794458
gs_object_data_352887074794474
gs_object_data_352887074813696
gs_object_data_352887074813712
gs_object_data_352887074813720
gs_object_data_352887074813753
gs_object_data_352887074813761
gs_object_data_352887074813803
900+ tables each having different location data.
Requirement: Loop through each table, fetch data for selected date range say:
"SELECT dt_server FROM gs_object_data_" . $row["imei"] . " WHERE dt_server BETWEEN '2022-02-05 00:00:00' AND '2022-02-12 00:00:00'";
Expected Result: Return result set containing data from each table containing information for the selected date range. That means having 1000 tables will have to be looped through each table and also fetch data in each table.
I agree with KIKO -- 1 table not 1000. But, if I understand the rest, there are really 2 or 3 main tables.
Looking at your PHP -- It is often inefficient to look up one list, then go into a loop to find more. The better way (perhaps 10 times as fast) is to have a single SELECT with a JOIN to do both selects at once.
Consider some variation of this MySQL syntax; it may avoid most of the PHP code relating to $days_ago:
CURDATE() - INTERVAL 3 DAY
After also merging the Selects, this gives you the rows for the last 7 days:
WHERE date >= CURDATE() - INTERVAL 7 DAY
(I did not understand the need for LIMIT 1; please explain.)
Yes, you can use DATETIME values as strings, but try not to. Usually DateTime functions are more efficient.
Consider "composite" indexes:
INDEX(imei, dt)
which will be very efficient for
WHERE imei = $imei
AND dt >= CURDATE() - INTERVAL 7 DAY
I would ponder ways to have less redundancy in the output; but that should mostly be done after fetching the raw data from the table(s).
Turn on the SlowLog with a low value of long_query_time; it will help you locate the worst query; then we can focus on it.
An IMEI is up to 17 characters, always digits? If you are not already using this, I suggest BIGINT since it will occupy only 8 bytes.
For further discussion, please provide SHOW CREATE TABLE for each of the main tables.
Since all those 1000 tables are the same it would make sense to put all that data into 1 table. Then partition that table on date, use proper indexes, and optimize the query.
See: Normalization of Database
Since you limit results to one user, and one row per device, it should be possible to execute a query in well below one second.

Insert multiple rows in MySQL and check for random string without long delay (~80 rows each minute)

For a research project I am obtaining data from a local bus company's GPS system (through their API). I created a php cron job that runs every minute to obtain data like the vehicle, route ID, location, destination, etc. The data did not contain a unique "run number" for each bus route (a unique number so that I can track the progression of a single bus along its route), so I created my own that checks if the vehicle ID, destination, and relative time are similar, and assigns the unique "run ID" to it so that I can track the bus along its route. If no run ID exists, a random one is generated. (Any vehicle with the same "vid" and "pid" within 2 minutes of the last inserted row "timeadded" is on the same run, and this is important for my research)
Each time the cron runs (1 minute), approximately 80 rows are added into the database.
Initially the job would run quickly. However, with over 500,000 rows now, I've noticed the job can take upwards of 40 seconds. I believe it's because for each of the ~80 rows, it has to check the entire table ("vehicles") to see if the same run ID exists, essentially querying a large table and inserting a row 80 times. I want to get at least a week's worth of data (on day 4 now), at which point I can export the data, erase all rows, and start over. My question is: Is there any way I can refactor my PHP/SQL code to make the process run faster? It's been years since I've worked with SQL, so I'm sure there's a more ingenious way to insert all this data.
<?php
// Obtain data from XML
$xml = simplexml_load_file("url.xml");
foreach ($xml->vehicle as $vehicle) {
$vid = $vehicle->vid;
$tm = $vehicle->tmstmp;
$dat = substr($vehicle->tmstmp, 0, 8);
$tme = substr($vehicle->tmstmp, 9);
$lat = $vehicle->lat;
$lon = $vehicle->lon;
$hdg = $vehicle->hdg;
$pid = $vehicle->pid;
$rt = $vehicle->rt;
$des = $vehicle->des;
$pdist = $vehicle->pdist;
// Database connection and insert
mysql_connect("redacted", "redacted", "redacted") or die(mysql_error()); mysql_select_db("redacted") or die(mysql_error());
$sql_findsim = "SELECT vid, pid, timeadded, run, rt FROM vehicles WHERE vid=" . mysql_real_escape_string($vid). " AND pid=" . mysql_real_escape_string($pid). " AND rt=" . mysql_real_escape_string($rt). " AND timeadded > DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 2 MINUTE);";
$handle = mysql_query($sql_findsim);
$row = mysql_fetch_row($handle);
$runid = $row[3];
if($runid !== null) {
$run = $runid;
} else {
$run = substr(md5(rand()), 0, 30);
}
$sql = "INSERT INTO vehicles (vid, tmstmp, dat, tme, lat, lon, hdg, pid, rt, des, pdist, run) VALUES ($vid,'$tm','$dat','$tme','$lat','$lon',$hdg,$pid,'$rt','$des',$pdist,'$run')";
$result = mysql_query($sql);
mysql_close();
}
?>
Thanks for any help with refactoring this code to get it to run more quickly and efficiently.
Do you have any indexes on the table? A compound index on (vid,pid,rt,timeadded) will make the query faster, avoiding a full table scan.
create index fastmagic on vehicles (vid,pid,rt,timeadded)
Alternatively, you could skip the select all together and just to the insert without assigning the "run" random value. This will keep your cron job at "constant time" since all you're doing is appending new data.
After you've got your week of data go back and write "second pass" code to step through each row (select * from vehicle order by timeadded). For each row, do your "select" similar to how you've already done it - then "update" the row you are processing now.
If you go with the alternate, you'll probably want an autoincrement "id" integer column to make row identification clearer (if you don't already have one).
I would suggest that,
Create a table as vehicle_ids ( or some meaningful name ) these fields.
vid, pid, run, rt
instead of checking in vehicles table for vid, you can check the above table for id, if not insert ( make vid as auto increment ).
Normalize your table and also index your vehicle table

Delete old rows in table if maximum exceeded

I have insert in table any time when users open any post on my site, in this way im get real time 'Whats happend on site'
mysql_query("INSERT INTO `just_watched` (`content_id`) VALUES ('{$id}')");
but now have problem because have over 100K hits every day, this is a 100K new rows in this table every day, there is any way to limit table to max 100 rows, and if max is exceeded then delete old 90 and insert again or something like that, have no idea what's the right way to make this
my table just_watched
ID - content_id
ID INT(11) - AUTO_INCREMENT
content_id INT(11)
Easiest way that popped into my head would be to use php logic to delete and insert your information. Then every time a user open a new post you would then add the count the database. (this you are already doing)
The new stuff comes here
Enter a control before the insertion, meaning before anything is inserted you would first count all the rows, if it does not exceed 100 rows then add a new row.
If it does exceed 100 rows then you before inserting a new row you, first do a delete statement THEN you insert a new row.
Example (sudo code) :
$sql = "SELECT COUNT(*) FROM yourtable";
$count = $db -> prepare($sql);
$count -> execute();
if ($count -> fetchColumn() >= 100) { // If the count is over a 100
............... //Delete the first 90 leave 10 then insert a new row which will leave you at 11 after the delete.
} else {
.................. // Keep inserting until you have 100 then repeat the process
}
More information on counting here. Then some more information on PDO here.
Hopefully this helps :)
Good luck.
Also information on how to set up PDO if you haven't already.
What I would do? :
At 12:00 AM every night run a cron job that deletes all rows from the past day. But thats just some advice. Have a good one.
Use this query for deleting old rows except last 100 rows:
DELETE FROM just_watched where
ID not in (SELECT id fromjust_watched order by ID DESC LIMIT 100)
You can run it by CRON in every n period where (n= hours, or minutes, or any)
$numRows = mysql_num_rows(mysql_query("SELECT ID FROM just_watched"));
if ($numRows > 100){
mysql_query("DELETE FROM just_watched LIMIT 90");
}
mysql_query("INSERT INTO `just_watched` (`content_id`) VALUES ('{$id}')");
I guess this should work fine.
You can get the number of rows in your table with:
$size = mysql_num_rows($result);
With the size of the table, you can check, if it's getting to big, and then remove 90 rows:
// Get 90 lines of code
$query = "Select * FROM just_watched ORDER BY id ASC LIMIT 90";
$result = mysql_query($query);
// Go through them
while($row = mysql_fetch_object($result)) {
// Delete the row with the id
$id = $row['id'];
$sql = 'DELETE FROM just_watched
WHERE id=$id';
}
Another way would be to just delete an old row if you add a new row to the table. The only problem is, that if something get's jammed, the table might get to big.
You may use
DELETE FROM just_watched ORDER BY id DESC LIMIT 100, 9999999999999999;
So, it'll delete all the rows from the offset 100 to a big number (for end of the tables). if you always run this query before you insert new one then it'll do the job for you.

MySQL - Blog post scheduling system

I am creating a blog post scheduling system using CodeIgniter. I want 10 posts to show up a day. There is a field in the posts table named scheduled_date which I will get the posts that are less than or equal to the current date. When an admin user adds a new record to the database, I need an SQL statement that somehow will help me COUNT the number of records with the latest date in the database. For example:
// 9 records returned for the date 2011-01-01
$numbers_of_records == 9;
if($numbers_of_records == 10){
// inserts record with `scheduled_date`='2011-01-01'
}else{
// inserts record with the date latest date +1 day
}
How would I efficiently accomplish this?
Thanks
This will do the trick. It is simple and efficient.
<?php
// It is very bad to have floating values, especially for settings
// it is good to use some sort of factory or settings class
$maxDailyPosts = (int) SettingsFactory::getSettings()->get('maxDailyPosts');
$date = '2011-01-01';
// Load # of post for data
$numberOfRecords = (int) getNumberOfPostPerDate($date);
// Figure out the interval for increment
$dayInterval = ($numberOfRecords >= $maxDailyPosts ) ? 1 : 0;
//
$query = "INSERT INTO tbl (publish_date, ...) VALUES (DATE_ADD('$date', INTERVAL $dayInterval DAY), ...)";
?>

Categories