I am trying to set up a database to record the last 30 days of information for each user. The data will be recorded once a day (i.e. by a cron job) and will be the value of an item (i.e. constantly changes).
What would be the best way to structure this? I was thinking of setting a table and then just storing the 30 days in the table and deleting the 31st day as I add the new day with the cron job (and shifting all of the others up one day) but this doesn't seem very efficient..
Thanks for the help.
What you can do is store the current date with each entry, then in your cron job, delete all entries that are greater than thirty days old.
For example (with MySQL),
DELETE FROM user_statistics WHERE DATEDIFF(NOW(), date_of_record) > 30;
Store the user data with its own date and delete the oldest when you exceed your limit. No need to shift anything.
I'd log by actual date using a DATE column. You can query up "last 30 days" pretty easily in MySQL.
As for purging, the cron job can delete anything older than 30 days pretty easily as well. Or, since it's so easy to ignore anything older than 30 days, you might even choose to not delete older records (at least not every day).
Related
Here is the scenario.
I have a schedule job running every minute which inserts data into a MYSQL table "demo". The total number of records per day are 60*24 = 1440.
Table demo already has 55000 records.
I want to clean records less than today's date. Therefore I am using below code to do the work daily at 10.00 AM.
$demo = Demo::whereDate('created_at','<', Carbon::today());
if(count($demo) > 0)
{
$demo->delete();
}
Now a point will come where at the same time I am inserting to the same table and deleting from the same table.
I want to know that it will be safe? Or there will be an error or any other impact.
I don't think this will be an issue, since Carbon::today() returns 00:00:00 as time and you are executing the deletion job at 10:00:00. Only records that are inserted more than 10 hours ago will be deleted.
According to me, there should not be any problem. If there are two requests at the same time MySQL server will handle them by itself. The same thing happens when a website is loaded. There is a lot of call at the same time.
There is no problem. Imagine you made 1000 requests per minutes, no one of them will overwrites an other one.
I am creating a system that requires a schedular for a particular task. Users may pick from times 24 hours a day, 7 days a week.
I came up with a few options for the database storage, but I don't think either one is the most efficient design, so I'm hoping for some possible alternatives that may be more efficient.
On the user side I created a grid of buttons with 2 loops to create the days, and the times, and I set each a unique value of $timeValue = "d".$j."-t".$i;
So d1-t0 will be Saturday at Midnight d3-t12= Tuesday at Noon, and so forth.
So, in the database I was first going to simply have a ID, day, time set up, but that would result in a possible 168 rows per event
Then I tried an ID, day, and time 0-23 (a column for each hour of the day) And I was simply going to have a boolean set up. 0 if not selected, 1 if it is.
This would result in 7 rows per event, but I think querying that data might be a pain.
I need to perform a few functions on this data. On each day, list the number of selected times into an array. But I don't believe having a select statement of SELECT * from schedule where time0, =1 or time1= 1 .... ect will work, nor will it produce the desired array. (times=(0,3,5,6,7...)
So, this isnt going to work well.
My overall system will need to also know every event that has each time selected for a mass posting.
"Select * from table where time = $time (0-23) and day= $day (1-7)
Do action with data...
So with this requirement, I'm going to assume that storing the times as an array within the database is likely not the most efficient way either.
So am I stuck with needing up to 168 rows of data per event, or is there a better way I am missing? Thanks
Update:
To give a little more clarity on what I need to accomplish:
Users will be creating event campaigns in which other users can bid on various time slots for something to happen. There will likely be 10-100 thousand of these campaigns at any one time and they are ongoing until the creator stops them. The campaign creators can define the time slots available for their campaign.
At the designated time each day the system will find every campaign that has an event scheduled and perform the event.
So the first requirement is to know which time slots are available for the campaign, and then I need the system to quickly identify campaigns that have an event on each hour and day and perform it automatically.
Not sure the best way to describe what I'm trying to do, so bear with me. I'm working in PHP with a mysql database.
I have a database of 10,000 records. Suppose I want to update 10000 / 365 each day throughout the year. Each record gets updated 1x per year, and nicely spread throughout the year.
One easy way to do this is to select all records, then for each, if ID % 365 = $day_of_year, update that record. I'm not worried about leap year.
Is there a way I can select only the records from the database that I need (around 27), rather than selecting all 10,000 and looping through each? This is a cron job that I will run in the middle of the night, so maybe this is a moot point. Still, it bugs me that I have to brute force my way through all 10,000. Would love to find a more elegant solution by only pulling the tiny fraction needed.
Add a column to your table indicating the day of the year the record should be updated.
Then add an event that runs once a year that resets that column values and calculates it new - 10000 records spread on the days of the year.
Then add another event that runs every night updating the records for the day.
Thanks for the response, especially Juergen and Mike Pomax. Here's the solution.
I added a new column and populated each row with a random value between 0 and 364. I could have populated it some other way but this was easy and close enough.
Here is my pseudocode:
$day_of_year = date('z');
if($day_of_year < 365){ //don't do on Dec 31 of leap year
$qry = "Select ... WHERE ID = $day_of_year";
... //do my thing
}
Works great. Thanks again.
I want to make a table where the entries expire 24 hours after they have been inserted in PHP and MySQL.
Ideally I want to run a "deleting process" every time a user interacts with my server, that deletes old entries. Since this is more frequent you should it will not have large amounts of data to delete so it should only take a few milliseconds.
I have given each entry a date/time added value.
How would I do this?
You could use MySQL's event scheduler either:
to automatically delete such records when they expire:
CREATE EVENT delete_expired_101
ON SCHEDULE AT CURRENT_TIMESTAMP + INTERVAL 24 HOUR DO
DELETE FROM my_table WHERE id = 101;
to run an automatic purge of all expired records on a regular basis:
CREATE EVENT delete_all_expired
ON SCHEDULE EVERY HOUR DO
DELETE FROM my_table WHERE expiry < NOW();
you shouldn't do a delete process when a user interacts. it slows down things, you should use a cronjob (every minute / hour)
you'll want to index the added timestamp value and then run DELETE FROM table WHERE added < FROM_UNIXTIME(UNIX_TIMESTAMP()-24*60*60)
maybe you'll want to checkout Partitions, which divide the table into different tables, but it behaves as one table. The advantage is that you don't need to delete the entries and you'll have seperate tables for each day.
i think that YOU think that much data slows down tables. Maybe you should use EXPLAIN (MySQL Manual) and optimize your SELECT queries using indexes (MySQL Manual)
UPDATE Check out eggyal's answer - This is another approach worth taking a look.
You can look into using Cron Job, http://en.wikipedia.org/wiki/Cron Make it run once every 24 hours when it matches your requirement.
This will help
Delete MySQL row after time passes
Is there a way to add articles (data) in mysql/php but make them
auto-publish during the day when I´m not available.
So lets say, if I have a news site but I´ll be busy tomorrow the whole day so I could pre-write articles the day before with timestamp and they would appear when I want
Is this possible?
How would the script be like:
SELECT FROM articles WHEN TIME is 2011-12-01 12:15
Thanks
As simple as:
SELECT * FROM articles WHERE timestamp <= NOW()
Though I never worked with them to me the easiest solutions seems to be Cronjobs combined with an extra waiting table and a script linking both.
You pre-write your article and store them in table together with the time stamp you want to publish them.
Your cron will invoke a script every 2,3, 5 hours (twice a day, whatever).
This script checks the time stamps in the table against the actual time and if it is about time to realise the article it will do so (or hand the information to the realise script).