I am looking for a way to check if there are certain number of records within mysql table. For example: After POST request been before putting data to dabase, it checks first how many records there are. If lets say there are 24 records, then it will delete record with latest date based on timestamp and then inster new value from POST request. Has anyone got idea on how to do it? Looking forward fpr your answers. Below I attached simple code i wrote to insert data from post request into table.
<?php
include("connect.php");
$link=Connection();
$temp1=$_POST["temp1"];
$hum1=$_POST["hum1"];
$query = "INSERT INTO `tempLog` (`temperature`, `humidity`)
VALUES ('".$temp1."','".$hum1."')";
mysql_query($query,$link);
mysql_close($link);
header("Location: index.php");
?>
When you say delete with the latest date I have to assume you mean the oldest record? Your description doesnt tell me the name of you date field so Lets assume its onDate. You also didnt mention what your primary key is so lets assume that is just id. if you run the below query before inserting it will purge all the oldest records leaving only the newest 23 in the database.
delete from templog where id in (
select id from (
select #rownum:=#rownum+1 'rowid', t.id from templog t, (select #rownum:=0)r order by t.onDate
)v where v.rowid > 23
);
Of course you should test on data you don't mind losing.
It is best to do a cleanup purge each time instead of removing a single row before adding a new one because in the event of exceptions it will never clean itself down to the 24 rows you wish to truly have.
I also want to note that you may want to reconsider this method all together. Instead leave the data there, and only query the most recent 24 when displaying the log. Since you are going through the trouble of collecting the data you might as well keep it for future reporting. Then later down the road if your table gets to large run a simple daily purge query to delete anything older than a certain threshold.
Hope this helps.
Related
Well, I'm afraid that I will not be able to post a minimum reproducible example, and for that I apologize. But, here goes nothing.
Ours is a weekly prepared meals service. I track order volume in many ways. Here is the structure of the relevant table:
So then I utilize the highlighted fields in many ways, such as indicating to delivery drivers if a customer is returning from the prior order being more than a month ago (last_order_w - prev_order_w > 4), for instance.
Lately I have been noticing that the data is not consistently updating properly. In the past 3 weeks, I would say it is an occurrence of 5%. If it were more consistent, I would be more confident in my ability to track down the issue, but I am not even sure how to provoke it, as I only really notice it after the fact.
The code that should cause the update is below:
<?php
//retrieve and iterate over IDs of orders placed since last synchronization.
$newOrders=array_map('reset',$dbh->query("select id from wp_posts where id > (select max(synced) from fitaf_weeks) and post_type='shop_order' and post_status='wc-processing'")->fetchAll(PDO::FETCH_NUM));
foreach($newOrders as $no){
//retrieve the metadata for the current order
$newMetas=array_map('reset',$dbh->query("select meta_key,meta_value from wp_postmeta where post_id=$no")->fetchAll(PDO::FETCH_GROUP|PDO::FETCH_UNIQUE));
//check if the current order is associated with an existing customer
$exist=$dbh->query("select * from fitaf_customers where id=".$newMetas['_customer_user'])->fetch();
//if not, gather the information we want to store from this post
$noExist=[$newMetas['_customer_user'],$newMetas['_shipping_first_name'],$newMetas['_shipping_last_name'],$newMetas['_shipping_address_1'],(strlen($newMetas['_shipping_address_2'])==0?NULL:$newMetas['_shipping_address_2']),$newMetas['_shipping_city'],$newMetas['_shipping_state'],$newMetas['_shipping_postcode'],$phone,$newMetas['_billing_email'],1,1,$no,$newMetas['_paid_date'],$week[3],$newMetas['_order_total']];
if($exist){
//if we found a record in the customer table, retrieve the data we want to modify
$oldO=$dbh->query("select last_order_id,last_order,last_order_w,lo,num_orders from fitaf_customers where id=".$newMetas['_customer_user'])->fetch(PDO::FETCH_GROUP|PDO::FETCH_ASSOC|PDO::FETCH_UNIQUE);
//make changes to the retrieved data, and make sure we are storing the most recently used delivery address and prepare the data points for the update command
$exists=[$phone,$newMetas['_shipping_first_name'],$newMetas['_shipping_last_name'],$newMetas['_shipping_postcode'],$newMetas['_shipping_address_1'],(strlen($newMetas['_shipping_address_2'])==0?NULL:$newMetas['_shipping_address_2']),$newMetas['_shipping_city'],$newMetas['_shipping_state'],$newMetas['_paid_date'],$no,$week[3],$oldO['last_order'],$oldO['last_order_id'],$oldO['last_order_w'],($oldO['num_orders']+1),($oldO['lo']+$newMetas['_order_total']),$newMetas['_customer_user']];
}
if(!$exist){
//if the customer did not exist, perform an insert
$dbh->prepare("insert into fitaf_customers(id,fname,lname,addr1,addr2,city,state,zip,phone,email,num_orders,num_weeks,last_order_id,last_order,last_order_w,lo) values(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)")->execute($noExist);
}
else{
//if the customer did exist, update their data
$dbh->prepare("update fitaf_customers set phone=?,fname=?,lname=?,zip=?,addr1=?,addr2=?,city=?,`state`=?,last_order=?,last_order_id=?,last_order_w=?,prev_order=?,prev_order_id=?,prev_order_w=?,num_orders=?,lo=? where id=?")->execute($exists);
}
}
//finally retrieve the most recent post ID and update the field we check against when the syncornization script runs
$lastPlaced=$dbh->query('select max(id) from wp_posts where post_type="shop_order"')->fetch()[0];
$updateSync=$dbh-> query("update fitaf_weeks set synced=$lastPlaced order by id desc limit 1");
?>
Unfortunately I don't have any relevant error logs to show, however, as I documented the code for this post, I realized a potential shortcoming. I should be utilizing the data retrieved from the initial query of new posts, rather than a selecting the highest post id after performing this logic. However, I have timers running on my scripts, and this section hasn't taken over 3 seconds to run in a long time. So it seems unlikely, that the script, which runs on a cron every 5 minutes, is experiencing this unintended overlap?
While I have made the change to pop the highest ID off of $newOrders, and hope it solves the issue, I am still curious to see if anyone has any insights on what could cause this logic to fail at such a low occurrence.
It seems likely your problem comes from race conditions between multiple operations accessing your db.
First of all, your last few lines of code do SELECT MAX(ID) and then uses that value to update something. You Can't Do That™. If somebody else adds a row to that wp_posts table anytime after the entry you think is relevant, you'll use the wrong ID. I don't understand your app well enough to recommend a fix. But I do know this is a serious and notorious problem.
You have another possible race condition as well. Your logic is this:
SELECT something.
make a decision based on what you SELECTED.
INSERT or UPDATE based on that decision.
If some other operation, done by some other user of the db, intervenes between step 1 and step 3, your decision might be wrong.
You fix this with a db transaction. The ->beginTransaction() operation, well, begins the transaction. The ->commit() operation concludes it. And, the SELECT operation you use for step one should say SELECT ... FOR UPDATE.
I wrote a small script which uses the concept of long polling.
It works as follows:
jQuery sends the request with some parameters (say lastId) to php
PHP gets the latest id from database and compares with the lastId.
If the lastId is smaller than the newly fetched Id, then it kills the
script and echoes the new records.
From jQuery, i display this output.
I have taken care of all security checks. The problem is when a record is deleted or updated, there is no way to know this.
The nearest solution i can get is to count the number of rows and match it with some saved row count variable. But then, if i have 1000 records, i have to echo out all the 1000 records which can be a big performance issue.
The CRUD functionality of this application is completely separated and runs in a different server. So i dont get to know which record was deleted.
I don't need any help coding wise, but i am looking for some suggestion to make this work while updating and deleting.
Please note, websockets(my fav) and node.js is not an option for me.
Instead of using a certain ID from your table, you could also check when the table itself was modified the last time.
SQL:
SELECT UPDATE_TIME
FROM information_schema.tables
WHERE TABLE_SCHEMA = 'yourdb'
AND TABLE_NAME = 'yourtable';
If successful, the statement should return something like
UPDATE_TIME
2014-04-02 11:12:15
Then use the resulting timestamp instead of the lastid. I am using a very similar technique to display and auto-refresh logs, works like a charm.
You have to adjust the statement to your needs, and replace yourdb and yourtable with the values needed for your application. It also requires you to have access to information_schema.tables, so check if this is available, too.
Two alternative solutions:
If the solution described above is too imprecise for your purpose (it might lead to issues when the table is changed multiple times per second), you might combine that timestamp with your current mechanism with lastid to cover new inserts.
Another way would be to implement a table, in which the current state is logged. This is where your ajax requests check the current state. Then generade triggers in your data tables, which update this table.
You can get the highest ID by
SELECT id FROM table ORDER BY id DESC LIMIT 1
but this is not reliable in my opinion, because you can have ID's of 1, 2, 3, 7 and you insert a new row having the ID 5.
Keep in mind: the highest ID, is not necessarily the most recent row.
The current auto increment value can be obtained by
SELECT AUTO_INCREMENT FROM information_schema.tables
WHERE TABLE_SCHEMA = 'yourdb'
AND TABLE_NAME = 'yourtable';
Maybe a timestamp + microtime is an option for you?
I'm trying to find a way to check if some IDs are already in the DB, if an ID is already in the DB I'd naturally try to avoid processing the row it represents
Right now I'm doing a single query to check for the ID, but I think this is too expensive in time because if I'm checking 20 id's the script is taking up to 30 seconds
I know i can do a simple WHERE id=1 OR id=2 OR id=3 , but I'd like to know of a certain group of IDs which ones are already in the database and which ones are not
I don't know much about transactions but maybe this could be useful or something
any thoughts are highly appreciated!
Depends how you determine the "Group of IDs"
If you can do it with a query, you can likely use a join or exists clause.
for example
SELECT firstname
from people p
where not exists (select 1 from otherpeople op where op.firstname = p.firstname)
This will select all the people who are not in the otherpeople table
If you just have a list of IDs, then use WHERE NOT IN (1,3,4...)
30 seconds for 20 queries on a single value is a long time. Did you create an index on the ID field to speed things up?
Also if you create a unique key on the ID field you can just insert all ID's. The database will throw errors and not insert those those ID's that already exist, but you can ignore those errors.
I am having a wee problem, and I am sure there is a more convenient/simpler way to achieve the solution, but all searches are throw in up a blanks at the moment !
I have a mysql db that is regularly updated by php page [ via a cron job ] this adds or deletes entries as appropriate.
My issue is that I also need to check if any details [ie the phone number or similar] for the entry have changed, but doing this at every call is not possible [ not only does is seem to me to be overkill, but I am restricted by a 3rd party api call limit] Plus this is not critical info.
So I was thinking it might be best to just check one entry per page call, and iterate through the rows/entires with each successive page call.
What would be the best way of doing this, ie keeping track of which entry/row in the table that the should be checked next?
I have 2 ideas of how to implement this:
1 ) The id of current row could be save to a file on the server [ surely not the best way]
2) an extra boolean field [check] is add to the table, set to True on the first entry and false to all other.
Then on each page call it;
finds 'where check = TRUE'
runs the update check on this row,
'set check = FALSE'
'set [the next row] check = TRUE'
Si this the best way to do this, or does anyone have any better sugestion ?
thanks in advance !
.k
PS sorry about the title
Not sure if this is a good solution, but if I have to make nightly massive updates, I'll write the updates to a new blank table, then do a SQL select to join the tables and tell me where they are different, then do another SQL UPDATE like
UPDATE table, temptable
SET table.col1=temptable.col1, table.col2=temptable.col2 ......
WHERE table.id = temptable.id;
You can store the timestamp that a row is updated implicitly using ON UPDATE CURRENT_TIMESTAMP [http://dev.mysql.com/doc/refman/5.0/en/timestamp.html] or explicitly in your update SQL. Then all you need to do is select the row(s) with the lowest timestamp (using ORDER BY and LIMIT) and you have the next row to process. So long as you ensure that the timestamp is updated each time.
e.g. Say you used the field last_polled_on TIMESTAMP to store the time you polled a row.
Your insert looks like:
INSERT INTO table (..., last_polled_on) VALUES (..., NOW());
Your update looks like:
UPDATE table SET ..., last_polled_on = NOW() WHERE ...;
And your select for the next row to poll looks like:
SELECT ... FROM table ORDER BY last_polled_on LIMIT 1;
Say you've got a database like this:
books
-----
id
name
And you wanted to get the total number of books in the database, easiest possible sql:
"select count(id) from books"
But now you want to get the total number of books last month...
Edit: but some of the books have been
deleted from the table since last month
Well obviously you cant total for a month thats already past - the "books" table is always current and some of the records have already been deleted
My approach was to run a cron job (or scheduled task) at the end of the month and store the total in another table, called report_data, but this seems clunky. Any better ideas?
Add a default column that has the value GETDATE(), call it "DateAdded". Then you can query between any two dates to find out how many books there were during that date period or you can just specify one date to find out how many books there were before a certain date (all the way into history).
Per comment: You should not delete, you should soft delete.
I agree with JP, do a soft delete/logical delete. For the one extra AND statement per query it makes everything a lot easier. Plus, you never lose data.
Granted, if extreme size becomes an issue, then yeah, you'll potentially have to start physically moving/removing rows.
My approach was to run a cron job (or scheduled task) at the end of the month and store the total in another table, called report_data, but this seems clunky.
I have used this method to collect and store historical data. It was simpler than a soft-delete solution because:
The "report_data" table is very easy to generate reports/graphs from
You don't have to implement special soft-delete code for anything that needs to delete a book
You don't have to add "and active = 1" to the end of every query that selects from the books table
Because the code to do the historical reporting is isolated from everything else that uses books, this was actually the less clunky solution.
If you needed data from the previous month then you should not have deleted the old data. Instead you can have a "logical delete."
I would add a status field and some dates to the table.
books
_____
id
bookname
date_added
date_deleted
status (active/deleted)
From there you would be able to query:
SELECT count(id) FROM books WHERE date_added <= '06/30/2009' AND status = 'active'
NOTE: It my not be the best schema, but you get the idea... ;)
If changing the schema of the tables is too much work I would add triggers that would track the changes. With this approach you can track all kinds of things like date added, date deleted etc.
Looking at your problem and the reluctance in changing the schema and the code, I would suggest you to go with your idea of counting the books at the end of each month and storing the count for the month in another table. You can use database scheduler to invoke a SP to do this.
You have just taken a baby step down the road of history databases or data warehousing.
A data warehouse typically stores data about the way things were in a format such that later data will be added to current data instead of superceding current data. There is a lot to learn about data warehousing. If you are headed down that road in a serious way, I suggest a book by Ralph Kimball or Bill Inmon. I prefer Kimball.
Here's the websites: http://www.ralphkimball.com/
http://www.inmoncif.com/home/
If, on the other hand, your first step into this territory is the only step you plan to take, your proposed solution is good enough.
The only way to do what you want is to add a column to the books table "date_added". Then you could run a query like
select count(id) from books where date_added <= '06/30/2009';