HI everybody, i have a question。
A sqlite have about 15 tables,and mysql same.i want sqlite to sync mysql every hour,what can
i do?
i have a idea ,use a (script / php) to export sqlite'tables value and (send to mysql or sent url),and mysql use a (script / php) to update DB.but this way like not good.
or have some tool that i can use?
thanks all.
If you don't want to do full exports/imports each time, what you would have to do have a "last_modified" field in each table that contains a date for when that row was last modified. Then on each sync, save the current timestamp, and on the subsequent sync, issue a query like:
SELECT * FROM TABLE WHERE last_modified > $last_sync_timestamp
..and you get your list of updated rows.
Note that this won't "propagate" rows that get deleted from the tables. Instead of deleting those rows, what you'll need to do is have another field called "visibility" that is toggled to false when a record is deleted, and your application modified so that said rows are not shown. Then have your syncing code actually delete the "not visible" rows from the table after a certain number of syncs.
Hope this helps.
Related
I am going to set up a cron job to update some data via an API. I want it to update the database with the new feeds.
i.e. I would have an existing feed of entries, a script would go through the new feed. If the entry is already there, then dont update, if it is not in the db, then add it, and all other entries need to be deleted.
I was wondering what a good way to do this was have a column called "updated". This would be 0 by default. When a new entry is added, or an existing one is checked, the columns value becomes 1. Once the cron job has completed its updating, it would then remove all values that are still 0, and reset the remainder to 0.
Is this the right way to do such a job, if it helps there are over 10 million rows.
First of all there is no right or wrong answer and it always depends.
Now that being said with your approach you'll be updating all 10m+ rows in your main (target) table twice each time you do the sync up, which depending on how busy this table is may or may not be acceptable.
You may consider a different approach that is widely used in ETL:
load your feed data into a staging table first; do batch inserts or if possible use LOAD DATA INFILE - the fastest way of ingesting data in MySQL
optionally build indexes to help with lookups
"massage" your data if necessary (clean up, transform, augment etc)
insert into main table all new rows that present in staging and not in the main table
delete all rows from the main table that don't present in the staging table
truncate staging table
Every week I need to load 50K~200K rows of records from a raw CSV file to my system.
Currently I am solution is to load the CVS to a temp table(empty it after the process), then run my Stored procedure to manipulate the data to different relevant tables in my system. If records already exists will run update query (80% records in CSV are already in my system table), if not exists will Insert the records.
The problem i am facing now is the tables are growing to few millions records, approx. 5~6 millions each tables.
"Select Exist" seems very slow too, after that i change to left join tables by batch also slow.
Even I just loaded 5K records it may took about few hours to finish the Stored Procedure process.
Any good and faster solutions to handle huge records when comparing tables to decide insert/update records?
Thanks!!
Jack
Do the following process which will reduce your time
First try to update the record and check the number of rows affected if number of rows affected = 0 then insert record.
But make sure every time you need to modify the modified_Date if modified_Date not exist in table then you need to add that because if the all data are same in new and old record then it will create new query just because there is no modification in table record so it will return 0.
Slow responds of MySQL is almost always a problem of wrong indexing or uncorrect use of it.
If you use keys or/and index correct, a INSERT ... ON DUPLICATE KEY UPDATE ... should work.
Try to work only on an existing index/key. Check your statements with a EXPLAIN SELECT.
IMHO your tmp-table based preprocessing is ok.
Is their anyway of checking whether data has been entered into the table and then displaying the updated data as soon as it is updated ??
What I can understand is that you are working for a Client-Server Message project.
You need to create a column for the timestamps in MySQL named anything like 'timestamp' and everytime you enter a entry in the table, for example a chat table, you need to insert the timestamp there in the same query.
In php you can use time() to get the current timestamp. You can set your timezone using this : date_default_timezone_set(), PHP timezones.
And save a variable which saves the last time retreived from the database. Send that with the query to the server.
Return the client all the messages based on the timestamp.
Use mysqli_error() to check, if this returns false you know the INSERT ran as you intended.
mysqli_query($conn,"INSERT INTO mytable (`columnone`,`columntwo`,`columnthree`)
VALUES ('$valueone','$valuetwo','$valuethree')")
or die(mysqli_error($db));
Now run a SELECT query to find the item you just inserted, and return it. Simple!
If you want all this to happen without reloading the page, use AJAX and return the result of the SELECT query upon successful submission.
If I understand you correctly I think you need to store a timestamp for every update and insert (or one for both). Just add a column to each table (I usually call it 'ts'). You can even make it update automatically: https://dev.mysql.com/doc/refman/5.0/en/timestamp-initialization.html
Edit: then you can obviously use that column to select the most recently updated row, see if anything's changed in X amount of time etc etc
I have some issues with sugarcrm.
As you know that sugarcrm table, they do have ID (which is a unique string), they not run by sequential. e.g
4bab37e4-798a-e01c-75de-4e4397f358b7
For example, I would like to copy the table sugarcrm.accounts to something.accounts, in something.accounts I added some custom file on it for another PHP process usage.
Now the problem is, my sugarcrm table got huge records there, I plan to run it batch by batch, each time I would like to copy 10,000 records to my somthing.accounts.
However, sugarcrm.accounts, their ID, not in sequential, how do I know offset parameter?
I do not want to amend sugarcrm table / or add a temporary table in sugarcrm. (e.g sugarcrm.account_index), it might caused me having problem to do the upgrade.
So anyone have any idea, how do I get the index number? Is MySQL got hidden index?
Or anyone have better idea to do the database table copy another database table?
One way is the following:
- Select all rows from sugarcrm.accounts and order by date_created ascending.
- Use limit to only select a subset of the rows (store the offset from batch to batch)
- Copy the subset of rows to something.accounts
If new records are added later they will still be copied, since they will be last in the set. However, if you delete records in sugarcrm.accounts while running the batch jobs, then you need to change the offset as well, since you might omit some rows.
Another way, if the two databases/tables are in the same MySQL instance, is to join the two tables, and select the next 10.000 which doesn't exist in something.accounts.
my site has lots of incoming searches which is stored in a database to show recent queries into my website. due to high search queries my database is getting bigger in size. so what I want is I need to keep only recent queries in database say 10 records. this keeps my database small and queries will be faster.
I am able to store incoming queries to database but don't know how to restrict or delete excess/old data from table.
any help??
well I am using PHP and MySQL
Hopefully you have a timestamp column in your table (or have the freedom to add one). AFAIK, you have to add the timestamp explicitly when you add data to the table. Then you can do something along the lines of:
DELETE FROM tablename WHERE timestamp < '<a date two days in the past, or whatever'>;
You'd probably want to just do this periodically, rather than every time you add to the table.
I suppose you could also just limit the size to the most recent ten records by checking the size of the table every time you are about to add a line, and deleting the oldest record (again, using the timestamp column you added) if adding the new record will make it too large.
Falkon's answer is good - though you might not want to have your archive in a table, depending on your needs for that forensic data. You could also set up a cron job that just uses mysqldump to make a backup of the database (with the date in the filename), and then delete the excess records. This way you can easily make backups of your old data, or search it with whatever tool, and your database stays small.
You should write a PHP script, which will be started by CRON (ie. once a day) and move some data from main table TableName to archive table TableNameArchive with exactly the same structure.
That SQL inside the script should looks like:
INSERT INTO TableNameArchive
SELECT * FROM TableName WHERE data < '2010-06-01' //of course you should provide here your conditions
next you should DELETE old records from TableName.