Short Story: I wrote a code in a loop that insert rows into mysql table, and kept it running for 3 hours accidentally. I tried to empty the mysql table, but then again it automatically starts to insert rows. I even tried to drop and re-create it, but it is automatically inserting rows (id in increasing order)
Full story:
2 days ago, I wrote a code that use an API and insert the data into mysql table (in a loop). And to mesure, how many records I have added, I made an table named "lastrequestdone" and started to insert the id number after every record inserted successfully.
But the code inserted all the ID's in 1 hour, and after that, the API started to give error:404 response, and the loop started to run very fast.
So assume that, I ran a loop for 3 hours, that insert an id in increasing order. But once i realized it, I stopped the loop. Tried to empty the table, but whenever I empty the table, more rows automatically starts to get inserted.
Then i dropped the table, and created it again today, but the same thing is happening. Rows are getting inserted automatically.
Look for the processes in your MySQL Server using any UI tools like MySQL Workbench or likes of it and try to find the process of the query that you left running and stop that particular process. You should probably resolve it this way.
Hope it might help you - https://dba.stackexchange.com/questions/63302/how-to-stop-the-execution-of-a-long-running-insert-query
Do you mean by "kill / stop the MySQL instance" the linux command kill -9 ...?
Yes, this might lead to corrupt data.
Which instead should not leave corrupt data is the build-in kill command of MySQL. See the lower part of the page, which states that it might take the thread some time to actually notice the presence of the kill flag. The part which states it does not roll back updates when transactions are not used (which implies that it rolls back updates if transactions are used), etc. I therefore assume this does not corrupt any data (besides killing repair and optimize as noted there).
You can find out the thread id you need for this command with the command show processlist.
Related
I am trying to migrate customers to a new database structure.
I have written a PHP script which loops through every record and determines if that customer has duplicates in the database or not. If there are multiple occurrences, I update all other tables linked to that customer, and update their linked customerID to the latest version of that customer.
The issue I am facing is that running this script by loading migrate.php in a browser takes so long that I get a timeout 504 when running it, even if I run the query for customers names starting with 'A'. Then change to 'B' and run the script, then 'C' and run the script etc... (which would take forever to do anyways).
So my question is: How can I run around 120,000 UPDATE queries either within the PHP loops in the migrate.php page and avoid a 504 timeout (currently set to limit of 5min), or by printing the Queries out, and running them through PHPMyAdmin?
I have printed out (snippet):
BEGIN;
UPDATE renewal_reminder SET renewalid='49884' WHERE renewalid='43239';
UPDATE renewal_note SET renewalid='49884' WHERE renewalid='43239';
UPDATE renewal_request SET renewalid='49884' WHERE renewalid='43239';
UPDATE renewal_timeline SET renewalid='49884' WHERE renewalid='43239';
COMMIT;
But this still kills PHPMyAdmin and returns an error.
Can this large list of UPDATES be used within PHPMyAdmin to run the script, maybe in batches?
UPDATE
Well...
I ended up indexing the customerID in each table that references it, and I ran my original script with the UPDATE queries and it did it in about 5 seconds...
Without the need to INSERT, then copy over the data.
So for anyone with the same problem, try indexing certain columns in your table so your script can reference them quickly.
We have an automatic car plate reader which records plates of the cars enter to firm. My colleague asked me if we can instantly get the plate number of the car coming. The software uses MySQL and I have only database access. Cannot reach/edit PHP codes.
My offer is to check using a query periodically. For example for 10 seconds. But in this way it is possible to miss the cars coming in 5 seconds. Then decreasing interval increases request/response count which means extra load for the server. I do not want the script to run always. It should run only a new db row added. It shows the plate and exits.
How can I get last recorded row from the db right after inserting? I mean there should be trigger which runs my PHP script after insertion. But I do not know.
What I want is MySQL could run my PHP script after a new record.
If your table is MyISAM, I would stick to your initial idea. Getting the row count from a MyISAM table is instant. It only takes the reading of one single value as MyISAM maintains the row count at all times.
With InnoDB, this approach can still be acceptable. Assuming car_table.id is primary key, SELECT COUNT(id) FROM car_table only requires an index scan, which is very fast. You can improve on this idea by adding another indexed boolean column to your table:
ALTER car_table ADD COLUMN checked BOOLEAN NOT NULL DEFAULT 0, ADD INDEX (checked);
The default value ensures new cars will be inserted with this flag set to 0 without modifying the inserting statement. Then:
BEGIN TRANSACTION; -- make sure nobody interferes
SELECT COUNT(checked) FROM car_table WHERE checked = FALSE FOR UPDATE; -- this gets you the number of new, unchecked cars
UPDATE car_table SET checked = TRUE WHERE checked = FALSE; -- mark these cars as checked
COMMIT;
This way, you only scan a very small number of index entries at each polling.
A more advanced approach consists in adding newly created cars ID's into a side table, through a trigger. This side table is scanned every now and then, without locking the main table, and without altering its structure. Simply TRUNCATE this side table after each polling.
Finally, there is the option of triggering a UDF, as suggested by Panagiotis, but this seems to be an overkill in most situations.
Although this is not the greatest of designs and I have not implemented it, there is way to call an external script through sys_exec() UDF using a trigger as mentioned here:
B.5.11: Can triggers call an external application through a UDF?
Yes. For example, a trigger could invoke the sys_exec() UDF.
http://dev.mysql.com/doc/refman/5.1/en/faqs-triggers.html#qandaitem-B-5-1-11
Also have a look on this thread which is similar to your needs.
Invoking a PHP script from a MySQL trigger
I am not an SQL magician so I'm venturing to ask for help. I have 4 tables to insert into a 5th one while checking a 6th table to ensure no duplicates. For example, no names in the 6th table can be inserted in the 5th one. I probably can try to figure out the best SQL query for the job but my head can't get around the right method? The final table size is small for now (5000 contact names), but will grow every month so I got to start right. I plan to use a PHP script with mysql connection to the database. This script will only run on my server (CenTOS 5).
Without seeing the schema, in general if you're going to prevent rows from entering tables based on other tables - in mySQL you'll need to utilize foreign keys. Overall, all of this will need to be done in a database transaction so that whatever logic you create in PHP to insert rows in various tables either succeed after total confirmation of success or fail and roll back to the prior state.
i have 2 sql tables of a script that i need to be sync to another, this can be done with php cron (this was my plan) exept from one row
Table 1 Table 2
row 1 <----> Row 1
Row 2 <----> row 2
row 3 no sync row 3
both databases on same server
and the same user has full rights for both
i am looking for a php code to do this via a cpanel cron
on an after thought would it be best to merge the two so both updates with new data?
the issue is that in the example above i am needing row 3 to not change on both databases
I am very noob so please be nice lol Thx in advance
Update *
i should learn how to explain a bit better.
both the databases are control panels for sites, one of the tables rows has the system url in it, so if i share the database "site 2" links refers back to "site 1" this is a complex problem for me as i am very new to this.
what i need is to keep both databases upto date except that single row which in turn be different for both databases.
i have not tried anything just yet as i wouldn't know where to start :( lol
You dont have to use cron. MySQL in current version supports TRIGGERS and EVENTS.
You can use TRIGGER to copy data to another table. That copy (or any other operation) may be triggered by some event (like insert, update or delete in table). TRIGGER may run any query or any PL/SQL code.
Other option is an EVENT. This is something like internal task sheduler built in MySQL. It can also run queries, or any PL/SQL code, but it is triggered by system time (like Linux Cron). It has many advantages compared to cron.
PL/SQL is procedural SQL, with loops, variables and more.
If you think you are "noob" - i have cure for you. Read books about MySQL or if you are lazy - watch some tutorials ( http://thenewboston.org , http://phpacademy.org ).
Nobody here will write code for you. We can only fix a bug, give advice etc. :)
EDIT.
Example of EVENT:
-- this is comment in SQL language (line starts with --)
CREATE EVENT event_daily_copy_something
ON SCHEDULE
EVERY 1 DAY
COMMENT 'This text will appear in MySQL Workbench as description of event'
DO
BEGIN
INSERT INTO your_db_name.target_table_name(id, field)
SELECT id, something
FROM your_db_name.source_table_name
WHERE id = 3;
END
Synchronization of tables is quite complicated. I think you need few operations in event.
Check for new rows and copy
Check for deleted rows and delete them in "copy" table
Check for changed rows (here trigger on source table would be very useful, because trigger "knows" what row is edited and you can access new field values in table 1 and use them to update table 2).
One of MySQL tutorials - thenewboston#youtube.
I have a scraper which visits many sites and finds upcoming events and another script which is actually supposed to put them in the database. Currently the inserting into the database is my bottleneck and I need a faster way to batch the queries than what I have now.
What makes this tricky is that a single event has data across three tables which have keys to each other. To insert a single event I insert the location or get the already existing id of that location, then insert the actual event text and other data or get the event id if it already exists (some are repeating weekly etc.), and finally insert the date with the location and event ids.
I can't use a REPLACE INTO because it will orphan older data with those same keys. I asked about this in Tricky MySQL Batch Query but if TLDR the outcome was I have to check which keys already exist, preallocate those that don't exist then make a single insert for each of the tables (i.e. do most of the work in php). That's great but the problem is that if more than one batch was processing at a time, they could both choose to preallocate the same keys then overwrite each other. Is there anyway around this because then I could go back to this solution? The batches have to be able to work in parallel.
What I have right now is that I simply turn off the indexing for the duration of the batch and insert each of the events separately but I need something faster. Any ideas would be helpful on this rather tricky problem. (The tables are InnoDB now... could transactions help solve any of this?)
I'd recommend starting with Mysql Lock Tables which you can use to prevent other sessions from writing to the tables whilst you insert your data.
For example you might do something similar to this
mysql_connect("localhost","root","password");
mysql_select_db("EventsDB");
mysql_query("LOCK TABLE events WRITE");
$firstEntryIndex = mysql_insert_id() + 1;
/*Do stuff*/
...
mysql_query("UNLOCK TABLES);
The above does two things. Firstly it locks the table preventing other sessions from writing to it until you the point where you're finished and the unlock statement is run. The second thing is the $firstEntryIndex; which is the first key value which will be used in any subsequent insert queries.