I have looked through a TON of the mysql view examples so I am guessing what I am trying to do may not be possible.
I have multiple databases : db1, db2, db3 .......
Each have a table: quotes
Columns in identical tables: DateSubmitted DATETIME, TimeFinished DATETIME, Status VARCHAR(64)
I am trying to get a view that I would create or re-create as new databases are added that can do some calculations between the DateSubmitted and TimeFinished based on Status (Error, TimeOut, Success)
The result I am looking for would be something like this:
Database|AvgTimeLast24Hours|AvgTimeLastWeek|AvgTimeLastMonth
|db1|60|48 | 40 |
| db2 | 185 | 125 | 105
|db3 | 14 | 18 | 23 |
The average columns would be in minutes, I have the queries to get the calculations but when I try to put it into a view I know I am doing it wrong.
Does anyone have any examples?
You have a query to generate the result set you want, presumably from one of your multiple databases. Let's say it's SELECT SUM(foo) foo, baz FROM quotes ORDER BY baz . (I know it isn't, but you didn't show your query.)
Then, in that same database you can create this as a view, easily.
CREATE VIEW summary
AS
SELECT SUM(foo) foo, baz FROM quotes ORDER BY baz;
Get that view working properly.
Then you can create another view taking the union of the tables in your various databases.
CREATE VIEW all_quotes AS
SELECT * FROM db1.quotes
UNION ALL SELECT * FROM db2.quotes
UNION ALL SELECT * FROM db3.quotes
/* etc etc ad nauseam */
Then change your first view to reference all_quotes instead of quotes. Of course, this only works if all the databases are on the same server (or if you do something with remote table references, which you can look up).
You can't use variables in queries for database or table names. Those must be constant text. Raymond is correct when he says you won't be able to write a query that successfuly refers to the union of all the databases in your system. You'll have to bootstrap that operation.
write a query (using information_schema.TABLES, probably) to write your CREATE VIEW AS... query, or some other query referencing all your databases.
run that query you made.
You can create and run it either in a php program, or using MySQL's server-side flavor of prepared statements (a different flavor from the prepared statements in mysqli or PDO).
The trick is to get things working stage by stage.
Pro tip: A separate database for each customer is notoriously hard to scale up. The more successful you become, the harder it gets. That's not good.
As I am using InnoDB as a database engine, the query goes slower sometimes it takes 20 seconds or more often.
I know the solution it can be done via my.conf to change the value of innodb_flush_log_at_trx_commit to 2 it can solve my problem I also want to do that but as I have shared hosting so they are not allowing me to do that.
MySQL version: 5.6.32-78.1
I also tried with MySQL query
mysql> SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';
+--------------------------------+-------+
| Variable_name | Value |
+--------------------------------+-------+
| innodb_flush_log_at_trx_commit | 1 |
+--------------------------------+-------+
1 row in set
And i tried this query
mysql> SET GLOBAL innodb_flush_log_at_trx_commit=2;
but it is also not allowing me too because I don't have super privileges to perform this action.
I have database with 25 tables, and in 4 tables there are 4000+ records and in rest tables, there are below 100 records
So is there any other solution to speed up the query performance.? Any help will be appreciated.
Use profile to check the cost time in each step;
set profiling=1;
Run you query;
Check the query:show profiles;
List the time cost: show profile block io,cpu for query N;
Find the step which has the high Duration
It shows like this
Possible problem: Index, Order by, File sort, Use temp table..
I have a performance and best-practice question concerning mysql tables.
I´m working on an application which connects to a database which gets filled by other programms.
This system is deployed in differnet locations, and from location to location the name of some databases-tables can change (but the fields in this tables etc stay the same).
As I don´t want to change all sql querys in my application for every location, I thought about creating a mysql view which simply mirrors the contents of this table to the normaly used table-name.
Is this a suitable solution, or could it get awfully slow with big tables?
Simple views (created as SELECT * FROM table) behave like the specified table performance wise.
It should be a suitable solution for your case.
mmm, this is tricky. If there are multiple tables then a quick and dirty version for this would be something like
SELECT * FROM (SELECT * FROM table1
Union
SELECT * FROM table2
Union
SELECT * FROM table3) t
Which I think will work. You will of course have problems with pagination, sorting and searching - because you will have to try and do this over 3 or more tables.
Another way would be this
Create a table with the table names and a counter
ImportTable
name
id
Now in this you can enter the names of the tables and the last id that you want to import from.
Create another table to import the records
TableRecords
source
id
field1
field2
etc
Now run something that goes through the tables in ImportTable grabs any new records and shoves them into `TableRecords.
Now this becomes really simply you can query TableRecords and have pagination sorting and searching with no of the previous troubles.
Make something that runs this every 2 minutes say so TableRecords will be 2 mins behind but everything will be really easy and run like a dream.
I have a PHP script that retrieves rows from a database and then performs work based on the contents. The work can be time consuming (but not necessarily computationally expensive) and so I need to allow multiple scripts to run in parallel.
The rows in the database looks something like this:
+---------------------+---------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+---------------+------+-----+---------------------+----------------+
| id | bigint(11) | NO | PRI | NULL | auto_increment |
.....
| date_update_started | datetime | NO | | 0000-00-00 00:00:00 | |
| date_last_updated | datetime | NO | | 0000-00-00 00:00:00 | |
+---------------------+---------------+------+-----+---------------------+----------------+
My script currently selects rows with the oldest dates in date_last_updated (which is updated once the work is done) and does not make use of date_update_started.
If I were to run multiple instances of the script in parallel right now, they would select the same rows (at least some of the time) and duplicate work would be done.
What I'm thinking of doing is using a transaction to select the rows, update the date_update_started column, and then add a WHERE condition to the SQL statement selecting the rows to only select rows with date_update_started greater than some value (to ensure another script isn't working on it). E.g.
$sth = $dbh->prepare('
START TRANSACTION;
SELECT * FROM table WHERE date_update_started > 1 DAY ORDER BY date_last_updated LIMIT 1000;
UPDATE table DAY SET date_update_started = UTC_TIMESTAMP() WHERE id IN (SELECT id FROM table WHERE date_update_started > 1 DAY ORDER BY date_last_updated LIMIT 1000;);
COMMIT;
');
$sth->execute(); // in real code some values will be bound
$rows = $sth->fetchAll(PDO::FETCH_ASSOC);
From what I've read, this is essentially a queue implementation and seems to be frowned upon in MySQL. All the same, I need to find a way to allow multiple scripts to run in parallel, and after the research I've done this is what I've come up with.
Will this type of approach work? Is there a better way?
I think your approach could work, as long as you also add some kind of identifier to the rows you selected that they are currently been worked on, it could be as #JuniusRendel suggested and i would even think about using another string key (random or instance id) for cases where the script resulted in errors and did not complete gracefully, as you will have to clean these fields once you updated the rows back after your work.
The problem with this approach as i see it is the option that there will be 2 scripts that run at the same point and will select the same rows before they were signed as locked. here as i can see it, it really depends on what kind of work you do on the rows, if the end result in these both scripts will be the same, i think the only problem you have is for wasted time and server memory (which are not small issues but i will put them aside for now...). if your work will result in different updates on both scripts your problem will be that you could have the wrong update at the end in the TB.
#Jean has mentioned the second approach you can take that involves using the MySql locks. i am not an expert of the subject but it seems like a good approach and using the 'Select .... FOR UPDATE' statement could give you what you are looking for as you could do on the same call the select & the update - which will be faster than 2 separate queries and could reduce the risk for other instances to select these rows as they will be locked.
The 'SELECT .... FOR UPDATE' allows you to run a select statement and lock those specific rows for updating them, so your statement could look like:
START TRANSACTION;
SELECT * FROM tb where field='value' LIMIT 1000 FOR UPDATE;
UPDATE tb SET lock_field='1' WHERE field='value' LIMIT 1000;
COMMIT;
Locks are powerful but be careful that it wont affect your application in different sections. Check if those selected rows that are currently locked for the update, are they requested somewhere else in your application (maybe for the end user) and what will happen in that case.
Also, Tables must be InnoDB and it is recommended that the fields you are checking the where clause with have a Mysql index as if not you may lock the whole table or encounter the 'Gap Lock'.
There is also a possibility that the locking process and especially when running parallel scripts will be heavy on your CPU & memory.
here is another read on the subject: http://www.percona.com/blog/2006/08/06/select-lock-in-share-mode-and-for-update/
Hope this helps, and would like to hear how you progressed.
We have something like this implemented in production.
To avoid duplicates, we do a MySQL UPDATE like this (I modified the query to resemble your table):
UPDATE queue SET id = LAST_INSERT_ID(id), date_update_started = ...
WHERE date_update_started IS NULL AND ...
LIMIT 1;
We do this UPDATE in a single transaction, and we leverage the LAST_INSERT_ID function. When used like that, with a parameter, it writes in the transaction session the parameter that, in this case, it's the ID of the single (LIMIT 1) queue that has been updated (if there is one).
Just after that, we do:
SELECT LAST_INSERT_ID();
When used without parameter, it retrieves the previously stored value, obtaining the queue item's ID that has to be performed.
Edit: Sorry, I totally misunderstood your question
You should just put a "locked" column on your table put the value to true on the entries your script is working with, and when it's done put it to false.
In my case i have put 3 other timestamp (integer) columns: target_ts , start_ts , done_ts.
You
UPDATE table SET locked = TRUE WHERE target_ts<=UNIX_TIMESTAMP() AND ISNULL(done_ts) AND ISNULL(start_ts);
and then
SELECT * FROM table WHERE target_ts<=UNIX_TIMESTAMP() AND ISNULL(start_ts) AND locked=TRUE;
Do your jobs and update each entry one by one (to avoid data inconcistencies) setting the done_ts property to current timestamp (you can also unlock them now). You can update target_ts to the next update you wish or you can ignore this column and just use done_ts for your select
Each time the script runs I would have the script generate a uniqid.
$sctiptInstance = uniqid();
I would add a script instance column to hold this value as a varchar and put an index on it. When the script runs I would use select for update inside of a transaction to select your rows based on whatever logic, excluding rows with a script instance, and then update those rows with the script instance. Something like:
START TRANSACTION;
SELECT * FROM table WHERE script_instance = '' AND date_update_started > 1 DAY ORDER BY date_last_updated LIMIT 1000 FOR UPDATE;
UPDATE table SET date_update_started = UTC_TIMESTAMP(), script_instance = '{$scriptInstance}' WHERE script_instance = '' AND date_update_started > 1 DAY ORDER BY date_last_updated LIMIT 1000;
COMMIT;
Now those rows will be excluded from other instances of the script. Do you work, and then update the rows to set the script instance back to null or blank, and also update your date last updated column.
You could also use the script instance to write to another table called "current instances" or something like that, and have the script check that table to get a count of running scripts to control the number of concurrent scripts. I would add the PID of the script to the table as well. You could then use that information to create a housekeeping script to run from cron periodically to check for long running or rogue processes and kill them, etc.
I have a system working exactly like this in production. We run a script every minute to do some processing, and sometimes that run can take more than a minute.
We have a table column for status, which is 0 for NOT RUN YET, 1 for FINISHED, and other value for under way.
The first thing the script does is to update the table, setting a line or multiple lines with a value meaning that we are working on that line. We use getmypid() to update the lines that we want to work on, and that are still unprocessed.
When we finish the processing, the script updates the lines that have the same process ID, marking them as finished (status 1).
This way we avoid each of the scripts to try and process a line that is already under processing, and it works like a charm. This doesn't mean that there isn't a better way, but this does get the work done.
I have used a stored procedure for very similar reasons in the past. We used the FOR UPDATE read lock to lock the table while a selected flag was updated to remove that entry from any future selects. It looked something like this:
CREATE PROCEDURE `select_and_lock`()
BEGIN
START TRANSACTION;
SELECT your_fields FROM a_table WHERE some_stuff=something
AND selected = 0 FOR UPDATE;
UPDATE a_table SET selected = 1;
COMMIT;
END$$
No reason it has to be done in a stored procedure though now I think about it.
I have the following table structure:
----------------------------------
ID | Acces | Group | User | Gate |
----------------------------------
1 | 1 | TR | tsv | TL-23|
----------------------------------
And I have a page with 3 functions:
Select group to see all gates where selected group has access.
Select gate to see all groups which have access to selected gate.
Select group to see all users that belong to selected group.
So basically:
SELECT Gate WHERE Group = TR
SELECT Group WHERE Gate = TL-23
SELECT User WHERE Group = TR
What I am trying to achieve is: The user should be able to run the all three queries in any order without the results of the former queries dissappearing.
Now, I know multi-threading is no longer possible in PHP, but there must be a way to temporarily save the results of a specific query until the same query is made again.
Any help and suggestions would be appreciated.
Firstly, PHP has never done MT out of the box, but can (still) with the use of pcntl extensions.
That said, that isn't the sauce you seek.
If you simply want the user on the front-end to interact 3 separate times, without having to hit the DB once the first time, twice the second (to redo both the first and the new query) you may benifit from caching the results of each call in the user's session.
If you actually want to make the 3 queries in a relational way at one exact time, try JOINs.
If you simply want to make all 3 separate queries at (what is essentially) the same time, look into TRANSACTIONs.
Hope that helps.