I have an app which takes heartbeats (a simple http request) from hosts, typically the host generates one request every x minutes. This results in a large number of completely independent php pages runs which does a read queries then (possibly) generates one single row insert to the RDS database, which doesn't really matter if it succeeds (one missed beat isn't a reason for alarm, several are)
However, with mysqli I have a significant overhead in IOPs - it sends a BEGIN, my single line insert, then a COMMIT - and therefore appears to use three IOPs where I only need one.
Is there any way to avoid the transactions entirely? I could change auto_commit, but it's useless as each run of the handler is separate, so there is no other insert to group with this one. Even turning auto_commit off still runs a transaction, but only ends it when the connection closes (which happens after one insert anyway.)
Or should I switch to raw mysql handling for efficiency (lots of work)? The old mysql php library (deprecated)? Something else?
If you really don't need transactions you can use MyISAM storage engine for your tables that doesn't support transactions.
I use always :
SET AUTOCOMMIT=1
for my InnoDB databases where I don't need use transactions just after connecting to database.
Related
It is said on the PHP docs site that:
while the transaction is active, you are guaranteed that no one else
can make changes while you are in the middle of your work
ref: http://php.net/manual/en/pdo.transactions.php
The question is: if I start transaction in Script A and then Script B tries to makes any changes before Script A commits -- what would be the outcome for the Script B? Will it "wait"? Will it fail with error? What exactly?
Transactions are controlled at the DBMS level, not by PHP. So if something is blocked, it's done by your DBMS. This varies, because different database engines block at different levels. For example, MySQL MyISAM engine will lock at the table level whereas the MySQL InnoDB engine can block at row-level. Locking can also happen at a per-session or global basis depending on how you configure your DBMS and how the transaction is performed. PDO only implements the driver for your vendor-specific DBMS so it has no real say in all of that.
I am using MariaDB in a PHP application. The problem is the following: using Doctrine DBAL with the MySQL adaptor I do an insert from one page and then redirect to another one, in which a SELECT is done. Both are very basic queries.
The problem is that the SELECT does not reflect the actual data, but older one. I am hosting this application on a shared hosting, so please consider that I won't have all DB configuration options/permissions available.
I have tried to flush after the first INSERT, but it does not work either, and it still shows outdated data. I believed that the Query Caches are invalidated if the data changes, and that they do not apply because, in fact, it is a different query.
I do not use transactions either, so the commit is supposedly done after the insert. Any idea on how to get the most recent data possible?
It sounds like you are doing Replication and the proxy for directing queries is oblivious to "Critical Reads".
In a replication setup (MariaDB or MySQL), there is one Master server and one Slave (at least). INSERTs (and other writes) must occur on the Master. Then they are replicated to the Slave(s). SELECTs, on the other hand, may be done on either server, but, for sharing the load, it is better to do them on the Slave(s).
Replication is "asynchronous". That is, the write is eventually sent to the Slave and performed there. Normally, the delay is sub-second. But, for a number of reasons, the delay could be arbitrarily large. One should not depend on how quickly writes are replicated.
But... There is a thing called a "Critical Read". This is when the SELECT needs to "see" the thing that was just written.
You have a "critical read".
I don't know what is deciding to direct your SELECT to a Slave.
If you are using the Galera clustering option of MariaDB, then you can protect yourself from the critical read problem by changing your select to
SET SESSION wsrep_sync_wait = 1;
SELECT ... (as before)
SET SESSION wsrep_sync_wait = 0;
However; the SETs must go to the same 'node' as the SELECT. Without knowing what kind of proxying is going on, I cannot be more specific.
I hope you are not reconnecting before each statement. That would be really bad.
More on Galera issues for developers
If you are using replication and Doctrine DBAL has nothing for critical reads, complain to them!
I have a rapidly growing, write-heavy PHP/MySql application that inserts new rows at a rate of a dozen or so per second into an INNODB table of several million rows.
I started out using realtime INSERT statements and then moved to PHP's file_put_contents to write entries to a file and LOAD DATA INFILE to get the data into the database. Which is the better approach?
Are there any alternatives I should consider? How can I expect the two methods to handle collisions and increased load in the future?
Thanks!
Think of LOAD DATA INFILE as a batch-method of inserting data. It eliminates the overhead of firing up an insert query for every statement therefore is much faster. However, you lose some of the control when handling errors. It's much easier to handle an error on a single insert query vs one row in the middle of a file.
Depending on whether you can afford to have the data inserted by the PHP not being instantly available in the table, then INSERT DELAYED might be an option.
MySQL will accept the data to be inserted and will deal with the insertion later on, putting it into a queue. So this won't block your PHP application while MySQL ensures the data to be inserted later on.
As it says in the manual:
Another major benefit of using INSERT DELAYED is that inserts from many clients are bundled together and written in one block. This is much faster than performing many separate inserts.
I have used this for logging data where a data loss is not fatal but if you want to be protected from server crashes when data from INSERT DELAYED hadn't been inserted yet, you could look into replicating the changes away to a dedicated slave machine.
The way we deal with our inserts is to have them sent to a message queue system like ActiveMQ. From there we have a separate application that loads the inserts using LOAD DATA INFILE in batches of about 5000. Error handling can still take place with the infile however it processes the inserts much faster. If setting up a message queue is outside of the scope of your application there is no reason that file_put_contents would not be an acceptable option -- Especially if it's already implemented and is working fine.
Additionally you may want to test disabling indexes during writes to see if that improves performance.
It doesn't sound like you should be using innoDB. Regardless, a dozen inserts per second should not be problematic even for crappy hardware - unless, possibly, your data model is very complex, but for that, LOAD DATA INFILE is very good because, among other things, it rebuilds the indexes only once, as opposed to on every insert. So using files is a decent approach, but do make sure you open them in append only mode.
in the long run (1k+ of writes/s), look at other databases - particularly cassandra for write heavy applications.
if you do go the sql insert route, wrap the pdo execute statements in a transaction. doing so will greatly speed up the process.
LOAD DATA is disabled on some servers for security reasons:
http://dev.mysql.com/doc/mysql-security-excerpt/5.0/en/load-data-local.html
Also I don't enjoy writing my applications upside down to maintain database integrity.
I'm using ajax, and quite often if not all the time, the first request is timed out. In fact, if I delay for several minutes before making a new request, I always have this issue. But the subsequent requests are all OK. So I'm guessing that the first time used a database connect that is dead. I'm using MySQL.
Any good solution?
Can you clarify:
are you trying to make a persistent connection?
do basic MySQL queries work (e.g. SELECT 'hard-coded' FROM DUAL)
how long does the MySQL query take for your ajax call (e.g. if you run it from a mysql command-line or GUI client.)
how often do you write to the MySQL tables used in your AJAX query?
Answering those questions should help rule-out other problems that have nothing to do with making a persistent connection: basic database connectivity, table indexing / slow-running SQL, MySQL cache invalidation etc.
chances are that you problem is NOT opening the connection, but actually serving the request.
subsequent calls are fast because of the mysql query cache.
what you need to do is to look for slow mysql queries, for example by turning on the slow query log, or by looking at the server at real time using mytop or "SHOW PROCESSLIST" to see if there is a query that takes too long. if you found one, use EXPLAIN to make sure it's properly indexed.
I have a particular PHP page that, for various reasons, needs to save ~200 fields to a database. These are 200 separate insert and/or update statements. Now the obvious thing to do is reduce this number but, like I said, for reasons I won't bother going into I can't do this.
I wasn't expecting this problem. Selects seem reasonably performant in MySQL but inserts/updates aren't (it takes about 15-20 seconds to do this update, which is naturally unacceptable). I've written Java/Oracle systems that can happily do thousands of inserts/updates in the same time (in both cases running local databases; MySQL 5 vs OracleXE).
Now in something like Java or .Net I could quite easily do one of the following:
Write the data to an in-memory
write-behind cache (ie it would
know how to persist to the database
and could do so asynchronously);
Write the data to an in-memory cache
and use the PaaS (Persistence as a
Service) model ie a listener to the
cache would persist the fields; or
Simply start a background process
that could persist the data.
The minimal solution is to have a cache that I can simply update, which will separately go and upate the database in its own time (ie it'll return immediately after update the in-memory cache). This can either be a global cache or a session cache (although a global shared cache does appeal in other ways).
Any other solutions to this kind of problem?
mysql_query('INSERT INTO tableName VALUES(...),(...),(...),(...)')
Above given query statement is better. But we have another solution to improve the performance of insert statement.
Follow the following steps..
1. You just create a csv(comma separated delimited file)or simple txt file and write all the data that you want to insert using file writing mechanism (like FileOutputStream class in Java).
2. use this command
LOAD DATA INFILE 'data.txt' INTO TABLE table2
FIELDS TERMINATED BY '\t';
3 if you are not clear about this command then follow the link
You should be able to do 200 inserts relatively quickly, but it will depend on lots of factors. If you are using a transactional engine and doing each one in its own transaction, don't - that creates way too much I/O.
If you are using a non-transactional engine, it's a bit trickier. Using a single multi-row insert is likely to be better as the flushing policy of MySQL means that it won't need to flush its changes after each row.
You really want to be able to reproduce this on your production-spec development box and analyse exactly why it's happening. It should not be difficult to stop.
Of course, another possibility is that your inserts are slow because of extreme sized tables or large numbers of indexes - in which case you should scale your database server appropriately. Inserting lots of rows into a table whose indexes don't fit into RAM (or doesn't have RAM correctly configured to be used for caching those indexes) generally gets pretty smelly.
BUT don't try to look for a way of complicating your application when there is a way of easily turning it instead, keeping the current algorithm.
One more solution that you could use (instead of tuning mysql :) ) is to use some JMS server and STOMP connection driver for PHP for write data to database server in a asynchronous manner. ActiveMQ have built-in support for STOMP protocol. And there is StompConnect project which is STOMP proxy for any JMS compilant server (OpenMQ, JBossMQ etc).
You can update your local cache (hopefully memcached) and then push the write requests through beanstalkd.
I would suspect a problem with your SQL inserts - it really shouldn't take that long. Would prepared queries help? Does your mysql server need some more memory dedicated to the keyspace? I think some more questions need asked.
How are you doing the inserts, are you doing one insert per record
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
or are you using a single query
mysql_query('INSERT INTO tableName VALUES(...),(...),(...),(...)');
The later of the two options is substantially faster, and from experience the first option will cause it to take much longer as PHP must wait for the first query to finish before moving to the second and so on.
Look at the statistics for your database while you do the inserts. I'm guessing that one of your updates locks the table and therefor all your statements are queued up and you experience this delay. Another thing to look into is your index creation/updating because the more indices you have on a table, the slower all UPDATE and INSERT statements get.
Another thing is that I think you use MYISAM (table engine) which locks the entire table on UPDATE.I suggest you use INNODB instead. INNODB is slower on SELECT-queries, but faster on INSERT and UPDATE because it only locks the row it's working on and not the entire table.
consider this:
mysql_query('start transaction');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('INSERT INTO tableName VALUES(...)');
mysql_query('commit;')
Note that if your table is INSERT-ONLY (no deletes, and no updates on variable-length columns), then inserts will not lock or block reads when using MyISAM.
This may or may not improve insert performance, but it could help if you are having concurrent insert/read issues.
I'm using this, and only purging old records daily, followed by 'optimize table'.
you can use CURL with PHP to do Asynchronous database manipulations.
One possible solution is fork each query into a separate thread but, PHP doesnot support threads. We can use PCNTL functions but it’s a bit tricky for me to use them. I prefer to use this another solution to create fork and perform asynchronous operations.
Refer this
http://gonzalo123.wordpress.com/2010/10/11/speed-up-php-scripts-with-asynchronous-database-queries/