I have a upload location so users can update a portion of my database with an uploaded file. The files are often up to 9gb, so inserting the 150,000,000 lines can take a few minutes.
After clicking the button on the website to update the database, PHP (using mysqli) basically goes on mysql lock down. If I open other tabs, they get nothing until the large update is complete.
However, I know it's not actually locking the database/table, because from CLI i can still "SELECT count(*) FROM table" and it gives me a result right away.
What would be the best method of inserting 150,000,000 records while still letting other php pages access the db (for reading only)?
You can use "INSERT DELAYED". The DELAYED option for the INSERT statement is a MySQL extension to standard SQL that is very useful if you have clients that cannot or need not wait for the INSERT to complete. This is a common situation when you use MySQL for logging and you also periodically run SELECT and UPDATE statements that take a long time to complete.
You can read about this resource on the official documentation here.
;-)
The issue was with sessions.
Since the upload validated against the login in the session, and I failed to session_write_close() before starting the db writes the session file remained locked for the entire 9gb read/write to the db.
This explains why I could still use CLI mysql, and basic php (the basic php I was using to test had no sessions in it).
Related
Maybe this is an obvious question, but it's just something I'm unsure of. If I have two standalone PHP applications running on one LAMP server, and the two PHP applications share the same MySQL database, do I need to worry about data integrity during concurrent database transactions, or is this something that MySQL just takes care of "natively"?
What happens if the two PHP applications both try to update the same record at the same time? What happens if they try to update the same table at the same time? What happens if they both try to read data from the database at the same time? Or if one application tries to read a record at the same time as the other application is updating that record?
What happens if the two PHP applications both try to update the same record at the same time?
What happens if they try to update the same table at the same time?
What happens if they both try to read data from the database at the same time?
Or if one application tries to read a record at the same time as the other application is updating that record?
This depend from several factor ..
the db engine you are using
the locking policy / transaction you have setted for you envirement .. or for you query
https://dev.mysql.com/doc/refman/8.0/en/innodb-locking-reads.html
https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html
the code you are using .. you could use a select for update for lock only the rows you want modify
https://dev.mysql.com/doc/refman/8.0/en/update.html
and how you manage transaction
https://dev.mysql.com/doc/refman/8.0/en/commit.html
this is just a brief suggestion
When the web server receives a request for my PHP script, I presume the server creates a dedicated process to run the script. If, before the script exits, another request to the same script comes, another process gets started -- am I correct, or the second request will be queued in the server, waiting for the first request to exit? (Question 1)
If the former is correct, i.e. the same script can run simultaneously in a different process, then they will try to access my database.
When I connect to the database in the script:
$DB = mysqli_connect("localhost", ...);
query it, conduct more or less lengthy calculations and update it, I don't want the contents of the database to be modified by another instance of a running script.
Question 2: Does it mean that since connecting to the database until closing it:
mysqli_close($DB);
the database is blocked for any access from other software components? If so, it effectively prevents the script instances from running concurrently.
UPDATE: #OllieJones kindly explained that the database was not blocked.
Let's consider the following scenario. The script in the first process discovers an eligible user in the Users table and starts preparing data to append for that user in the Counter table. At this moment the script in the other process preempts and deletes the user from the Users table and the associate data in the Counter table; it then gets preempted by the first script which writes the data for the user no more existing. These data become in the head-detached state, i.e. unaccessible.
How to prevent such a contention?
In modern web servers, there's a pool of processes (or possibly threads) handling requests from users. Concurrent requests to the same script can run concurrently. Each request-handler has its own connection to the DBMS (they're actually maintained in a pool, but that's a story for another day).
The database is not blocked while individual request-handlers are using it, unless you block it explicitly by locking a table or doing a request like SELECT ... FOR UPDATE. For more information on this deep topic, read about transactions.
Therefore, it's important to write your database queries in such a way that they won't interfere with each other. For example, if you need to learn the value of an auto-incremented column right after you insert a row, you should use LAST_INSERT_ID() or mysqli_insert_id() instead of trying to query the data base: another user may have inserted another row in the meantime.
The system test discipline for scaled-up web sites usually involves a rigorous load test in order to shake out all this concurrency.
If you're doing a bunch of work on a particular entity, in your case a User, you use a transaction.
First you do
BEGIN
to start the transaction. Then you do
SELECT whatever FROM User WHERE user_id = <<whatever>> FOR UPDATE
to choose the user and mark that user's row as busy-being-updated. Then you do all the work you need to do to fill out various rows in various tables relating to that user.
Finally you do
COMMIT
If you messed things up, or don't want to go through with the change, you do
ROLLBACK
and all your changes will be restored to their state right before the SELECT ... FOR UPDATE.
Why does this work? Because if another client does the same SELECT .... FOR UPDATE, MySQL will delay that request until the first one either gets COMMIT or ROLLBACK.
If another client works with a different userid, the operations may proceed concurrently.
You need the InnoDB access method to use transactions: MyISAM doesn't support them.
Multiple reads can be done concurrently, if there is a write operation then it will block all other operations. A read will block all writes.
The hosting package im using wont allow sql and will have to pay more monthly if i want sql and its on a shared server so i cant have a client server set up for my app.
Im making a leaderboard for my first ever app.
so iv decided to do everything using php. the app will launch a link in the browser which will look something like this:
....../myAppLeaderBoard.php?opt=submit&?id=5465&name?='myName'&dat=DKGHKDHGKHDKGHSAJDHKJAHGJKHDFGHKJDFHGLKHDFGJHSDJLFGHJKSDHFGKJDSHFKGJHSLKDFHGLSJDHFGLJSHDFGJHSLDFJHGLSDJHFGLSDHJFGLSHDFGHG
All those alphabets is because i plan on using my own encrypting technique to prevent a user from cheating and giving themselves a highscore.
when its submitted it will read everything from a textfile into an array
it will check if the user exists and if they do it will change that record and if they dnt exist it will add a new record,then it will write everything from the array back to the textfile.
Then it will display a success message and show the user on the leaderboard,Now what will happen if say 100000 users each simultaneously submit there scores.
If it reads and writes one record at a time there wont be a problem but if it does this simultaneously then some records might be deleted by a simultaneous write.
So is it done simultaneously or one at a time?
Feel free to give suggestions for a better way to do this.
File access is simultaneous but you can use flock to block other handles from accessing the file while you perform your read/write operations. It sounds like the best solution to your problem would be to use PDO and SQLite (if available). This way you have the database handle all locking for you but do not need a dedicated database server. SQLite is entirely file based.
If SQLite is not available, you'll want to make use of flock's LOCK_EX operation, this will only allow one write stream to access the file at a time, e.g.
// create the file handle
$hnd = fopen($destpath, 'ab+');
if ($isReadOperation) {
// shared lock - simultaneous reads can happen at one time
flock($hnd, LOCK_SH);
// perform your read operation
} else {
// exclusive lock - only one write can occur at a time after all shared locks are released
flock($hnd, LOCK_EX);
// perform your read/write operation
}
// release the lock
flock($hnd, LOCK_UN);
// release the file handle
fclose($hnd);
File access (and this applies to sqlite databases as well, as they're file-based) is, unfortunately, not supposed to handle many simultaneous read & write operations. Therefore, you will run into problems with that.
I'm afraid your only sensible option is buying a hosting plan that offers a real database, e.g. MySQL.
I have a SugarCRM installation.
My problem is when I do a search in sugarcrm the search query block all other queries.
Id User Host db Command Time State Info
49498 sugar xx.xx.xx.xx:59568 sugarcrm Query 5 Sorting result SELECT leads.id ,leads_cstm.cedula_c,leads_cstm.numplanilla_c,leads_cstm.profession_c,leads_cstm.b
49502 sugar xx.xx.xx.xx:59593 sugarcrm Sleep 5 NULL
As you can see the query Id 49502 is, I presume, waiting for query 49498 to finalize.
first query is a search query which last a looong time to execute
second query is a query for the index page
The odd thing is that if I open two terminals and connect to mysql using the same user as my sugarcrm installation I can execute both queries concurrently but if I make a search in the browser and open a new tab and try to access the index page, that second tab hungs until the first tab completes execution or it get a timeout from server.
I have tested this using php both as a module and as cgi.
So I guess it should be something with the mysql_query function itself?
Any ideas? It's very hard to optimize the db (lots of tables, lots of content) but at least the site should be able to be used concurrently...
Probably because you're using file-based sessions. PHP will lock session files while they're in used, which essentially makes all requests by a particular session to become serial.
The only ways around this are to either use session_write_close() to release the session lock on a per-script basis, after any session-changing code is complete, or implementing your own session handler which can have its own locking logic.
We have a web application running on LAMP stack, the application depends on various services. These services gets data from cache (memcached) which is being refreshed using cron (from MySQL). Cron processes are running every 5 mins.
In this approach we can not serve data which is updated recently as cache is getting refreshed every 5 mins.
Is there any mechanism exists which can trigger cache refresh as soon as data gets updated in MySQL?
I don't know if this is the best solution, but what you can do is create MySQL trigger which gets executed on insert/update/delete.
Inside that MySQL trigger execute a UDF. From that UDF you can execute any PHP script using sys_exec().
Read about Triggers
Read about UDF
Read about using sys_exec() and more
My solution for this problem was to have the MySQL query in a function with the memcached query, for example (in Python, because I don't know PHP, you can look at it an change it to PHP):
def insert(user,value):
#execute on memcached first
key="user:"+user
memcClient.set(key,value)
#then execute it on MySQL
cursor.execute("INSERT INTO tables VALUES (%s,%s)",(user,value))
db.commit()
And you will do the same for delete and update, sorry I couldn't do it in PHP, I'm just a teen coder, good luck.
What about identifying the mysql updation at php level only & refresh the memcache accordingly ?
<?php
if($update_db){ // if we need to update the db then update db & memcacache together !
// Code to Update the database ....
// Code to Reset the memcached keyvalue pair.
}
?>
In the approach suggested by Manu, we are using expensive db trigger which is actually not needed to achieve the cache update.