I have a problem with the time to query my database with PHP. I recently started creating an MMO RPG ONLINE using the program engine001, I also use PHP and MySQL to query info of my database but it takes too long to make a query (it usually takes one sec to query).
Steps:
First of all I query a webpage of my website which has the .php file (site.com/file.php)
This php file, will query into my database
This is printed (echo) at the page so my game pick the value e then make whatever it wants.
Here we have an example, this PHP file will send the new position of an actor:
include("mysqlconfig.inc");
$id = $_REQUEST['id'];
$positionX = $_REQUEST['positionX'];
$positionY = $_REQUEST['positionY'];
$query = "UPDATE players SET positionX = '$positionX',positionY = '$positionY' WHERE ID = $id ";
$res = mysql_query($query);
mysql_close($con);
I have no idea how online games are made and why they are so fast but this is how I do mine. It's the only way I figured out how to do that.
My question is, is there any way to make it a little bit faster?
I don't think this approach will ever be fast enough for a real time game. As you have discovered, making database queries for things like player positions is far too slow. Real time games do not typically use web-based backends because HTTP is a stateless protocol, but you need to have the game state persist in the process's memory to make it fast enough. So you will probably have to look at writing the backend as a custom server using sockets.
Related
I have a JS script that does one simple thing - an ajax request to my server. On this server I establish a PDO connection, execute one prepared statement:
SELECT * FROM table WHERE param1 = :param1 AND param2 = :param2;
Where table is the table with 5-50 rows, 5-15 columns, with data changing once each day on average.
Then I echo the json result back to the script and do something with it, let's say I console log it.
The problem is that the script is run ~10,000 times a second. Which gives me that much connections to the database, and I'm getting can't connect to the database errors all the time in server logs. Which means sometimes it works, when DB processes are free, sometimes not.
How can I handle this?
Probable solutions:
Memcached - it would also be slow, it's not created to do that. The performance would be similar, or worse, to the database.
File on server instead of the database - great solution, but the structure would be the problem.
Anything better?
For such a tiny amount of data that is changed so rarely, I'd make it just a regular PHP file.
Once you have your data in the form of array, dump it in the php file using var_export(). Then just include this file and use a simple loop to search data.
Another option is to use Memcached, which was created exactly this sort of job and on a fast machine with high speed networking, memcached can easily handle 200,000+ requests per second, which is high above your modest 10k rps.
You can even eliminate PHP from the tract, making Nginx directly ask Memcached for the stored valaues, using ngx_http_memcached_module
If you want to stick with current Mysql-based solution, you can increase max_connections number in mysql configuration, however, making it above 200 would may require some OS tweaking as well. But what you should not is to make a persistent connection, that will make things far worse.
You need to leverage a cache. There is no reason at all to go fetch the data from the database every time the AJAX request is made for data that is this slow-changing.
A couple of approaches that you could take (possibly even in combination with each other).
Cache between application and DB. This might be memcache or similar and would allow you to perform hash-based lookups (likely based on some hash of parameters passed) to data stored in memory (perhaps JSON representation or whatever data format you ultimately return to the client).
Cache between client and application. This might take the form of web-server-provided cache, a CDN-based cache, or similar that would prevent the request from ever even reaching your application given an appropriately stored, non-expired item in the cache.
Anything better? No
Since you output the same results many times, the sensible solution is to cache results.
My educated guess is your wrong assumption -- that memcached is not built for this -- is based off you planning on storing each record separately
I implemented a simple caching mechanism for you to use :
<?php
$memcached_port = YOUR_MEMCACHED_PORT;
$m = new Memcached();
$m->addServer('localhost', $memcached_port);
$key1 = $_GET['key1'];
$key2 = $_GET['key2'];
$m_key = $key1.$key2; // generate key from unique values , for large keys use MD5 to hash the unique value
$data = false;
if(!($data = $m->get($m_key))) {
// fetch $data from your database
$expire = 3600; // 1 hour, you may use a unix timestamp value if you wish to expire in a specified time of day
$m->set($m_key,$data,$expire); // push to memcache
}
echo json_encode($data);
What you do is :
Decide on what signifies a result ( what set of input parameters )
Use that for the memcache key ( for example if it's a country and language the key would be $country.$language )
check if the result exists:
if it does pull the data you stored as an array and output it.
if it doesn't exist or is outdated :
a. pull the data needed
b. put the data in an array
c. push the data to memcached
d. output the data
There are more efficient ways to cache data, but this is the simplest one, and sounds like your kind of code.
10,000 requests/second still don't justify the effort needed to create server-level caching ( nginx/whatever )
In an ideally tuned world a chicken with a calculator would be able to run facebook .. but who cares ? (:
I have a PHP page that could potentially result in a lot of mySQL queries depending on what the user searches. For searches that will take a long time, I want to show the user that something is happening, so they don't think the page is broken or hanging. I can't make a progress bar because I do not know how many queries will be executed every time. It depends on the search. I was thinking of having a number on the side of the page that would correspond to the number of queries and this number would increase continuously until the search is over.
This is how I am executing my queries.
$query = "SELECT name, id, direct_owner FROM $table WHERE parent_id = '$ownerID'";
$result = mysqli_query($connection, $query);
I am submitting a form for each search and then I use
if(isset($_POST['submit'])
PHP is server-side, it will not be able to show anything in realtime, you'd need client-side like Javascript or Angular or Jquery for any frameworks.
I "lied" a bit, you can do real-time, but it's very hackish. You can read about flush() here. I wouldn't recommend it, and regardless, you can't do a progress bar with just PHP.
What you should do is create this PHP file as a postRequest type and query it using AJAX, which is capable of doing live-output and a progress bar. There are multiple libraries out there to help you.
I have an Embedded system, i.e. basically an ATMEGA based microcontroller with GSM Module. The GSM module exploits GPRS connection of the SIM to send GET request to a Webpage on my server. In simpler words, it is same as a person opening that webpage from his mobile device.
When that webpage is opened, nothing special happens, I just extract the GET parameters and update the database. Now the problem comes. The database is online on a GoDaddy Server and when I send that update request from GSM device, it hangs for 4-5 seconds. Is there any other way by which I can update database and save my time ?
Moreover, I would like to know, for online database, what takes more time,
* Initiating a database connection, or
* using an UPDATE query to update the table ?
There are a lot of things going on here and you may have issues in many places. A bit too little info to solve the issue but here are some possible places to look.
Obviously, you have the issue of network latency and general response time from your web/database server(s) on GoDaddy. My first question would be how does a response from the MC compare to a get call via a web pages.
To specifically answer your question - initiating a database connection is usually the most costly part of the transaction. I am not sure what you are using on the database side so I cannot point you to specific resources. I am guessing MySQL? If so take a peek at https://dba.stackexchange.com/questions/16969/how-costly-is-opening-and-closing-of-a-db-connection for suggestions. On my own database I tend to tune them for performance. On GoDaddy you may be quite limited in what you can do.
However, I am going to qualify what I said above a little bit. Generally an update to a database should not be that slow. We could be dealing with poor database design or very large tables that have to have indexes updates as well. Again something to think about in your particular case. The other item to note is that you may be doing updates as shown below:
update myTable set myField = 1 where somesensor = 'a';
update myTable set myField = 1 where somesensor = 'b';
update myTable set myField = 1 where somesensor = 'c';
.....
and depending on the number of updates you are doing, how you are making the connection, etc. and the rest of your particular situation..... If you are using MySQL take a look at this example How to bulk update mysql data with one query? for possible ideas. Benchmark this!
I would suggest doing an explain plan to see what is happening to see if you can id where the problem is (check your version of MySQL). See http://dev.mysql.com/doc/refman/5.7/en/explain.html for syntax, etc.
There really is not enough info to say exactly but maybe this will give you some ideas. Good luck!
hi i created a simple php/mysql/Ajax chat application and I have a few questions. before that let me explain how it works.
So, if a user is on the chat page, the ajax script sends a request to a php file that shows the chat histories (latest messages), and returns it in HTML. This request is looped every second to show the latest messages to the user viewing the page.
so far its been working great.
now my question and concern is, 1.) What are the cons of using a method like this, if any? 2.) What things should i worry most about, if it gets a large user base and many people are using it simultaneously? (mostly because its making a request every second, for each user on it..)
the mysql table is an innodb table, and I'm using only one SELECT statement without a WHERE clause.. something like SELECT * FROM table ORDER BY id DESC LIMIT 10 etc.. (basically, I'm making mysql do something very easy like cake)
3.) Any suggestion are welcome ;)
thanks very much
vikash
Definitely, you will need to look at scalability issues for both the web server and database server. There are technologies such as MySQL clustering for improving performance on the database and web clustering for the HTTP side of things.
With large scale use you may also look at trimming down the table by removing early posts and dumping them to a separate table for low-frequency access. You could also have some method of caching the database requests via some worker threads so the database reads are minimal, but the front-end will have the ability to cope with the high volume of requests.
I got 60 people in phpFreeChat (php/ajax/mysql chat) and it was a complete processor hog. It brought an 8 core server to its knees.
I have 2 websites, lets say - example.com and example1.com
example.com has a database fruits which has a table apple with 7000 records.
I exported apple and tried to import it to example1.com but I'm always getting "MYSQL Server has gone away" error. I suspect this is due to some server side restriction.
So, how can I copy the tables without having to contact the system admins? Is there a way to do this using PHP? I went through example of copying tables, but that was inside the same database.
Both example.com and example1.com are on the same server.
One possible approach:
On the "source" server create a PHP script (the "exporter") that outputs the contents of the table in an easy to parse format (XML comes to mind as easy to generate and to consume, but alternatives like CSV could do).
On the "destination" server create a "importer" PHP script that requests the exporter one via HTTP, parses the result, and uses that data to populate the table.
That's quite generic, but should get you started. Here are some considerations:
http://ie2.php.net/manual/en/function.http-request.php is your friend
If the table contains sensitive data, there are many techniques to enhance security (http_request won't give you https:// support directly, but you can encrypt the data you export and decrypt on importing: look for "SSL" on the PHP manual for further details).
You should consider adding some redundancy (or even full-fleshed encryption) to prevent the data from being tampered with while it sails the web between the servers.
You may use GET parameters to add flexibility (for example, passing the table name as a parameter would allow you to have a single script for all tables you may ever need to transfer).
With large tables, PHP timeouts may play against you. The ideal solution for this would be efficient code + custom timeout for the export and import scripts, but I'm not even sure if that's possible at all. A quite reliable workaround is to do the job in chunks (GET params come handy here to tell the exporter what chunk do you need, and a special entry on the output can be enough to tell the importer how much is left to import). Redirects help a lot with this approach (each redirect is a new request, to the servers, so timeouts get reset).
Maybe I'm missing something, but I hope there is enough there to let you get your hands dirty on the job and come back with any specific issue I might have failed to foresee.
Hope this helps.
EDIT:
Oops, I missed the detail that both DBs are on the same server. In that case, you can merge the import and export task into a single script. This means that:
You don't need a "transfer" format (such as XML or CSV): in-memory representation within the PHP is enough, since now both tasks are done within the same script.
The data doesn't ever leave your server, so the need for encryption is not so heavy. Just make sure no one else can run your script (via authentication or similar techniques) and you should be fine.
Timeouts are not so restrictive, since you don't waste a lot of time waiting for the response to arrive from the source to the destination server, but they still apply. Chunk processing, redirection, and GET parameters (passed within the Location for the redirect) are still a good solution, but you can squeeze much closer to the timeout since execution time metrics are far more reliable than cross-server data-transfer metrics.
Here is a very rough sketch of what you may have to code:
$link_src = mysql_connect(source DB connection details);
$link_dst = mysql_connect(destination DB connection details);
/* You may want to truncate the table on destination before going on, to prevent data repetition */
$q = "INSERT INTO `table_name_here` (column_list_here) VALUES ";
$res = mysql_query("SELECT * FROM `table_name_here`", $link_src);
while ($row = mysql_fetch_assoc($res)) {
$q = $q . sprintf("(%s, %s, %s), ", $row['field1_name'], $row['field2_name'], $row['field3_name']);
}
mysql_free_result($res);
/* removing the trailing ',' from $q is left as an exercise (ok, I'm lazy, but that's supposed to be just a sketck) */
mysql_query($q, $link_dst);
You'll have to add the chunking logics in there (those are too case- & setup- specific), and probably output some confirmation message (maybe a DESCRIBE and a COUNT of both source and destination tables and a comparison between them?), but that's quite the core of the job.
As an alternative you may run a separate insert per row (invoking the query within the loop), but I'm confident a single query would be faster (however, if you have too small RAM limits for PHP, this alternative allows you to get rid of the memory-hungry $q).
Yet another edit:
From the documentation link posted by Roberto:
You can also get these errors if you send a query to the server that is incorrect or too large. If mysqld receives a packet that is too large or out of order, it assumes that something has gone wrong with the client and closes the connection. If you need big queries (for example, if you are working with big BLOB columns), you can increase the query limit by setting the server's max_allowed_packet variable, which has a default value of 1MB. You may also need to increase the maximum packet size on the client end. More information on setting the packet size is given in Section B.5.2.10, “Packet too large”.
An INSERT or REPLACE statement that inserts a great many rows can also cause these sorts of errors. Either one of these statements sends a single request to the server irrespective of the number of rows to be inserted; thus, you can often avoid the error by reducing the number of rows sent per INSERT or REPLACE.
If that's what's causing your issue (and by your question it seems very likely it is), then the approach of breaking the INSERT into one query per row will most probably solve it. In that case, the code sketch above becomes:
$link_src = mysql_connect(source DB connection details);
$link_dst = mysql_connect(destination DB connection details);
/* You may want to truncate the table on destination before going on, to prevent data repetition */
$q = "INSERT INTO `table_name_here` (column_list_here) VALUES ";
$res = mysql_query("SELECT * FROM `table_name_here`", $link_src);
while ($row = mysql_fetch_assoc($res)) {
$q = $q . sprintf("INSERT INTO `table_name_here` (`field1_name`, `field2_name`, `field3_name`) VALUE (%s, %s, %s)", $row['field1_name'], $row['field2_name'], $row['field3_name']);
}
mysql_free_result($res);
The issue may also be triggered by the huge volume of the initial SELECT; in such case you should combine this with chunking (either with multiple SELECT+while() blocks, taking profit of SQL's LIMIT clause, or via redirections), thus breaking the SELECT down into multiple, smaller queries. (Note that redirection-based chunking is only needed if you have timeout issues, or your execution time gets close enough to the timeout to threaten with issues if the table grows. It may be a good idea to implement it anyway, so even if your table ever grows to an obscene size the script will still work unchanged.)
After struggling with this for a while, came across BigDump. It worked like a charm! Was able to copy LARGE databases without a glitch.
Here are reported the most common causes of the "MySQL Server has gone away" error in MySQL 5.0:
http://dev.mysql.com/doc/refman/5.0/en/gone-away.html
You might want to have a look to it and to use it as a checklist to see if you're doing something wrong.