I am trying to integrate node.js app into an existing PHP/MySQL system. node-mysql is used to allow node.js to access the MySQL database. A table named tableA in the MySQL database updates with new data every 10 minutes.
Problem: Using node, I need the node server to continuously check tableA for changes whenever tableA gets updated. What is a good way to achieve this?
I am thinking if I used setInterval(checkDb(tableA), 10*60*1000), or an infinite loop, the checkDbquery function may not coincide with the database update.
I also happen to be using socket.io so will a good method be for the PHP system to emit('dbUpdated') to the node server if it is even possible for PHP to send a message to node.js?
Node Code
var dbCheckQueue = function(callback) {
client.query('SELECT * from tableA',
function(error, results, fields) {
if error
throw error;
callback(results);
});
}
// 10mins-interval checking of table
setInterval(function() {
dbCheckQueue(function(results) {
console.log('Fresh meat!')
});
}, 10*60*1000);
Ideally you would trigger the event from the where ever the update is being made from after completion.
Since you are using sockets you could create an "Events" connection or similar (authenticate or some other method to indicate it is connecting to trigger an event) which would connect to your socket and trigger an update and disconnect. This could be called from anywhere regardless of platform as long as you are able to create the connection. Once connected the script would send whichever command you want to be triggered and disconnect/finish the updating process.
You could either do this directly from where the update is made or setup a trigger on mysql with sys_exec() to execute another script which does the same as above.
Related
For some reasons (that I think it is not the point of my question, but if it help, ask me and I can describe why), I need to check MySQL tables continuously for new records. If any new records come, I want to do some related actions that are not important now.
Question is, how I should continuously check the database to make sure I am using the lowest resources and getting the results, close to the realtime.
For now, I have this:
$new_record_come = false;
while(! $new_record_come) {
$sql = "SELECT id FROM Notificatins WHERE insert_date > (NOW() - INTERVAL 5 SECONDS)";
$result = $conn->query($sql);
if ($result)
{
//doing some related actions...
$new_record_come = true;
}
else
{
sleep(5); //5 seconds delay
}
}
But I am worry that if I get thousands of users, it will make the server down, even if the server is a high price one!
Do you have any advice to make it better in performance or even change the way completely or even change the type of query or any other suggestion?
Polling a database is costly, so you're right to be wary of that solution.
If you need to scale this application up to handle thousands of concurrent users, you probably should consider additional technology that complements the RDBMS.
For this, I'd suggest using a message queue. After an app inserts a new notification to the database, the app will also post an item to a topic on the message queue. Typically the primary key (id) is the item you post.
Meanwhile, other apps are listening to the topic. They don't need to do polling. The way message queues work is that the client just waits until there's a new item in the queue. The wait will return the item.
A comment suggested using a trigger to invoke a PHP script. This won't work, because triggers execute while the transaction that spawned them is not yet committed. So if the trigger runs a PHP script, which probably needs to read the record from the database. But an uncommitted record is not visible to any other database session, so the PHP script can never read the data that it was notified about.
Another angle (much simpler than message queue I think):
I once implemented this on a website by letting the clients poll AND compare it to their latest id they received.
For example: You have a table with primary key, and want to watch if new items are added.
But you don't want to set up a database connection and query the table if there is nothing new in it.
Let's say the primary key is named 'postid'.
I had a file containing the latest postid.
I updated it with each new entry in tblposts, so it contains alsways the latest postid.
The polling scripts on the clientside simply retrieved that file (do not use PHP, just let Apache serve it, much faster: name it lastpostid.txt or something).
Client compares to its internal latest postid. If it is bigger, the client requests the ones after the last one. This step DOES include a query.
Advantage is that you only query the database when something new is in, and you can also tell the PHP script what your latest postid was, so PHP can only fetch the later ones.
(Not sure if this will work in your situation becuase it assumes an increasing number meaning 'newer'.)
This might not be possible with your current system design but how about instead of using triggers or a heartbeat to poll the database continuously that you go where the updates, etc happen and from there execute other code? This way, you can avoid polling the database continuously and code will fire ONLY IF somebody initiates a request?
Current situation: I have a web page that uses AJAX/JQUERY to refresh all the content on the page every 17 seconds. Every time this happens the server queries the database for data from two tables, one of which is large (450MiB in size, 11 columns) and processes all the data.
This is too resource intensive. I want:
The server queries the database only when one of the two tables have changed.
The page then reloads the page through AJAX only when the tables have been updated and the server has re-processed the data.
I think this falls under the category of comet programming. I'm not sure.
2 is easy. The webpage calls 'update.php' every 17 (or maybe less) seconds. The PHP script returns no data if no changes have been made. Only if data is returned then the current page is replaced with the new data. Please advise me if there is a better way.
As for 1 my googling tells that every time one of my two tables is updated I can put a notification in a table (or maybe just a single byte in a file) to indicate that I must query the database again and then the next time that the webpage sends an AJAX request I return the data.
The problem is that I'm working with a rather large code base I didn't write and I don't know of all the places that either of the two tables may be updated. Is there an easier way to check when the database is modified.
I'm working with PHP, Apache, Drupal and MYSQL.
You can chekout Server Sent Events
A server-sent event is when a web page automatically gets updates from a server.
there is an excellent article on HTML5rocks.com - Server Sent Events
All You have to do is create an object
var source = new EventSource('xyz.php'); //Your php files which will return the updates.
Once you create an object,you can listen to the events
source.addEventListener('message', function(e) {
console.log(e.data);
}, false);
Many desktop applications (e.g, those built with Delphi) use "Database aware components", such as a grid, which display the contents of a database - usually the result of a query - and auto-update their display when the database contents change.
Does such a thing exist for PHP (maybe displaying the result of a query in an HTML table, and updating)?
If not, how would we go about creating it?
(Note: This seemingly similar question didn't help me)
It's not technically possible to update a HTML page once rendered with pure PHP because of the static nature of the HTTP protocol so any solution would have to include JavaScript and AJAX calls.
Emulating using AJAX to re-render the table every 5 minutes
It wouldn't be hard to emulate though, just make a PHP page which gets the results of a database and puts them in a table, then use jQuery's .load() function to get that table and render it in a DIV of your choice on an interval of 5 seconds or whatever.
Something like:
function updateTable(){
$('#tableDiv').load(url);
}
var url = 'renderTable.php';
setInterval(updateTable,5000);
You can put this in any PHP (or HTML) page with a DIV with id tableDiv and it will render the output of renderTable.php in that div every 5 seconds without refreshing.
Actually monitoring the database
This is possible, you'd have to set up a PHP file on a cron for every 5 seconds (or an AJAX call every 5 seconds) to run something like SELECT MD5(CONCAT(rows,'_',modified)) AS checksum FROM information_schema.innodb_table_stats WHERE table_schema='db' AND table_name='some_table'; (assuming innoDB), you'd then compare this to the previous checksum.
If the checksums are identical, you could pass 'false' to your modified AJAX call, which would tell it not to render anything over the table. If they aren't, you could pass it the HTML of the new table to render in place.
it could be done but with mix of different technologies
if I would like to monitor in real time changes made in database I would think about triggers and sockets - trigger in database should call (on insert or update) function that will add event to queue - here's example function for postgresql (plsh is custom handler)
CREATE FUNCTION propagate_event(tablename) RETURNS text AS '
#!/bin/sh
# execute script that will add event to message queue
/var/scripts/change_in_table.sh "$1"
' LANGUAGE plsh;
client connects to socket and receives that events in real time
Our company deals with sales. We receive orders and our PHP application allows our CSRs to process these orders.
There is a record in the database that is constantly changing depending on which order is currently being processed by a specific CSR - there is one of these fields for every CSR.
Currently, a completely separate page polls the database every second using an xmlhhtp request and receives the response. If the response is not blank (only when the value has changed on the database) it performs an action.
As you can imagine, this amounts to one databse query per second as well as a http request every second.
My question is, is there a better way to do this? Possibly a listener using sockets? Something that would ping my script when a change has been performed without forcing me to poll the database and/or send an http request.
Thanks in advance
First off, 1 query/second, and 1 request/second really isn't much. Especially since this number wont change as you get more CSRs or sales. If you were executing 1 query/order/second or something you might have to worry, but as it stands, if it works well I probably wouldn't change it. It may be worth running some metrics on the query to ensure that it runs quickly, selecting on an indexed column and the like. Most databases offer a way to check how a query is executing, like the EXPLAIN syntax in MySQL.
That said, there are a few options.
Use database triggers to either perform the required updates when an edit is made, or to call an external script. Some reference materials for MySQL: http://dev.mysql.com/doc/refman/5.0/en/create-trigger.html
Have whatever software the CSRs are using call a second script directly when making an update.
Reduce polling frequency.
You could use an asynchronous architecture based on a message queue. When a CSR starts to handle an order, and the record in the database is changed, a message is added to the queue. Your script can either block on requests for the latest queue item or you could implement a queue that will automatically notify your script on the addition of messages.
Unless you have millions of these events happening simultaneously, this kind of setup will cause the action to be executed within milliseconds of the event occuring, and you won't be constantly making useless polling requests to your database.
Can someone lead me down the right way to make a live notifications
e.g Knowing when a new Row in Added in Mysql
know if a php file has changed ???
how should i go about it?
You could routinely check the server for updates using setInterval(), or you could employ long-polling with javascript. The benefit of setInterval() is that it doesn't keep connections opened on your server for too long, but you may have updates during the 'downtime' between server-calls. Long-polling will give you near-instant updates, as it waits with the connection opened until it receives new information. But obviously, the down side is that you've got connections staying opened all over the place.
Routine Checks...
setInterval(function(){
$.get("updates.php", {}, function(results){
if ($(results).length) {
$("results").each(function(){
// do something with update messages
});
}
});
}, 30000); // Every 30 seconds.
Long Polling with PHP/jQuery Example:
You can find an example of long polling with PHP and jQuery at http://blog.perplexedlabs.com/2009/05/04/php-jquery-ajax-javascript-long-polling/
You can use db triggers to watch for changes in certain tables and insert notification data into a new table. Then. query that db table periodically with Jquery and ajax.
Workflow:
Create trigger that watched table users for inserts, updates, and deleted
Once users is altered, the trigger inserts a new record into notifications detailing what was changed
Using a periodical updater, check the notifications table for new records and display them to the user.
This simple workflow might not be as easy to implement as you would hope but it would get the job done in an efficient manner.