I am not really sure how to phrase my question, but I'll try to be as clear as possible.
I want to create a demo website which allows people to input sentences which are then sent to a remote host, and classified in a Python script on that host. Basically, the Python script, when initialised, needs to load some big classifier files into NLTK classifier objects, which is preferably done only once to save time. So I would like to keep these objects alive. I realise I can execute a while loop in order to wait for any incoming data. The loop would parse the data and do whatever with it:
#pseudocode for Python script
while True:
if(some_remote_input){
parse, classify, etc.;
}
My question: What is a recommended package to collect data sent remotely? socket, perhaps?
And how would I go about this in PHP/AJAX? The desired procedure is like this:
User enters a sentence in the web app
Sentence is validated/prepared
Sentence is sent to remote host as input for python script (which is always running (like a server)
The result of the python classification script is returned to the website
Results are formatted and printed to the user
I am concerned about step 3.
I prefer not to integrate the Python environment on the server where the website is hosted, as there is not a lot of free space and it requires a lot of annoying packages.
If you need any more info, please let me know!
What you are looking for is a message broker or message queue system.
There are several message brokers available:
RabbitMQ
Kafka
AMQ
This lets you pipe data betwen the message transmitters (web server) and message consumers (the natural language processing pipeline) in a way that is agnostic to what technologies are used in the web server and the NLP-pipeline
Related
I am writing a JavaScript for an in-browser IM client for the sake of practicing and learning JavaScript and AJAX.
I need to be able to check for a change in the file size of a text file that is being used as a temporary storage for 40-80 SQL entries that contain messages so that it can update the display.
At the moment I am using a setInterval function to periodically check for a change in file size using short PHP script, but this can cause issues, if the interval is to long, messages are delayed, if it is shorter, it means a lot of php scripts running very quickly, which takes up server resources.
What is the best way to do this if the main concern is to reduce server resource usage?
(I am running my server off of a rather low tech PC I've scraped together(2gb ram, 2.8ghz AMD seperon processor))
Preferably, I would want to do this using an AJAX event triggered by someone sending a message, I.E. When user B triggers the event that edits the file by pressing enter, that triggers a function on user A's side that updates the HTML file
Any ideas? I am open to any solution to this particular problem. I gave specific examples of what I want to happen in the specific languages in order to give a better idea of what it is I am attempting to do.
If there is a way to do this that isn't JavaScript/PHP, I'd also be open to exploring that as an option.
Doing this with PHP can be a bit cumbersome. You could try doing something like long polling where you keep the HTTP request open until the server has new data to send to the user. If messages are sent frequently, this might not be ideal. You might want to consider using event-driven web technologies like node.js with something like Socket.IO.
In any case, you'll likely want to maintain a connection with the server if you want to get the message in near real-time. There are ways to use WebSockets with PHP as well, but PHP isn't really the best for this because it's not designed to keep scripts running for long periods (also see What exactly entails setting up a PHP Websocket Server?).
Browsers & HTTP/ AJAX generally work by a "pull" model. The browser/ or AJAX sends the server a request, then the server answers a response.
There isn't generally much provision for the server to contact the browser, to "push" an event. This can however be simulated by a long-running request, to which the server writes data when the event/ or events occur.
For example, this could be a request that answers "empty" after a timeout of 10-30 seconds.. or the server returns & answers immediately, if there are event(s) in its queue.
With a Java server this is easy to do, and I've used this successfully for event notification in a major integration project a few years back.
However I'm not sure in PHP how much ability there is (probably very near zero) to maintain an overall server state, coordinate or communicate between threads/requests, or maintain event queues.
You could look into something like a Java webapp running on Tomcat. All you need is a basic web.xml and one Servlet class, and you can build just about anything from there.
I'm new to php. Please can anyone guide me through this? I have a php where I'm having some variables as sort of flags. When those flags are set/true, I start those services in my Blackberry otherwise not. My application keeps on listening to my php and when those flags are true, it starts the respective services, others are kept off. How can I achieve this? Thanks.
It sounds like you want push notifications.
If you do it by "listening" to a php page, you're going to have to repeatedly make requests to that php page. You could do long-polling, but that's not going to work very well on a cell phone... each time the connection disconnects, you need to connect again, and it's messy to write all that client side code yourself. PHP also isn't a great language for push notifications, because on the web-server side, there's no easy way to push information to php once php has a connection from your blackberry. Since PHP isn't multithreaded (in typical webserver configurations), you'll again most likely be busy-polling some notification in a file on the disk of the web server, rather than just receiving and handling a notification as from a message queue or socket/io connection.
Most phone development kits have built in APIs to handle push notifications. From what I understand, re-using these systems leverages existing architecture, meaning the phone only connects to one server, which saves battery life.
I've never done any blackberry development, but you should probably start by searching for information/tutorials on implementing push notifications in blackberry.
If I'm wrong, and you just want to be able to get configuration options for your blackberry app, then that's easy. Your php page just spits out the information in any format the client (blackberry) can read (like json, xml, key=value, whatever), blackberry parses it, and acts on it.
I want to write a web application to play chess against the engine Crafty. I'm not new to PHP and javascript, but must learn how to interact with a server process : how can a web application and/or (jQuery) ajax interact bi-directionally with a (linux) program running on the server?
At this moment i am developing on (Apache) local host. Crafty is installed on my Ubuntu PC. This well-known chess engine has no GUI, it runs in terminal by the command
$ /usr/games/crafty
and so you can play chess against it and even see it's calculations :
I can make Crafty run by PHP, using the functions proc_open() or exec(), and most documentation i found states that the output stream should be a file .. But i think i don't want such setup, because then the webpage should be constanty polling that file (eg. by ajax) to see if some new data was appended, right?
How can Crafty talk to the web page directly, saying "i have calculated another variation" or "i have decided a move" etc, then display this info on the web page and let the user give some counter move, just like in terminal. Isn't it possible to use some session / stream / listener?
I have no clue at all, can anybody point me in a right direction?
I recommend you make use of fifos and the & operator - here is why:
You do not want to start crafty on every PHP request, you want to start it only once per game
You don't want to have crafty end at the end of your Request
Your move-requests will want to interact with this allready running instance.
So what I would do is something like:
Prepare a pair of FIFOs using mkfifo - you can do this from PHP or from the shell
On game start, run something like /usr/games/crafty <stdin.fifo >stdout.fifo 2>stderr.fifo &
For your moves, make an AJAX PHP request write to stdin.fifo
For the server moves do long polling with AJAX, on the server side opening stdin.fifo, then stream_select()
Hi,
From the image above, I have a webserver a linux machine and client/device.. Now i need for this 3 to communicate. The webserver sends data to an ip address(client/device) based on button pressed on the webpage. but before the data is sent, the data must first access the linux machine, the machine then sends the data down to the device which then the device reads the data and act based on the command sent.. then the device sends back data to the linux machine which then the linux machine sends it to the webserver for ack'd. meaning data is received by the device without any problems.
Php is for the webserver. Now how will php sends data to an ip adress.
The linux machine handles all requests and sends everything down to the device and when the device got the data it will send a data to linux machine which then machine sends an ok to the webserver that the data arrived succesfully.(I read about socket programming and i think of creating an application that reads requests.) or if you have any idea how can i do this?.
How can the device read a data sent by the webserver?..
Thanks,
EDIT: The device is not connected to the linux machine. the device is only connnected via the ethernet cable.
Let's call the topmost machine 'Server', the middle machine 'Controller' and the bottom machine 'Device'. It does not matter if the device is a peripheral (say, USB or serial device), or a computer.
The first task is to get the Controller to query the Device. The best way to do this really depends on the Device. If you consider things like USB audio/video devices, they need to be tuned, then they send a continuous stream of data. Things like temperature or humidity sensors are told to do a measurement, then they respond with data.
Usually you write the required functions into a small library, and verify it works using command line tools. In some cases the library may not be necessary, for example if the Device is already supported by the kernel in Controller, and the information is trivially available. (For example, consider the temperature sensors in hard drives: if Device(s) are hard disks, then Controller can simply use the command hddtemp /dev/sda to get the temperature of the /dev/sda (first SATA/ATA/SCSI hard disk). I'd expect the end user to be able to pick which hard disks she is interested in, so that choice would have to flow from Server to Controller.)
Next, you write a service that will run on the Controller. This service will incorporate the library functions already written and tested, so it can easily access the Device. (This way you know the Controller-Device communication works, and don't need to worry about it. One thing at a time.)
There are many different designs for the service, from plain TCP/IP or UDP/IP sockets to Remote Procedure Calls (RPC), to high-level protocols like HTTP. In recent years, the last, using HTTP, has become more and more common, with responses being XML, plain text, or binary media (usually images). The idea is to have the service be basically just another web server that can access the Device directly. Security is simpler, because it does not need to be world-accessible: it can very well only answer to requests coming from the Server only. I've written such services using basic shell scripting (Bash), PHP (both PHP-CGI and command-line PHP, PHP-CLI), and C, among others. The best choice depends on the details, really. I personally prefer either a simple text-based TCP/IP socket, or HTTP.
On the Server, you can write a PHP page, that connects to Controller, requesting whatever it wants to request (usually depends on user data, first checked for sanity and safety, of course). PHP has easy built-in facilities for doing both HTTP requests and connecting using raw TCP/IP, so it suits quite well for this. If HTTP protocol wrappers are enabled, then it is just $handle = fopen("http://192.168.x.x/myservice?param1=" . urlencode($param1) . "¶m2=" . urlencode($param2), "r+b");. To get a socket connection, you use the fsockopen() function instead. (For details, see fopen(), http wrappers, and fsockopen() at the PHP Function Reference at www.php.net.)
In practice the PHP page code first creates a connection to the Controller. Then it sends a request, containing the relevant sanitized commands/parameters received from the end user. Then it waits for the Controller to respond with the results (by simply reading the response), then closes the connection. The response should contain all the data needed, so the PHP page is free to construct the page to the end user.
None of this is really difficult, but there is a lot to do. I've found the Controller-Device communication to require the most work; after that is done, the rest has always been quite straightforward.
If you can provide more details what the Controller-Device connection is, what kind of data (text? numbers? images? a lot of binary data?) the Device provides, and what kind of parameters/commands (just "one result, please?", basic commands like "move up", "where are you?") do you expect you need to send to the Controller/Device, I could perhaps be more specific.
Also, are you limited to PHP, or would you be comfortable writing the Controller service using C? I've found that to be a very good combination myself.
Edited to add:
In a nutshell, the three points can be answered as follows:
Either using fopen("http://ip.add.re.ss:port/", "r+b"); if using the HTTP protocol and PHP is configured to allow http wrappers (they usually are), or using fsockopen(). See the PHP documentation linked above for details.
With an IP-connected Device, Controller is basically a relay or translator. Usually this means a daemon running on Controller, managing incoming requests from Server (or Servers), and responses from Device (or Devices). This is more common when there are a varying number of Devices, and/or more than one interface is needed. In practice, the Controller runs a daemon just like described above, except the protocols may be standard or simple enough so there is no need to write a library.
The PHP running on the Server must contain the request details (exactly what is desired) to the Controller. The Controller must pass them on to the Device. If the Controller provides a http URL for the PHPs on the server connect to, it can parse the query parameters, and translate them into a format the Device understands.
One particular issue in practice is to handle concurrent accesses. There is usually only a single connection from Controller to Device, but more than one PHP might connect to the Controller simultaneously. So there is some book-keeping involved.
In some cases the Device provides a continuous stream of data (or regular updates of data) to the Controller, and the Controller simply keeps tabs on it. When a PHP running on the Server queries something from the Controller, the Controller simply looks up the latest data (without contacting the Device at all, just receiving the data as normal), and responds with it. Here, it is common to include a timestamp, or better yet, the age of the data, in the response from Controller to Server.
You really should add some details to your question. (I suspect the downvote is due to lack of details.) You don't need to tell us the exact make and model of the Device, only whether it is a receiver (TV? radio? weather station?) or a sensor cluster or a door lock, and if you know any details on the communications protocols (which ones)? Thus far, we only know it uses IP. That does not help at all, just about everything uses IP nowadays. This is also why my answer is so vague; I'd like to be more precise, but you do not provide enough information for me to do so.
I'm not sure how it is being handled, so I wanted to ask it.
The new generation MMORPG games are trying to bridge their trading post (auction) systems on webserver, so the players can use it without any hassle even from their mobile phones.
The last example of it is Guild Wars 2. Basically, the system works like this:
1. You login a locally running client.
2. Open up the auction house in game client. (like HTML frames in this case)
3. The auction house connects to a webserver instead of sending packets to actual game server, like auctionhouse.guildwars2.com, which is also accessable via browser.
4. You want to sell your Sword, if the auction house successfully takes the Sword, it gets deleted from user inventory (client) and server, hence, client somehow gets informed by webserver reply.
So... I'm guessing...
How does the locally running client (a C++ application) knows if the item is sold or something failed? Does the webserver return a XML/JSON output for verification?
Like, webserver returns this;
// XML reply
<auctionResponse>
<itemId>184818478A</itemId>
<success>Successful</success>
<verifyKey>AG8918ADHWDHA</verifyKey>
</auctionResponse>
and client checks it like;
if(auctionHouse.auctionResponse == 'Successful')
{
if(auctionHouse.auctionResponse == getVerifyKeyFromServer()) //so the server confirms
{
DeleteFromInventory(auctionHouse.itemId); //item will be removed from user inventory
}
}
I've asked this to the developer of GW2 auction house, but he told me he can't share information about this.
So, basically, how does it work? The JSON/XML outputs by webserver or something entirely different to carry data?
Any help would be appreciated.
Ps. It is not a TCP connection. The auction house itself runs on port 80 and the backend is coded with a language like PHP.
While I do not know how they are doing it specifically, I write similar applications on a regular basis. Here is how it generally works:
PHP can use proc_open() to pipe to a process being executed. This means you can send commands to another application from within PHP and use the results within your PHP script. This includeds custom exe written in anything.
Another option is a server application running on localhost or any other machine where PHP uses CURL to communicate with that server. The results can be sent in JSON, XML, plain text, etc...
Both of the options allow the PHP webserver to run on port 80.
EDIT:
I suspect the client app you download is communicating with the server, probably a server that shares the same database with the web app. Therefore when you purchase something in the client, a DB on their server is updated, and their web app then has the updated information.