I have a static website where all the content is rendered by elm.
Right now all the data is hard-coded into the elm source code. In the future I would like to add a small amount of database interaction to the project.
The web server I use support MySQL databases and PHP.
I was thinking it would be nice to be able to use the get function in the elm Http package to point to a php script on the server, which would query the database, and return json data that my elm program could interpret and render.
I would like to know if:
This approach is possible
There is a better (more convenient or correct) way to do this
What you describe is a good way to do it. See this chapter in elm-tutorial that covers this http://www.elm-tutorial.org/080_fetching_resources/cover.html
As an alternative you could seed the data in the html and pass it via ports.
That approach is very much possible (I do the same to access TCP connections on my server by using a GET request to a CGI module on the same server as the web page).
This is, as far as I know, the best way to do this for all client side pages. I work for a company and we use PHP, Node and MySQL, with about half of the scripts in Node and the other half in PHP, all of them just talk between the front end and the database.
Related
I'm currently trying to find the best way of doing this:
We have a python program (client side) we use to upload metrics on a mySQL database on a server and later, via web check it and filter it, etc.
The problem arises when we try to plot any query from to the database. It's unclear to us what aproach to use. The main page was made with Joomla in php.
Currently I was looking into python alternatives to run on the server side, somehow capture the query, process the data, create the image and then return it to the client side as an image or as a string to be reconstructed on the client side. But as I have read it seems also possible (and maybe easier) to do the same in PHP or JavaScript which (as I understand it) run on the client side, leaving less to worry about.
Is it that so? Are my assumptions right? Which aproach would you use/pŕefear? Is there some link or info you could give me to continue my search?
I would prefer to do it on Python using something like matplotlib, plotly, bokeh, etc. but as I see it, the problem is not about creating the image, but about comunicating and sending the image information between server and client.
Thanks!
I think it possible to do what you want to with bokeh. I dont know joomla but it think its not that important.
Check out that part of the bokeh documentation Embedding Bockeh
then for the server parts in python i would recommend flask to start with.
You can just prototype your app with one of the bokeh server apps examples from the bokeh repo.. there are flask examples too.
Then you can start to extend the bokeh server app from the examples with a query to your database with sqlalchemy or
mysql-flask
Is there any way you can push data to a page rather than checking for it periodically?
Obviously you can check for it periodically with ajax, but is there any way you can force the page to reload when a php script is executed?
Theoretically you can improve an ajax request's speed by having a table just for when the ajax function is supposed to execute (update a value in the table when the ajax function should retrieve new data from the database) but this still requires a sizable amount of memory and a mysql connection as well as still some waiting time while the query executes even when there isn't an update/you don't want to execute the ajax function that retrieves database data.
Is there any way to either make this even more efficient than querying a database and checking the table that stores the 'if updated' data OR tell the ajax function to execute from another page?
I guess node.js or HTML5 webSocket could be a viable solution as well?
Or you could store 'if updated' data in a text file? Any suggestions are welcome.
You're basically talking about notifying the client (i.e. browser) of server-side events. It really comes down to two things:
What web server are you using? (are you limited to a particular language?)
What browsers do you need to support?
Your best option is using WebSockets to do the job, anything beyond using web-sockets is a hack. Still, many "hacks" work just fine, I suggest you try Comet or AJAX long-polling.
There's a project called Atmosphere (and many more) that provide you with a solution suited towards the web server you are using and then will automatically pick the best option depending on the user's browser.
If you aren't limited by browsers and can pick your web stack then I suggest using SocketIO + nodejs. It's just my preference right now, WebSockets is still in it's infancy and things are going to get interesting once it starts to develop more. Sometimes my entire application isn't suited for nodejs, so I'll just offload the data operation to it alone.
Good luck.
Another possibility, if you can store the data in a simple format in a file, you update a file with the data and use the web server to check its timestamp.
Then the browser can poll, making HEAD requests, which will check the update times on the file to see if it needs an updated copy.
This avoids making a DB call for anything that doesn't change the data, but at the expense of keeping file system copies of important resources. It might be a good trade-off, though, if you can do this for active data, and roll them off after some time. You will need to ensure that you manage to change this on any call that updates the data.
It shares the synchronization risks of any systems with multiple copies of the same data, but it might be worth investigating if the enhanced responsiveness is worth the risks.
There was once a technology called "server push" that kept a Web server process sitting there waiting for more output from your script and forwarding it on to the client when it appeared. This was the hot new technology of 1995 and, while you can probably still do it, nobody does because it's a freakishly terrible idea.
So yeah, you can, but when you get there you'll most likely wish you hadn't.
Well you can (or will) with HTML5 Sockets.
This page has some great info about this technology:
http://www.html5rocks.com/en/tutorials/websockets/basics/
I am looking to create a Web Chat system using PHP, MySQL and JavaScript.
Currently, I am storing messages in a MySQL database with an incremental ID (Yes, it is indexed), a timestamp, the sender, and the message itself. I am then using AJAX to query the database every 500ms, seeing if there are any more recent messages than the last one received. However, I have a feeling that this is probably horribly inefficient as it will result in a large load on the MySQL server when multiple users are online. Having looked around a bit on Google and on here, everything seems to point to this way of doing it.
My question: is there a better way to do this? Any tips on how to reduce the load on the server would also be welcome.
I'm using PHP 5.3, on an Apache webserver, so libraries or plugins compatible with those would be fine.
EDIT:
Forgot to mention in the original post, but I'm not worried about supporting IE or other outdated browsers.
Potentially viable basic approach:
Cache your 50 most recent messages in memcache. Reset this whenever a new entry is added to the database. When new users connect, serve them these 50 messages to populate their chatroom.
Use a third party service like http://www.pubnub.com/ to send messages to your clients. Whenever a new message is sent to your chatroom, send it out on pubnub. Your server code will do this after writing to your database successfully.
notes: I'm not affiliated with pubnub. You don't need to use 50 messages above either. You don't even have to give them any messages when they connect depending on how you want to set it up. The point is that you want to avoid your users reading from your database in this case - that model isn't likely to scale for this type of application.
Ideally, an evented environment would be ideal for this kind of app. The LAMP stack is not particularly well suited.
I would recommend using this library, Pubnub. Pubnub is an easy way to send radio signals via javascript, or any TCP language (such as PHP) - and javascript instantly recieves the sent messages.
In PHP, you could simply have it save to your database - then use Pubnub's PHP API's to send the message to everyone else on the page.
If your familiar with Html, Javascript, and PHP - it can be fairly easy to learn. I would recommend it.
You are asking about a web chat system specifically built in PHP, MySQL and HTML with JavaScript. There are many options including Pre-built solutions: http://www.cometchat.com/ and http://www.arrowchat.com/ which all have chat comet services powered by a cloud offering like http://www.pubnub.com/ with options to host it yourself. See more about CometServices http://www.cometchat.com/cometservice/third-party-alternatives where you compare the service providers. There are several more options, however I recommend starting there. If you needs something more simple, like HTML and JavaScript only solution, you can check out http://www.pubnub.com/blog/build-real-time-web-apps-easy which is a blog about building real-time web apps easy with an example chat app in 10 lines of JavaScript Code. The solution Cuts Development Time by providing full Cross Platform for all browsers and mobile devices.
You should look into ajax long polling, in a nutshell this a simple ajax call but will not return a result from the server if there is no new data. You just do a simple loop on the server side until new data will be available then return it. Of course you have to stop this eventually if there's no result to send to client after a while (eg. 1 minute) then restart the call.
I suppose, that chat is too intensive for storage engines MySQL. Maybe, MEMORY table type will be ok, never used it. I spoken to several developers and everybody agree, that best option for chat is Memcache or even writing your own custom daemon (with memory-only storage as weel).
For client part you may read about short-polling, long-poling and web-sockets / sockets via flash/Java object.
using AJAX to query the database every 500ms
Is short-polling.
Sockets are a better solution than AJAX polling, however isn't much around about how you can integrate socket based chats with MySQL.
I have done a few tests and have a basic example working here: https://github.com/andrefigueira/PHP-MySQL-Sockets-Chat
It makes use of Ratchet (http://socketo.me/) for the creation of the chat server in PHP.
And you can send chat messages to the DB by sending the server JSON with the information of who is chatting, (if of course you have user sessions)
Would it be safe to use a MySQL database to record positions of players on the screen?
Then everyone second, Flash retrieves the new position data in the database and sets the players' positions of the map to the new positions?
I'm not sure how slow this would be.
I know PHP.
I know SQL.
I am not very experienced in ActionScript, but I can do basic things like set positions of objects.
I do not know how to retrieve information from a database via Flash.
I do not know how to make Flash send out queries.
Do you think you could give me a bit of help?
It would be safe to use MySQL.
But, I strongly wouldn't recommend using PHP + MySQL as a game server though, or your server will tend to lock up from the influx of requests. The HTTP protocol was not designed for this.
It might take a bit of time, but I would learn an easy programming language (especially something like Java or C#) to create a basic server. Then you can store their user information within RAM, instead of constantly accessing the database repeatedly. But, you could also have it where the server updates a database every n amount of minutes, in case the server is shutdown and needs to be started back up with the same data.
Look up 'Flash Remoting' for flash<->server communications. An open-source server-side handler for that is AMFPHP. Flash would send out AMF messages, AMFPHP translates that back into normal PHP data structures, and then you'd have the PHP code handle interfacing to the database.
you would have php be a controller between your db and flash. So flash would send/receive info from php and php would query db.
Yeah, MySQL is pretty secure, as long as you strip all tags, mysql injection etc from the string. And it should be pretty instant.
However, hundreds of MySQL requests every second will be a lot of bandwidth, although I can't think of any alternatives.
For example I need to grab from http://gmail.com/ the number of free storage:
Over <span id=quota>2757.272164</span> megabytes (and counting) of free storage.
And then store those numbers in a MySql database.
The number, as you can see, is dynamically changing.
Is there a way i can setup a server side script that will be grabbing that number, every time it changes, and saving it to database?
Thanks.
Since Gmail doesn't provide any API to get this information, it sounds like you want to do some web scraping.
Web scraping (also called Web
harvesting or Web data extraction) is
a computer software technique of
extracting information from websites
There are numerous ways of doing this, as mentioned in the wikipedia article linked before:
Human copy-and-paste: Sometimes even
the best Web-scraping technology can
not replace human’s manual examination
and copy-and-paste, and sometimes this
may be the only workable solution when
the websites for scraping explicitly
setup barriers to prevent machine
automation.
Text grepping and regular expression
matching: A simple yet powerful
approach to extract information from
Web pages can be based on the UNIX
grep command or regular expression
matching facilities of programming
languages (for instance Perl or
Python).
HTTP programming: Static and dynamic
Web pages can be retrieved by posting
HTTP requests to the remote Web server
using socket programming.
DOM parsing: By embedding a
full-fledged Web browser, such as the
Internet Explorer or the Mozilla Web
browser control, programs can retrieve
the dynamic contents generated by
client side scripts. These Web browser
controls also parse Web pages into a
DOM tree, based on which programs can
retrieve parts of the Web pages.
HTML parsers: Some semi-structured
data query languages, such as the XML
query language (XQL) and the
hyper-text query language (HTQL), can
be used to parse HTML pages and to
retrieve and transform Web content.
Web-scraping software: There are many
Web-scraping software available that
can be used to customize Web-scraping
solutions. These software may provide
a Web recording interface that removes
the necessity to manually write
Web-scraping codes, or some scripting
functions that can be used to extract
and transform Web content, and
database interfaces that can store the
scraped data in local databases.
Semantic annotation recognizing: The
Web pages may embrace metadata or
semantic markups/annotations which can
be made use of to locate specific data
snippets. If the annotations are
embedded in the pages, as Microformat
does, this technique can be viewed as
a special case of DOM parsing. In
another case, the annotations,
organized into a semantic layer2,
are stored and managed separated to
the Web pages, so the Web scrapers can
retrieve data schema and instructions
from this layer before scraping the
pages.
And before I continue, please keep in mind the legal implications of all this. I don't know if it's compliant with gmail's terms and I would recommend checking them before moving forward. You might also end up being blacklisted or encounter other issues like this.
All that being said, I'd say that in your case you need some kind of spider and DOM parser to log into gmail and find the data you want. The choice of this tool will depend on your technology stack.
As a ruby dev, I like using Mechanize and nokogiri. Using PHP you could take a look at solutions like Sphider.
Initially I thought it was not possible thinking that the number was initialized by javascript.
But if you switch off javascript the number is there in the span tag and probably a javascript function increases it at a regular interval.
So, you can use curl, fopen, etc. to read the contents from the url and then you can parse the contents looking for this value to store it on the datanase. And set this up a cron job to do it on a regular basis.
There are many references on how to do this. Including SO. If you get stuck then just open another question.
Warning: Google have ways of finding out if their apps are being scraped and they will block your IP for a certain period of time. Read the google small print. It's happened to me.
One way I can see you doing this (which may not be the most efficient way) is to use PHP and YQL (From Yahoo!). With YQL, you can specify the webpage (www.gmail.com) and the XPATH to get you the value inside the span tag. It's essentially web-scraping but YQL provides you with a nice way to do it using maybe 4-5 lines of code.
You can wrap this whole thing inside a function that gets called every x seconds, or whatever time period you are looking for.
Leaving aside the legality issues in this particular case, I would suggest the following:
Trying to attack something impossible, stop and think where the impossibility comes from, and whether you chose the correct way.
Do you really think that someone in his mind would issue a new http connection or even worse hold an open comet connection to look if the common storage has grown? For an anonimous user? Just look and find a function that computes a value based on some init value and the current time.