OK, so I bought a 46" screen for the office yesterday, and with the imminent risk of being accused for setting up an "elaborate World Cup procrastination scheme", I'd better show my colleagues what it's meant for ;)
Looking at my simple sketch, and at these great projects from which I was inspired, I would like to get some input on the following:
Pseudocode for the skeleton: As some methods should be called every 24 hours ("Today's date in the heading"), others at 60 second intervals ("Twitter results"), what would be a good approach using JavaScript (jQuery) and PHP?
EDIT: Alsciende: I can agree that #1 and #8 are too vague. Therefore I remove #8 and try to clarify #1: With "Pseudocode for the skeleton", I basically mean could this be done entirely using JavaScript timers and how would you set up the various timers?
Library for Google Analytics: Which libraries support the Google Analytics API and can produce neat charts. Preferably HTML5, JavaScript-based like Protovis.
Library for Twitter: Which libraries would you recommend for fetching twitter search results and latest tweets from profiles.
Libraries for Typography/CSS/HTML5: Trying to learn some HTML5 etc. in the process, please advice on any other typography/css libraries that could be of relevance.
Scraping/Parsing? I'll give you a concrete example: Trying to fetch today's menu from this restaurant's website, how would you go about? (it's in Swedish - but you get the point - sorry ;) )
Real-time stats? I'm using the WassUp-plugin for WordPress to track real-time visitors on our website. Other logging software (AWStats etc.) is probably also installed on the webserver. Any ideas on how to extract information from these and present in real-time on the dashboard?
Browser choice? Which Browser and OS would you pick? Stable, Full-screen, HTML5.
alt text http://www.freeimagehosting.net/uploads/cb7af2ef28.png
I've built a dashboard similar to what you're talking about for our office. I spent about a day working on it, the possibilities are really (pretty much) endless. Basically, all the computational stuff I handle via PHP and do interval AJAX calls to the appropriate PHP script, which returns JSON data to present.
#2:
For graphs, I use/recommend flot (http://code.google.com/p/flot/). The documentation isn't really that great, but once you figure out how things generally work - it's a great library, and it generates graphs using HTML5 Canvas tag.
I've not integrated external libraries with Google Analytics before, but I assume you could pull data from analytics and format it for flot to build appropriate graphs for. This might be the hard-way around, but I'm more familiar with flot than most other graphing libraries (and it doesn't suck, as much as a lot of others do) so for me, this would be the easiest way to get it done.
#3:
For twitter, it's pretty easy to pull data from their search API using JSON-P. Basically what this does, is dynamically adds a <script> tag to your DOM, which GET parameters that twitter interprets, then calls a predefined javascript method (which you pass via the URI) with a json-encoded hash of the results.
#5:
Scraping and parsing individual sites is going to be a painstaking process. Every site is going to have it's own "pattern" (or non-pattern) for publishing their daily menu or specials. I'd build a "menu" script which knew how to call a few functions, and write a function / class for scraping each restaurant's site which you're interested in showing the menu for in PHP (or whatever other language you're comfortable with). It can reply with json, which is (imo) the easiest way to manipulate/process data in Javascript.
#6:
Real time stats are pretty much the same as #5. I'd build a couple classes that knew how to fetch the statistics from whatever data sources I was interested in pulling from, and present the data in json to javascript, via an ajax call.
#1: Writing javascript code to load data on a timer is really simple, look into the setInterval, clearInterval, setTimeout, and clearTimeout methods. They all take a function name (or closure) and a timeout to wait before calling that function (in ms). You could easily call a master timer function every 60 seconds which would basically be a "scheduler" or a "cron" style function, which would just look for the stuff that needed to run "right now" and execute those functions from the scheduler.
Hopefully this gives you some ideas on where to go, and how to go about getting there.
For the Rails Rumble we developed Boarrd that is exactly what you would like to develop!
We were impressed by Panic too :)
In our team page on RailsRumble you'll find details about the tools used. I know that is not in PHP but maybe you'll try our tool and decide for better development environment ;)
Related
Currently I have made a system where many screens are just lists, which update every N seconds depending on how many results per page the user has selected.
This is just a simple ajax call in a setTimeout loop. The main disadvantage I see with this approach I have used is that it is super inefficient in that it creates a lot of unnecessary network traffic and database calls.
The reason for this is that the data on the screen may change at any time, but the user needs to be aware if something new does come into the list so they can react to it as quickly as possible.
I was thinking of creating a Twitter style list where all you get is little notification at the top, which is just a COUNT of all the new items that you haven't seen. The user would then click on them to have the list refresh with the new items included.
The problem I have with this is that it is not truly a real-time list.
I have just started learning about Socket.io and NodeJS but I think it might take me some time to learn what I need to implement a real-time list that is efficient.
What are your thoughts?
Take the time to learn Node.js and Socket.io. If you are looking for realtime between the browser and the server you can't really beat that combo. Socket.io is very efficient and will choose the best option the browser supports (polling, websockets, etc).
The documentation for Socket.io is a little thin on good examples, but you can try their examples or the HowToNode article.
You'll most likely want to put Express in there as well, it's a fantastic module.
Good luck.
I am defining out specs for a live activity feed on my website. I have the backend of the data model done but the open area is the actual code development where my development team is lost on the best way to make the feeds work. Is this purely done by writing custom code or do we need to use existing frameworks to make the feeds work in real time? Some suggestions thrown to me were to use reverse AJAX for this. Some one mentioned having the client poll the server every x seconds but i dont like this because it is unwanted server traffic if there are no updates. I was also mentioned a push engine like light streamer to push from server to browser.
So in the end: What is the way to go? Is it code related, purely pushing SQL quires, using frameworks, using platforms, etc.
My platform is written in PHP codeignitor and DB is MySQL.
The activity stream will have lots of activities. There are 42 components on the social networking I am developing, each component has approx 30ish unique activities that can be streamed.
Check out http://www.stream-hub.com/
I have been using superfeedr.com with Rails and I can tell you it works really well. Here are a few facts about it:
Pros
Julien, the lead developer is very helpful when you encounter a problem.
Immediate push of new feed entries which support PubSubHubHub.
JSon response which is perfect for parsing whoever you'd like.
Retrieve API in case the update callback fails and you need to retrieve the latest entries for a given feed.
Cons
Documentation is not up to the standards I would like, so you'll likely end up searching the web to find obscure implementation details.
You can't control how often superfeedr fetches each feed, they user a secret algorithm to determine that.
The web interface allows you to manage your feeds but becomes difficult to use when you subscribe to a loot of them
Subscription verification mechanism works synchronous so you need to make sure the object URL is ready for the superfeedr callback to hit it (they do provide an async option which does not seem to work well).
Overall I would recommend superfeedr as a good solution for what you need.
I want to develop an application that does a bunch of cool stuff. The first thing that I need in it is to get information about the page a person is browsing.
With that said, I need for example to know how long a user stayed in a page and where was the scrollbar. While getting that data, It's all saved to a database.
The thing is, I prefer doing that in Flash [although I have no experience in it] over Ajax since I want to hide the code - which as far as I know not possible in Javascript/Ajax.
So, can I do all that in Flash? - Read the content of the page, get the status of the scroll bar..
Plus, I then need to go threw the gathered information that is saved in the database. Since there could be many calculations i thought C++ .Net is better than PHP [which I know better].
Is that all possible or am I just crazy? :)
Thanks ahead.
Server side
I think it doesn't matter whether you'll use PHP/C++/Java/Ruby/Python/whatever... each of these is fast enough to do complex calculations, especially if we talk about pure math.
So if PHP is what you know the best, then it's obvious to use it.
Client side
Flash is pretty cool for animations and others visual effects, but for things like scroll position, time spend on website JavaScript is just better. It doesn't require third-parties plugins, it's integrated with DOM. Personally I just thing JavaScript is the most proper tool for this task.
I vote for JavaScript, you can do in JS all that you mention, using the DOM, and it has not a proprietary license.
Although you cannot compile JS code, obfuscation tools offers a decent level of protection. Closure is worth of mention too, YUY minifier etc.
Also check this ready made JS heatmap.
I advise against using C++ for server side programming. You'll be better off with Ruby/PHP etc.
I know this seems like a really subjective question, but from what I'm reading, I'm generally confused. We are developing a web-based application that has tabbed windows, etc-very much a "desktop-like" application graphically. About half the page never or rarely changes, and the other half is content that could be handled dynamically within a main window.
So from the ajax proponents, this seems like a perfect example of something that would benefit from ajax. The only thing stopping me (other than lack of ability :-D) is that I'm concerned we would degrade the speed of the app (which isn't really all that great anyway...)
Can anyone either shed some light on this subject or point me to more reading material on performance differences when changing lots of data at once via ajax?
About the differences that can often be seen :
If you load more data via AJax requests, and less via "full-page" requests, it means :
less data will go through the network (only content of pages, not menus, sidebars, ...)
less thins to generate on the PHP/SQL side (same : no menu, ...)
But more Ajax requests can also mean more requests, depending on how your application is developped
Which can mean more load on the server...
You also might want to think about :
Indexation by search engines,
Accessibility,
Graceful degration,
and all the possible stuff like that...
Which, in the end, can mean you'll maybe spend lots of additionnal time developping the "more Ajax" version ; and time is money -- and a few days of developper-time can cost more than renting/buying a second server.
In the end, I would say what really matter is not "performance" : you'll probably have greater gains spending a couple of days optimizing what you already have (profiling on the backend side, minifying/compressing JS/CSS, merging JS/CSS files to have less HTTP requests, ...) than by transforming everything to "full-Ajax".
And, of course, what really matters is user-experience !
As someone who just finished developing a similiar application i can say that using ajax to create a "desktop - like" application.... well... looks cool....
But there are certain points that you should be considering before you start...
Dealing with forms when you a load a page with ajax will require the additional knowledge of a javascript framework like Jquery.... Because loading a page portion of a page with ajax does not affect the DOM tree of the page and hence the newly added elements will not be identified by javascript functions.... That's where you will be needing the livequery plugin for jquery.
If you are looking do develop a site optimized for search engine indexing.... ajax - refreshed content may not be a good idea. I developed only the backend part. i.e the administrator module like a desktop application. The frontend where the user actually interact was your normal, day-to-day, website...!!
I have been working on a site that makes some pretty big use of AJAX and dynamic JavaScript on the front end and it's time to start stress testing. But how do you properly stress test something that requires clicking several links on the front-end? One way I was able to easily hit every page of the site quickly and repeatedly was to point a Google Mini at it. But that's not going to click links and then navigate Modal windows and things like that.
Edit - I should point out that the site is done in PHP5 and the JavaScript library used is jQuery. Not sure if this would make any difference but felt it might be useful to know.
JMeter is great at this. You may record your sessions and tweak them to your liking.
So-called 'ajax load testing' is a recurring subject on this site, and is often confused. So let's get it straight: There is really no difference between load testing a normal web page and load testing with ajax. It all boils down to discrete requests; they just happen to not be full page refreshes.
One thing to keep in mind is there is a distinct difference between load testing the server processing the requests (a load test) and the performance on screen of the UI components being updated (how well your javascript performs.)
Simple load test example:
initial page load
login
navigate?
5-10 'ajax' requests (or whatever may fit your application usage pattern)
logout
There are load testing tools that can support AJAX. For example, WebLoad
http://www.radview.com/solutions/ajax-load-testing.aspx
What you really want is to stress test is the server's ability to handle the ajax requests. Use a load tool that looks at the requests while "recording" the test, and then tune as appropriate. I have only used the vs test edition one, so I can't point you to a low cost one.
I disagree with Nathan and Freddy to some degree. They are correct that "AJAX testing" is really no different in that HTTP requests are made. But it's not that simple. See my article on Ajaxian.com on Why Load Testing Ajax is Hard.
JMeter, Pylot, and The Grinder are all great tools for generating HTTP requests (I personally recommend Pylot). But at their core, they don't act as a browser and process JavaScript, meaning all they do is replay the traffic they saw at record time. If those AJAX requests were unique to that session, they may not be suitable/correct to replay in large volumes.
The fact is that as more logic is pushed down in to the browser, it becomes much more difficult (if not impossible) to properly simulate the traffic using traditional load testing tools.
In my article, I give a simple example of how difficult it becomes to test something like Google's home page when you want to query 1000's of different search terms (an important goal during load testing). To do it with JMeter/Pylot/Grinder you effectively end up re-writing parts of the AJAX code (in your case w/ jQuery) over again in the native language of the tool.
It gets even more complex if your goal is to measure the response time as perceived by the user (which is arguably the most important thing at the end of the day). For really complex applications that use Comet/"Reverse Ajax" (a technique that keeps open sockets for long periods of time), traditional load tools don't work at all.
My company, BrowserMob, provides a load testing service that uses Firefox browsers, powered by Selenium, to drive hundreds or thousands of real browsers, allowing you to measure and time the performance of visual elements as seen in the browser. We also support traditional virtual users (blind HTTP traffic) and a simulated browser (via HtmlUnit).
All that said, usually a mix of a service like BrowserMob plus traditional load testing is the right approach. That is, real browsers are great for a full-fidelity load test, but they will never be as economical as "virtual users", since they require 10-100X more RAM and CPU. See my recent blog post on whether to simulate or not to simulate virtual users.
Hope that helps!
You could use something like openSTA.
This allows a session with a web site to be recorded and then played back via a relatively simple script language.
You can also easily test web services and write your own scripts.
It allows you to put scripts together in a test in any way you want and configure the number of iterations, the number of users in each iteration, the ramp up time to introduce each new user and the delay between each iteration. Tests can also be scheduled in the future.
It's open source and free.
It produces a number of reports which can be saved to a spreadsheet. We then use a pivot table to easily analyse and graph the results.