I was looking at Twitter Streaming API for getting a real-time feed.
But I dont want it stored on my server.
I just want it pulled from the server and the browser page will retrieve the data from my server's twitter pull URL.
But I want to avoid polling my server every few mill-seconds.
Is there a way for my server script to keep pushing to my browser page ?
Just how live do you want it? There are ways to set up sockets, but they can be fairly complex, and still consume their fair share of bandwidth.
Is it acceptable to poll every 5, 10 seconds or so? Every few milliseconds would give you pretty darn "live" results, but I would not be upset as a user if it took a matter of seconds for something to appear on your website. That would be satisfactorily "live" for me.
Check out COMET.
In web development, Comet is a neologism to describe a web application model in which a long-held HTTP request allows a web server to push data to a browser, without the browser explicitly requesting it.
I've always wanted to try this method but I haven't gotten around to it:
Hidden IFrame
A basic technique for dynamic web application is to use a hidden IFrame HTML element (an inline frame, which allows a website to embed one HTML document inside another). This invisible IFrame is sent as a chunked block, which implicitly declares it as infinitely long (sometimes called “forever frame”). As events occur, the iframe is gradually filled with script tags, containing JavaScript to be executed in the browser. Because browsers render HTML pages incrementally, each script tag is executed as it is received.[8]
One benefit of the IFrame method is that it works in every common browser. Two downsides of this technique are the lack of a reliable error handling method, and the impossibility of tracking the state of the request calling process.[8]
Related
I have a web system (with php) in which I must implement an audio streaming. I have seen many solutions that lead to the use of web-sockets and node.js, but for certain reasons I can not make use of it.
Will there be any way to do it with ajax? That has as recorded, saved on server and that can be consulted by customers. (for example)
Based on your question, I don't think ajax is going to be the solution. The reason being, that ajax is not a continuous stream, but rather a callback between your end users client (browser) and the server. Once the ajax call completes, the connection to the server ends. You may be able to find a drop in jquery solution to play the audio client side, but the best you'll be able to do with ajax is possibly utilize it as a means of delivering files/references to files. For example, you COULD use ajax to call back to the server, get file data and pass back the data to the client side script that handles the audio playback. Also, not sure why this question got a downvote, as it's a perfectly valid question.
I am creating an application that will use Web Sockets for a notification system. Is it better to have the application in an iframe with the Web Sockets in the parent so there isn't a new connection every time a page is loaded? Or maybe it should re-connect?
What are your thoughts?
If anyone has any other way in PHP to get push like notifications without sending a AJAX request every 10 seconds then let me know.
Thanks.
This is one of the options you have described.
The problem with that option is that there wont be direct control over the content of that inner iframe, and you will need to implement push message window communication between parent of iframe, in order to be able to change iframe src attribute, in case someone will refresh parent page, and iframe should refresh to actual state, not initial page.
Second problem, there will be no SEO at all. So your page wont be crawlerable by search engine robots. If SEO is important for your application - then this is not an option.
In WebSockets, if you work with sessions, it is important to make session available for normal PHP script and WebSockets logic, in order to keep consistent access to data it self. PHP will make it not an easy task at all.
You might consider Long Pull technique as well, as it allows to open one AJAX request and then get responses back, and this request can last some time but will eventually close and have to be reopened on client-side.
Another option is to review actual application architecture, and think of single-page application. It have as well cons and pros.
Good thing about it, is UX will be much higher. Response times as well as you will load less content and data.
Pros are that it requires lots of development on front-end side in javascript. As well there is two major routes you can do single-page applications. Consistent and inconsistent. In first case you need to make sure that your back-end will server static html on refresh or just navigating to specific link, the same way as your single-page application would generate using java-script. Then it solves issues with SEO. While inconsistent approach, will just be purely on javascript (front-end), and will have issues with SEO.
WebSockets usually used with single-page applications, for example Facebook Chat is great example of such. Or Google Talk while you are in Gmail account.
They are not meant to be used with often refreshed pages, as Handshake process is a bit more heavy than normal HTTP request.
I am starting a long polling request for every page of my website. It looks every second at the database and if there is something new it marks it as seen and outputs the notification. The calling JavaScript is then starting a new ajax request.
Now I got problems having opened multiple tabs on the website, because only one will recieve a new notification. This is also a problem cross-browser with the same username logged in...
What would be the smartest way to solve this fool-proof?
Than you for your input!
I think it is better to avoid browser pulling. You will have browsers problems and also your infrastructure should be huge to support it.
Try a server side pushing tech like Commet,
Comet is a web application model in which a long-held HTTP request
allows a web server to push data to a browser, without the browser
explicitly requesting it.
Other approach could be using WebSockets.
I've done web scraping before but it was never this complex. I want to grab course information from a school website. However all the course information is displayed in a web scraper's nightmare.
First off, when you click the "Schedule of Classes" url, it directs you through several other pages first (I believe to set cookies and check other crap).
Then it finally loads a page with an iframe that apparently only likes to load when it's loaded from within the institution's webpage (ie arizona.edu).
From there the form submissions have to be made via buttons that don't actually reload the page but merely submit a AJAX query and I think it just manipulates the iframe.
This query is particularly hard for me to replicate. I've been using PHP and curl to simulate a browser visiting the initial page, gather's the proper cookies and such. But I think I have a problem with the headers that my curl function is sending because it never lets me execute any sort of query after the initial "search form" loads.
Any help would be awesome...
http://www.arizona.edu/students/registering-classes -> "Schedule of Classes"
Or just here:
http://schedule.arizona.edu/
If you need to scrape a site with heavy JS / AJAX usage - you need something more powerful than php ;)
First - it must be full browser with capability to execute JS, and second - there must be some api for auto-browsing.
Assuming that you are a kid (who else would need to parse a school) - try Firefox with iMacros. If you are more seasoned veteran - look towards Selenium.
I used to scrap a lot of pages with JS, iframes and all kinds of that stuff. I used PhantomJS as a headless browser, that later I wrapped with PhantomCurl wrapper. The wrapper is a python script that can be run from command line or imported as a module
Are you sure you are allowed to scrape the site?
If yes, then they could just give you a simple REST api?
In rare case when they would allow you to get to the data, but would not provide API, my advice would be to install some software to record your HTTP interaction with web site, maybe wireshark, or some HTTP proxy, but it is important that you get all details of http requests recorded. After you have that, analyze it, and try to replay it up to the latest bit.
Among possible chores, it might be that at some point in time server sends you generated javascript, that needs to be executed by the client browser in order to get to the next step. In this case you would need to figure how to parse received javascript, and figure out how to move next.
An also good idea would be not to fire all your http requests in burst mode, put put some random delays so that it appears to the server more "human" like.
But in the end you need to figure out if all this is worth the trouble? Since almost any road block to scraping can be worked around, but it can get quite involved and time consuming.
I have been working on a site that makes some pretty big use of AJAX and dynamic JavaScript on the front end and it's time to start stress testing. But how do you properly stress test something that requires clicking several links on the front-end? One way I was able to easily hit every page of the site quickly and repeatedly was to point a Google Mini at it. But that's not going to click links and then navigate Modal windows and things like that.
Edit - I should point out that the site is done in PHP5 and the JavaScript library used is jQuery. Not sure if this would make any difference but felt it might be useful to know.
JMeter is great at this. You may record your sessions and tweak them to your liking.
So-called 'ajax load testing' is a recurring subject on this site, and is often confused. So let's get it straight: There is really no difference between load testing a normal web page and load testing with ajax. It all boils down to discrete requests; they just happen to not be full page refreshes.
One thing to keep in mind is there is a distinct difference between load testing the server processing the requests (a load test) and the performance on screen of the UI components being updated (how well your javascript performs.)
Simple load test example:
initial page load
login
navigate?
5-10 'ajax' requests (or whatever may fit your application usage pattern)
logout
There are load testing tools that can support AJAX. For example, WebLoad
http://www.radview.com/solutions/ajax-load-testing.aspx
What you really want is to stress test is the server's ability to handle the ajax requests. Use a load tool that looks at the requests while "recording" the test, and then tune as appropriate. I have only used the vs test edition one, so I can't point you to a low cost one.
I disagree with Nathan and Freddy to some degree. They are correct that "AJAX testing" is really no different in that HTTP requests are made. But it's not that simple. See my article on Ajaxian.com on Why Load Testing Ajax is Hard.
JMeter, Pylot, and The Grinder are all great tools for generating HTTP requests (I personally recommend Pylot). But at their core, they don't act as a browser and process JavaScript, meaning all they do is replay the traffic they saw at record time. If those AJAX requests were unique to that session, they may not be suitable/correct to replay in large volumes.
The fact is that as more logic is pushed down in to the browser, it becomes much more difficult (if not impossible) to properly simulate the traffic using traditional load testing tools.
In my article, I give a simple example of how difficult it becomes to test something like Google's home page when you want to query 1000's of different search terms (an important goal during load testing). To do it with JMeter/Pylot/Grinder you effectively end up re-writing parts of the AJAX code (in your case w/ jQuery) over again in the native language of the tool.
It gets even more complex if your goal is to measure the response time as perceived by the user (which is arguably the most important thing at the end of the day). For really complex applications that use Comet/"Reverse Ajax" (a technique that keeps open sockets for long periods of time), traditional load tools don't work at all.
My company, BrowserMob, provides a load testing service that uses Firefox browsers, powered by Selenium, to drive hundreds or thousands of real browsers, allowing you to measure and time the performance of visual elements as seen in the browser. We also support traditional virtual users (blind HTTP traffic) and a simulated browser (via HtmlUnit).
All that said, usually a mix of a service like BrowserMob plus traditional load testing is the right approach. That is, real browsers are great for a full-fidelity load test, but they will never be as economical as "virtual users", since they require 10-100X more RAM and CPU. See my recent blog post on whether to simulate or not to simulate virtual users.
Hope that helps!
You could use something like openSTA.
This allows a session with a web site to be recorded and then played back via a relatively simple script language.
You can also easily test web services and write your own scripts.
It allows you to put scripts together in a test in any way you want and configure the number of iterations, the number of users in each iteration, the ramp up time to introduce each new user and the delay between each iteration. Tests can also be scheduled in the future.
It's open source and free.
It produces a number of reports which can be saved to a spreadsheet. We then use a pivot table to easily analyse and graph the results.