I'm a C++ programmer and I'm new to web development.
I need to figure out how I can log/dump the html of a dynamic 3rd party website to a static html file on my computer, every second?
The dynamic webpage refreshes every second and updates a html table with latest price info. I would like a static snapshot of this table (or the whole html page) to be saved to disk every second. That way I can parse the file with my own program and add the updated price info to a database.
How do I do this?
If I cant do it this way, is there a way to eves drop (and log) on the the post/get messages and replies the dynamic webpage sends?
Look into the cURL Library. I believe Scraping the content from a website, and doing your processing/business logic, then inserting or updating your database would be the most efficient way to do it, rather than saving the files contents to disk.
Alternatively, file_get_contents() works pretty well assuming you have allow_url_fopen enabled.
It would be easy to do with Selenium Webdriver. You can use Selenium to create a browser object with a method, getPageSource, that pulls the entire HTML from the page, but it doesn't seem there are any C++ bindings for Selenium. If it's convenient to use Ruby, Python, or Java as part of your application, just in order to open up a browser or headless browser and pull the data, then you should be able to set up a web service or a local file to transfer that data back into your C++ application.
Web automation from C++ addresses the challenge of no Selenium C++ bindings
Or, alternately you could write your own C++ bindings for Selenium (probably more difficult)
However -- for simply pulling the HTML, you may not need Selenium if one of Dan's answers above will work.
Hej someone else.
insed of running there page every second to record there data so you can have a updated
view of there prices, why not call there web service directly (the one there ajax call makes)
Gl
Related
I can't seem to find any other info on this topic anywhere on the interwebz.
SharePoint 365 seems so powerful, looks like an awesome solution for a whole bunch of business IT requirements.
But everything is in ASP and XML, but all my data is inside an external MySQL server, accessible through PHP pages which process and spit out JSON objects. This has worked nice and tidy for all my other apps so far.
How can I get my SharePoint site (which is basically HTML CSS and JavaScript, right?) to hit my PHP server (with HTTP GET or POST) to access my JSON data (and then process and display it however I like)?
I know AJAX can't hit external servers, so on my SharePoint site I'd need something equivalent to cURL perhaps... but with the functionality of AJAX?
Another possibility I'd avoid would be to create some kind of middle-man page/file/app in SharePoint which can process the data locally via AJAX, and send/receive data to/from the external PHP server. But this would all be in C# or ASP, ughhh surely there's an easier way?
EDIT :
Came across this post regarding XML content, the best answer was to "create a custom .NET assembly" to parse the XML... I can't believe SharePoint with all its awesomeness wouldn't be able to handle external JSON content??
You are right that SharePoint at it's heart is just ASP.net engine rendering HTML, JS and CSS down to the browser. For SP to access your data, you would need to expose the MySQL data via a REST service. This could be written in any language since it is a service. It does not need to be C#, .net or ASP as all SP is doing is calling your endpoint and receiving the json data. So no reason you could not use PHP for this.
Once you have the service, you can create a javascript file to call it using jQuery and the $.ajax() or getJSON() method to get the data. You render the data onto the SharePoint page by adding the Script Webpart in the proper location. Once added, click the Edit Snippet link on the script webpart and insert html with a reference to script file and a div to write the results. There are many examples of how to do this type of request and parse the data on the web as it is not something specific to SharePoint. Place the javascript file in the SharePoint site. I typically put it in a custom folder in the _catalogs/masterpage library.
If you just need to display this external data on one page along side your SP data, like in a dashboard, this solution will work. If the external data needs to interact with the SP data or you need to place it in multiple locations, then you may want to look into building a SharePoint App and you can use PHP to do this. Here is a link to a sample project on GitHub. https://github.com/OfficeDev/PHP-App-for-SharePoint
I have been search for a few days now for a way to work on this.
Basically I could like to send flashvars to my swf file using some backend language like PHP or Python, then execute it and capture the output.
I looked up to Ming but it seems that this is only a lib for create SWF file, not much on manipulate already created file.
Thank you!
If you need to do it 'on load basis', you can simply provide flash vars (for example put some flashvars depending on situation and embed the swf into html).
But if you need to have almost real time control over the flash, you have a few options:
socket connections - very hard (not to say impossible) to implement with php, and a kind of complex job
make the swf make a call to php, and get some information; then execute what you need depending on that information
I would advise you to use the second option - chose the action depending on situation, and let the swf client get it itself (either save it in file, db or calculate it realtime - it's up to you; the important is that you don't pass it to client, client makes query to get it).
This is a very broad question, so I'm just looking for the best way of doing this.
I want to periodically monitor certain pages on my website.
I am looking to write a PHP script which will load the page as if it is being loaded in a browser. So that means, it loads all CSS, Javascript, Images, Videos, etc...
I want to just get the load time of these pages and then email the results to myself in a crontab. For this I was going to use microtime() and a phpMailer.
Does anyone know of a script to load a complete page, or have any suggestions on how to go about this?
Thanks.
What if the page has dynamic content? You will also need to execute all the JavaScript and fetch all CSS images to get the final amount of time. I believe that is impossible using only PHP.
A php script you run from the same server you host your site will give you abnormal readings (very low) since it's loading on the first hop essentially. What you really would want to do is run a script from various servers outside of your own. There are also limitations with what php can see ie JS and JQuery etc.
The simplest is to check from your home pc, using jmeter. You set your home browser to use it as a proxy and go to whichever website you want. Jmeter will record statistics. When you are happy you can choose to save the stats.
This avoids the problems of handling JS and JQuery through a script.
This could get very complicated. You'd basically have to parse the HTML, and then there's tons of edge cases, like JS including resources, etc... I would definitely recommend using something like the network tab of Chrome's dev tools instead.
I'm working on a biology web based application and trying to figure out what language to use. The features I need to include are:
Image viewing frame - This area will display the current image that the biologists wish to see. The application needs to take in a number of coordinates from a file and draw those points on the image displayed here. When the biologist wishes to change images there needs to be no flickering from the refresh. Will do this using multiple image buffers probably. Content needs to be scrollable and able to be zoomed in.
There need to be labeled buttons that advance, step back, zoom, and play the images displaying in the image frame. There also needs to be some type of list view where images titles can be selected to be displayed.
There will be a bunch of folders of images on the server that can be selected from. The application must allow the user to select which folder of images to be loaded. It also must be able to read from either an txt or xml file and visually display the information there by way of line graph.
Would like to be able to run scripts on the server from the application.
I feel that all these things are doable by a web application but I have no idea what language to use. Most people recommend php, but i don't want to delve deeper until I know what its limitations are. Any suggestions are welcome. Thanks in advance.
-Mike
PHP can do everything you need for the back end, but most of the stuff that you describe is UI based, and this is dependent on the client, which is, of course, the browser. For highly graphical projects, you can do a lot in JavaScript and some JavaScript libraries have a lot of these capabilities built in. You might also consider Flash or Flex.
You might even consider a desktop application that runs outside of the browser. You can use Java, which is easy to deploy, but still requires the user to have the Java Runtime Engine, or you could go with a language that you can compile down to a native application.
Regardless of the front end technology that you choose, you'll still need a back end, and PHP can handle this.
You will find almost every server side platforms such as php , asp.net, asp, etc will do all of the above.
PHP is a language that resides on the server and handles all requests. Javascript (and associated libraries) is a language which is executed by the client's browser and handles (almost) all interaction. PHP is definitely able to do what you want, but for the interaction stuff (particularly the zoom, scrolling, etc.), you'll also need to use Javascript.
So, short answer, PHP is good, but you're going to need to use client-side scripting as well.
PHP is more than capable of doing this. You are going to need to use it in combination with some Javascript to handle the client side effects you describe. I would look into modifying galerific for your needs and then whip up some javascript to write points over the images.
From your concerns about image refresh/flicker, it really sounds like a desktop app is what you are looking for, for a rapid response on image changes. The requirements on this really seem to need to be defined better before you can choose a language... PHP can do all the server side stuff you mentioned, but you might have a harder time getting the image viewing "frame" to provide the functionality you want.
Due to the image manipulation requirements it might be easier to go with something like flash with a php backend or asp.net with silverlight. It might be difficult to prevent flicker and delays with using pure javascript as opposed to flash/silverlight.
Image viewing frame
This will most likely need to be done on the client side using tools/frameworks such as jQuery, the canvas element, silverlight, or any of the other 100's that are out there.
There need to be labeled buttons that advance, step back, zoom, and play the images displaying in the image frame. There also needs to be some type of list view where images titles can be selected to be displayed.
PHP or any other server-side scripting language could pull this off. If this is meant to be a quick project running on free/cheap hardware then PHP would be a good choice. If the plan is a large application that will have to be maintained over the course of many years and hosting/price is not an issue then I would suggest something like ASP.NET
There will be a bunch of folders of images on the server that can be selected from. The application must allow the user to select which folder of images to be loaded. It also must be able to read from either an txt or xml file and visually display the information there by way of line graph.
Again any server side language could do the folder listing portion. As for reading files and creating graphs, this would most likely be a combination of server side and client side programming. jQuery for example, has plugins that could quite easily take a xml file and create a line graph.
Would like to be able to run scripts on the server from the application.
PHP, ASP.NET - both could do this. I'm sure many others could, but these are the ones i use most often
The issue with PHP is that quite often, the code turns into a mess over time. This is maybe not so much an issue with the language as the people using it and the purpose the app was built for (a quick, one time project). Classic ASP also has the same issues.
ASP.NET is a good combination of OOP programming that allows you to separate presentation from logic with minimal effort.
I have found some libraries or web services in PHP that does the job. The problem is that the conversion is done when the page is fully loaded, I would like to convert the page to PDF after some content dynamically added via AJAX in onload event.
Thank you very much,
Omar
You could post back document.getElementsByTagName('html')[0].innerHTML to the server (possibly using AJAX) and generate a PDF from that.
When you add content with AJAX, that's only happening on the client's machine, so the server-side PHP conversion code will not work with it. You'll need to come up with a server-side method of including the dynamic content if you want to do the PDF creation on the server.
You could possibly implement an AJAX call that sent the page content/state back to serverside, after the dynamic contens is added, where it could be rendered as a pdf. You might not need all the page, depending on where this "dynamic content" is going to go.
Seems a bit messy, but without knowing the project it's hard to say whether there could be a cleaner method to do what you're trying to do.
This is a client-side requirement due to the Ajax situation. If they're using a Mac they can simply save as PDF but otherwise you have little control over what they can do.
What you can do however for a server-side implementation is keep a record of the page and all their AJAX requests since loading that page, and construct the HTML and PDF on the server from that record if it is requested. Of course this is not a simple task and quite overkill. You'd probably be better off having a different mechanism to create a server-side PDF report of a page other than turning HTML into PDF.
If your page can be updated solely on the client with no trip to the server, you'll have to post back up to your app. That way you'll have all the content and will not have rebuild, which would not be possible if there are client side only interactions.
Wow, thank you everyone I didn't know that this is comunity is so active. To answer Beepcake about the project:
When the page loads the app retrieves, from more than 40 servers, biological information via AJAX request, then a unique view is displayed where you can manipulate the graphic with many options.
So, the cool thing will be to print when the user makes his own version of the graphic. I think that the best solution is to POST the entire HTML with document.getElementsByTagName('html')[0].innerHTML as RoBorg has said and then generate the PDF with a library such as dompdf