I am currently trying to load an HTML page via cURL. I can retrieve the HTML content, but part is loaded later via scripting (AJAX POST). I can not recover the HTML part (this is a table).
Is it possible to load a page entirely?
Thank you for your answers
No, you cannot do this.
CURL does nothing more than download a file from a URL -- it doesn't care whether it's HTML, Javascript, and image, a spreadsheet, or any other arbitrary data; it just downloads. It doesn't run anything or parse anything or display anything, it just downloads.
You are asking for something more than that. You need to download, parse the result as HTML, then run some Javascript that downloads something else, then run more Javascript that parses that result into more HTML and inserts it into the original HTML.
What you're basically looking for is a full-blown web browser, not CURL.
Since your goal involves "running some Javascript code", it should be fairly clear that it is not acheivable without having a Javascript interpreter available. This means that it is obviously not going to work inside of a PHP program (*). You're going to need to move beyond PHP. You're going to need a browser.
The solution I'd suggest is to use a very specialised browser called PhantomJS. This is actually a full Webkit browser, but without a user interface. It's specifically designed for automated testing of websites and other similar tasks. Your requirement fits it pretty well: write a script to get PhantomJS to open your URL, wait for the table to finish rendering, and grab the finished HTML code.
You'll need to install PhantomJS on your server, and then use a library like this one to control it from your PHP code.
I hope that helps.
(*) yes, I'm aware of the PHP extension that provides a JS interpreter inside of PHP, and it would provide a way to solve the problem, but it's experimental, unfinished, would be still difficult to implement as a solution, and I don't think it's a particularly good idea anyway, so let's not consider it for the purposes of this answer.
No, the only way you can do that is if you make a separate curl request to ajax request and put the two results together afterwards.
Related
<?php
include 'components/server.php';
Is it possible to make it include server.php for desktops and server-mobile.php for mobile devices?
While technically possible, it's absolutely not the best way of doing things.
Why?
Because PHP runs on the server and only the output of that PHP execution is given to the browser. You would probably be wanting something using javascript which can load and then seamlessly react to the browser conditions, such as screen size and/or dimensions.
If you're trying to change which PHP script is running based on the browser criteria (as mentioned above) this sounds very much like your programming logistics are simply wrong.
If you somehow really do need to change PHP script execution based on end-client (browser) characteristics you could do this by calling a script based on javascript AJAX or using mechanisms mentioned in comments above, but as said, you're almost certainly "doing it wrong".
Alternative
It would be far better to load everything you need in PHP and then pass all of that content to the browser (as output; HTML, CSS, Javascript, etc.) for the Javascript in the browser to then decide which parts of the data it needs to use and ignoring the others.
I'd like to create an RPG in JavaScript. The RPG will have a lot of dialog in it. I have heard XML is the way to go..., but I have no experience using XML, and a good amount of experience with PHP and MySQL.
My questions are:
Would it be better to store dialog in a MySQL database and access it with PHP using IDs? Or should I just keep it all on the front end by accessing XML files with Javascript?
If I use XML, do I need to run the javascript on a webserver (Like WAMP)? Or can I parse it locally with:
function parseXML() {
xmlhttp=new window.XMLHttpRequest();
xmlhttp.open("GET","dialog.xml",false);
xmlhttp.send();
xmlDoc=xmlhttp.responseXML;
}
(Which is currently giving the error XMLHttpRequest cannot load file:///C:/Users... Cross origin requests are only supported for HTTP.
Even after reading up on XML, I'm still not sure how I can integrate it with JavaScript. Can someone help give me an example of how I could take a sample rpg XML dialog and meaningfully parse it in JavaScript?
Thank you
Why don't you use JSON instead? It's much easier to use from JavaScript code. While you're at it, avoid usind XMLHttpRequest by hand and use some library that handles the bad stuff for you (e.g., jQuery).
For the "dialog" part, I'm not sure what you're gonna use MySQL for. Anyway, you can use any form of storage on the server side, provided that your PHP scripts give well-formed replies to the client side code.
Would it be better to store dialog in a MySQL database and access it
with PHP using IDs? Or should I just keep it all on the front end by
accessing XML files with Javascript?
The main advantage of having a database here is that you gain a lot of flexibility. You can create/edit/delete a piece of dialogue and its relations quite easily and without having to maintain/regenerate a huge data file every time (As you would do with a XML or JSON or w/e formalism you choose).
On the other hand, this solution might be overkill if you just have simple dialogues with no relations and limited content.
Since you are about to do a game in Javascript, and if your content is not changing that much, you can consider a JSON encoding for your dialogues. You could parse the data instantly via Javascript and still have the possibility to make all kind of complex relations between dialogues as you would in XML.
If I use XML, do I need to run the javascript on a webserver (Like WAMP)? Or can I parse it locally with: ...
You have this error because you indeed need to run a server here. If you just want to do some tests, no need to use WAMP or whatever big webserver, just run a python server for now and that will do the trick.
cd /yourProject
python -m SimpleHTTPServer
Let the server run and open your browser at:
http://127.0.0.1:8000
Give the input file a base url like http://localhost/rpgame/dialog.xml.
I have a small script that pulls HTML from another site using Javascript.
I want to include that static HTML that gets pulled in a PHP page without any of the Javascript code appearing in the final PHP page that gets displayed.
I tried doing an include of the file with the Javascript code in the PHP page, but it just included the actual Javascript and not the results of the Javascript.
So how would I go about doing this?
You would need to fetch the page, execute the JavaScript in it, then extract the data you wanted from the generated DOM.
The usual approach to this is to use a web automation tool such as Selenium.
You simply can't.
You need to understand that PHP and Javascript operate on different places, PHP on the server and Javascript on the client.
Your only solution is to change the way all this is done and use "file_get_contents(url)" from PHP to get the same content your javascript used to get. This way, there is no javascript anymore and you can still pre-process your page with distant content.
You wouldn't be able to do this directly from within PHP, since you'd need to run Javascript code.
I'd suggest passing the URL (and any required actions such as click event, etc) to a headless browser such as Phantom or Zombie, and capturing the DOM from it once the JS engine has done it's work.
You could also use a real browser, but of course you don't need a UI in your case, and it might actually get in the way of what you're trying to do, so a headless browser might be better.
This sort of thing would normally be used for automated testing of a site (ie Functional Testing).
There is a PHP tool named Mink which can run these sorts of scripts from within a PHP program. It is aimed at writing test scripts, but I would imagine you could use it for your purposes.
Hope that helps.
I want to change my HTML page as an image. Is there a way in PHP to change or save an HTML page as an image?
This is not easy; as NullUserException says in his comment, you would need to render the HTML page on the server-side, which is not something PHP (or any other server-sided language) has built in.
The approach that comes to mind would be to write a program (probably not in PHP, but rather something like C# or C++) that runs on your server, fires up a web browser, and does a series of screen captures (possibly combined with page scrolls). As this is a very nontrivial and bug-prone process, I would suggest looking into third-party components that are capable of doing this.
You would then execute this program from PHP, and when it's done running, display the results from the file it output.
I would advise you to use an external service with an api. This list might be a good start: http://blogs.sitepoint.com/2008/07/10/9-ways-to-put-site-screenshots-in-your-web-app/
Thumbalizr seems great, they allso provide a php script so you can cache the images locally:
http://www.thumbalizr.com/apitools.php
Try taking a look at browsershots.org - source code is available for it if you want to install it locally. Essentially it uses a browser to take screenshots, and can be controlled via an XML-RPC interface, which you can call from PHP.
As others have said this is not a simple job, and not something you can do directly in PHP, so use an external service.
(I'm not affiliated with browsershots.org in any way)
While cross-site scripting is generally regarded as negative, I've run into several situations where it's necessary.
I was recently working within the confines of a very limiting content management system. I needed to include database code within the page, but the hosting server didn't have anything usable available. I set up a couple bare-bones scripts on my own server, originally thinking that I could use AJAX to import the contents of my scripts directly into the template of the CMS (thus retaining dynamic images, menu items, CSS, etc.). I was wrong.
Due to the limitations of XMLHttpRequest objects, it's not possible to grab content from a different domain. So I thought iFrame - even though I'm not a fan of frames, I thought that I could create a frame that matched the width and height of the content so that it would appear native. Again, I was blocked by cross-site scripting "protections." While I could indeed load a remote file into the iFrame, I couldn't execute JavaScript to modify its size on either the host page or inside the loaded page.
In this particular scenario, I wasn't able to point a subdomain to my server. I also couldn't create a script on the CMS server that could proxy content from my server, so my last thought was to use a remote JavaScript.
A remote JavaScript works. It breaks when the user has JavaScript disabled, which is a downside; but it works. The "problem" I was having with using a remote JavaScript was that I had to use the JS function document.write() to output any content. Any output that isn't JS causes script errors. In addition to using document.write() for every line, you also have to ensure that the content is escaped - or else you end up with more script errors.
My solution was as follows:
My script received a GET parameter ("page") and then looked for the file ({$page}.php), and read the contents into a variable. However, I had to use awkward buffering techniques in order to actually execute the included scripts (for things like database interaction) then strip the final content of all line break characters (\n) followed by escaping all required characters. The end result is that my original script (which outputs JavaScript) accesses seemingly "standard" scripts on my server and converts their standard output to JavaScript for displaying within the CMS template.
While this solution works, it seems like there may be a better way to accomplish the same thing. What is the best way to make cross-site scripting work specifically for the purpose of including content from a completely different domain?
You've got three choices:
Create a server side proxy script.
Create a remote script to read in remote dynamic HTML. Use a library like jQuery to make this easier. You can use the load function to inject HTML where needed. EDIT What I originally meant for example # 2 was utilizing JSONP, which requires the server side script to recognize the "callback=?" param.
Use a client side Flash proxy and setup a crossdomain.xml file on your server's web root.
Personally, I would call to that other domain on the server and get and parse the data there for use in your page. That way you avoid any problems and you get the power of a server-side language/platform for getting and parsing the data.
Not sure if that would work for your specific scenario...hard to know even with your verbose description...
You could try easyXDM, by including very little code, you can pass data or method calls between documents of different domains.
I've come across that YDN server side proxy script before. It says it's built to work with Yahoo's Search APIs.
Will it work with any domain, if you simply trim the Yahoo API code out? Or do you need to replace it with the domain you want it to work with?
iframe remote content can be accessed by local javascript.
The remote server just have to set the document.domain of the page.
Eg:
Site A contain an iframe with src='Site B/home.php'
home.php looks like this :
[php stuff]...[/php]
[script type='text/javascript']document.domain='Site A'[/script]