I'm trying to parse data from http://skytech.si/
I looked around a bit and I find out that the site uses http://skytech.si/skytechsys/data.php?c=tabela to show data. When I open this file in my browser I get nothing. Is the file protected and can run only from server side or something?
Is there any way to get data from it? If I cold get HTML data (perhaps in a table?) I would probably know how to parse it.
If not, would it be still possible to parse website and how?
I had a look at the requests made;
http://skytech.si/skytechsys/?c=graf&l=bf0b3c12e9b2c2d65bd5ae8925886b57
http://skytech.si/skytechsys/?c=tabela
Forbidden
You don't have permission to access /skytechsys/ on this server.
This website doesn't allow 'outside' GET requests. You could try parsing the data via file-put-contents but I don't think you will be able to get specific data tables (aside from those on that home) due to AJAX requests that need to be made. I believe the /data? is the controller to handle data which is not exposed via the API.
When you open this URL in your browser you send GET request. Data returned under this address is accessible after sending POST request with params as follows c:tabela, l:undefined, x:undefined. Analyze headers next time and look on Network log if you are using Chrome/Chromium.
If that website does not expose an API, it is not recommended to parse the data, as their HTML structure is prone to change.
See:
http://php.net/manual/en/function.file-put-contents.php
And then you can interpret it with an HTML-parsing engine or with an regular expression (not recommended).
Related
I need to store XML data sent over HTTP POST to my server. In the log files I see that the data is successfully sent to my server. But I have no idea how to get the data.
I tried to catch them with the php://input stream like in the code below. The problem I see is that php://input is just read when the file containing the code is called.
$xml = file_get_contents("php://input");
$var_str = var_export($xml, true);
file_put_contents('api-test/test.txt', $var_str);
Is there any way to set some kind of listener/watcher to the php://input stream? Maybe PHP is the wrong technology to realize this. Is there some other way like AJAX?
The problem I see is that php://input is just read when the file containing the code is called.
Yes.
That's how PHP (in a server-side programming context) works.
The client makes an HTTP request to a URL
The server receives the HTTP request and determines that that URL is handled by a particular PHP program (typically by matching the path component of the URL to a directory and file name unless the Front Controller Pattern is being used)
The PHP program is executed and has access to data from the request
The server sends the output of the PHP program back
Is there any way to set some kind of listener/watcher to the php://input stream?
You get a new stream every time a request is made. So the typical way to watch it is to put a PHP script at the URL that the request is being made to.
Then make sure each request is made to the same URL.
(If you need to support requests being made to different URLs, then look into the Front Controller Pattern).
Maybe PHP is the wrong technology to realize this.
It's a perfectly acceptable technology for handling HTTP requests.
Is there some other way like AJAX?
Ajax is a buzzword meaning "Make an HTTP request with JavaScript". Since you are receiving the requests and not making them, Ajax isn't helpful.
I need to get the contents of a website through PHP, however, the content is only available when JavaScript is enabled. The workaround that I am using now is making an applescript to open the website in Safari, and selecting all of the page content, copying it to the clipboard, and pasting it.
That will be really hard to achieve I guess. If you observe the JS on that page that is responsible for getting the content ready, you may discover its just another AJAX call that you may be able to call directly from your PHP script.
best possible solution: ask the website owner for api/export access ;)
If that is not possible, you can only pray that you can analyze the requests that are initialized via JavaScript and imitate them.
(possible tools: firefox with firebug or tamper data plugin).
Warning the owner of the website might not like this approach, in fact, it may be disallowed to scrape the data automatically
What do you mean by:
the content is only available when JavaScript is enabled
Does the page pull data from somewhere via JS? Would it be easier to analyse where the data is coming from and access that place directly from PHP?
I need to display information from an SQL database which resides on a server, to a remote webworks mobile device. I am extremely new to passing information from a server so bear with me. My normal understanding is that I would have an HTML file that accesses a php script which then itself connects to the database and displays the information.
However, in webworks the HTML/Javascript files reside on the device and are separated from any php file so I need a method to communicate to get the data from the database. I have looked through JSON and read all the tutorials on w3schools and I understand the syntax but I don't understand how to use it. How could it connect to a database? My aim is to simply display the table entries on a mobile device app running HTML5 webworks. Again I am very new to this so any explanation would be very helpful.
Chances are, you should get a book. This is not something that can be explained in detail in a short answer on this site.
in summary however, you can either
1) send requests to your php script by submitting a form on an html page, which will load a new page filled with whatever PHP sends back. in this case you do not need to use JSON at all as PHP would be returning a full html page.
2) you can use AJAX. AJAX is a javascript method of sending requests to the server (PHP), and getting a response without ever loading a new page. you would use AJAX to send a request to the php page, the php page would access the database and send back a response, the javascript would then take the response and do whatever it needs with it. the response data is usually formatted in a JSON format, because PHP can easily create JSON, and javascript can easily decode JSON once it receives it as a response. to make using AJAX simpler, you may want to look in to using jQuery, a javascript library which can simplify the process.
i'm trying to grab the content of a website based on ajax and https but with no luck.
Is this possible.
The website i'm trying to crawl is this:
https://www.bet3000.com/en/html/home.html#!https://www.bet3000.com/html/en/eventssportsbook.html?category_id=2117
Thanks
If you take a look at the HTTP requests that this page is doing (using, for example, Firebug for Firefox), you'll notice it makes several Ajax requests.
Instead of trying to execute the Javascript code, a possible solution could be for you to request one of those URLs, and get the data -- you'd also not have to parse the HTML, this way.
In this specific case, one of those requests is made to the following URL :
https://www.bet3000.com/ajax/en/sportsbook.json.html?category_id=2117&offset=&live=&sportsbook_id=0
This URL seems to return some JSON data, that should interest you quite a bit ;-)
(There is a few characters before and after the JSON, that will need to be removed, but, asides from that, I don't see anything that doesn't look good.)
What i basically want to do is to get content from a website and load it into a div of another website. This should be no problem so far.
The problem is, that the content that should be fetched is located on a different server and i have no source access to it.
I'd prefer a solution using JavaScript of jQuery.
Can i use a .htacces redirect to fetch the content from a remote server with client-side (js) techniques?
I will also go with other solutions though.
Thanks a lot in advance!
You can't execute an AJAX call against a different domain, due to the same-origin policy. You can add a <script> tag to the DOM which points at a Javascript file on another domain. If this JS file contains some JSON data that you can use, you're all set.
The only problem is you need to get at the JSON data somehow, which is where JSON-P callbacks come into the picture. If the foreign resource supports JSON-P, it will give you something that looks like
your_callback( { // JSON data } );
You then specify your code in the callback.
See JSONP for more.
If JSONP isn't an option, then the best bet is to probably fetch the data server-side, say with a cron job every few minutes, and store it locally on your own site.
You can use a server-side XMLHTTP request to grab your content from the other server. You can then parse it on you server (A.K.A screen-scraping) and serve-up the portion you want along with your web page.
If the content from the other website is just an HTML doc that you want to display on your site, you could also use an iframe to pull it in. You won't have access to any of its content because of browser security rules.
You will likely have to "scrape" the data you need and store it on your server.
This is a great tutorial on how to cache data from an external site. It is actually written to fetch and store XML, so it'll need some modification. Also, if your site doesn't allow file_get_contents then you may have to modify it to use cUrl.