Getting json data from a webpage using PHP - php

Question

I am trying to fetch a response from here (example url), and first, I thought I should use file_get_contents()
When I tried this, I got the following error:
Warning: file_get_contents(https://steamcommunity.com/market/pricehistory/?country=US&currency=1&appid=730&market_hash_name=SG%20553%20|%20Damascus%20Steel%20(Factory%20New)): failed to open stream: HTTP request failed! HTTP/1.0 400 Bad Request
I know this is because it is converting & to &. I have tried numerous ways to counter this, however they have all failed and after a quick google I came to the conclusion that file_get_contents() converts & to & automatically.
My next step was to try curl. I tried the below code first:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://steamcommunity.com/market/pricehistory/?country=US&currency=1&appid=730&market_hash_name='.$hash,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) ChromePlus/4.0.222.3 Chrome/4.0.222.3 Safari/532.2'
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
But this returned ‹ŠŽÿÿ)»L as the response. I wondered if this was to do with json encoding, so I tried putting it through json_decode() but it didn't work.
Next, I tried:
// Get cURL resource
$curl = curl_init();
// Set some options - we are passing in a useragent too here
curl_setopt_array($curl, array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_URL => 'http://steamcommunity.com/market/pricehistory/',
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) ChromePlus/4.0.222.3 Chrome/4.0.222.3 Safari/532.2',
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => array(
country => "US",
currency => 1,
appid => 730,
market_hash_name => "SG%20553%20|%20Damascus%20Steel%20(Factory%20New)"
)
));
// Send the request & save response to $resp
$resp = curl_exec($curl);
// Close request to clear up some resources
curl_close($curl);
But again got the response ‹ŠŽÿÿ)»L.
What does this response mean, and can I parse it? If not, how should I correctly fetch this data? Furthermore, why didn't file_get_contents() work?

Answer 1

I'm pretty sure this is happening because you need some type of access token to access the steam web API.
See this answer on SO.
Essentially, Steam is returning an error with the "400 Bad Request" status. This error can be ignored, however, by doing this:
<?php
$url = "https://steamcommunity.com/market/pricehistory/?country=US&currency=1&appid=730&market_hash_name=SG%20553%20%7C%20Damascus%20Steel%20(Factory%20New)";
$context = stream_context_create(array(
'http' => array(
'ignore_errors'=>true,
'method'=>'GET'
// for more options check http://www.php.net/manual/en/context.http.php
)
));
$response = file_get_contents($url, false, $context);
echo $response; // returns "[]"
?>
Make sure you take a look at this answer on SO.

Answer 2

May be your response is gzip, try to use CURLOPT_ENCODING.
curl_setopt($curl ,CURLOPT_ENCODING, '')
If you use https don't forget to disable CURLOPT_SSL_VERIFYPEER.
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false)
One thing, if I follow your link with my browser and open my debug console.
I see you request have a 400 Status code (Bad Request).

Answer 3

I cant say about your enpoint, but you can get around your Bad Request error by using urlencode():
$url = urlencode('https://steamcommunity.com/market/pricehistory/?country=US&currency=1&appid=730&market_hash_name=SG%20553%20%7C%20Damascus%20Steel%20(Factory%20New))'
file_get_contencts($url);