When running on my localhost or my server I can’t load a particular external a page. However, if I load the page in my browser it loads or using Postman it loads fine.
How can I fix this and how is Spotify preventing this?
The URL of the content I want to load is this.
<?php
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
$page = file_get_contents($url);
echo $page; //returns nothing
$page = get_data($url);
echo $page; //returns nothing with a http code of 0
$url="https://www.google.com";
$page = file_get_contents($url);
echo $page; //returns google
$page = get_data($url);
echo $page; //returns google with a http code of 200
/* gets the data from a URL */
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return $data;
}
?>
Try setting CURLOPT_POSTFIELDS to true & set the URL parameters in CURLOPT_POSTFIELDS like this. Note the URL changes as a result since the parameters are now in CURLOPT_POSTFIELDS. I set the params as an array called $post_fields since I find that to be an easier way to read that way when debugging.
UPDATE: The post params didn’t work. But adding CURLOPT_SSL_VERIFYHOST set to false as well as CURLOPT_SSL_VERIFYPEER set to false seems to do the trick on my side.
Here is my cleaned up version of your code. Removed your tests & commented out the post param stuff I thought would help before:
// Set the `url`.
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
// Set the `post` fields.
$post_fields = array();
// $post_fields['uri'] = 'spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI';
// Set the `post` fields.
$page = get_data($url, $post_fields);
// Echo the output.
echo $page;
// Gets the data from a `url`.
function get_data($url, $post_fields) {
$curl_timeout = 5;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
// curl_setopt($ch, CURLOPT_POST, true);
// curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $curl_timeout);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
// echo curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $data;
}
Related
I'm trying to get content of a page through curl call. But I'm getting only an empty array. Here is my code
print_r(get_data('https://www.realestate.com.au/sold/in-9%2f20%3b/list-1'));
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
// the url to fetch
curl_setopt($ch, CURLOPT_URL, $url);
// return result as a string rather than direct output
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
// set max time of cURL execution
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I have also tried different method like file_get_contents function but always getting empty page.
given function is to complicated I think its a copy paste function, as I can see from
return $result; } which is not needed and you should get syntax error.
probably that code will work without that bracket.
I cleared and shortened code, and giving a working and (tested) solution.
NOTE: You can add the code I removed back
$url 'your url';
//Usaqe of function I recomend to use parse_url();
echo get_data($url);
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I want to open a JSON from url in php/laravel file. this is my code :
{{ini_set("allow_url_fopen", 1)}}
{{$id_ = $blog_post->featured_media}}
{{$url_ = 'http://example.net/blog/wp-json/wp/v2/media/'.$id_}}
{{$data = #file_get_contents($url_)}}
{{$json = #json_decode($data, true)}}
{{var_dump(#$json)}}
when i try reload page i get this error :
something went wrong
how can i read JSON from url ?
Use cURL to get the json data. Like this
$url = 'www.yoururl.com/full-url';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_HTTPHEADER, Array("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.15) Gecko/20080623 Firefox/2.0.0.15") );
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$result= curl_exec ($ch);
curl_close ($ch);
$info = json_decode($result, true);
print_r($info); // print all data
I am trying to find where I'll be redirected at. So I tried to functions for this, but none of those are working properly.
the links is here. when you try to enter, you will be redirected:
https://lions-mansion.jp/MA141070/
so I tried use cURL,
function redirect1($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$data = curl_exec($ch);
$data = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL );
curl_close($ch);
return $data;
}
and also this:
function redirect($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
if (preg_match('~Location: (.*)~i', $result, $match)) {
$location = trim($match[1]);
}
return $result;
}
But I couldn't find the redirected url.
this page does not use a redirect-scheme that libcurl understands (it uses a html <meta http-equiv="REFRESH"-redirect, unsupported by libcurl), so libcurl can neither tell you where it is being redirected, nor can libcurl auto-follow the redirect (because libcurl does not understand it)
you need to parse out the redirect url yourself from the HTML, eg
function redirect1($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$data = curl_exec($ch);
$domd=#DOMDocument::loadHTML($data);
$xp=new DOMXPath($domd);
// <META http-equiv="REFRESH" content="0;URL=http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo" />
$location=$xp->query("//meta[#http-equiv='REFRESH']")->item(0)->getAttribute("content");
// 0;URL=http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo
$location=substr($location,stripos($location,'URL=')+4);
curl_close($ch);
return $location;
}
var_dump(redirect1('https://lions-mansion.jp/MA141070/'));
output:
C:\projects\misc>php re.php
string(57) "http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo"
If keep you CURLOPT_RETURNTRANSFER to true, after executing the CURL command you can use this function call to get the redirect of effective URL:
$finalUrl = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
Interestingly, I have not been able to find a working example of this. Using php, I'm trying to scrape/re display all the images of a given url, onto another website. I know how to do this with text, but images, I'm not sure. Anyone know of a good working example? I get how to grab all contents, but not specifically just the images. ex this does the whole page:
<?php
$curl = curl_init();
curl_setopt ($curl, CURLOPT_URL,
"https://en.wikipedia.org/wiki/Wikipedia:Picture_of_the_day");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec ($curl);
curl_close ($curl);
echo $result;
?>
Thanks much. -Wilson
*ideally actually this would just grab the first image, such as in the example above. But I won't get ahead of myself, just trying to get this function down.
You may use a file to save the result.
$fp = fopen($filename, 'a+');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:29.0) Gecko/20100101 Firefox/29.0');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip, deflate');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_NOPROGRESS, false);
curl_setopt($ch, CURLOPT_PROGRESSFUNCTION, function ($dltotal, $dlnow, $ultotal, $ulnow) {
});
curl_setopt($ch, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_setopt($ch, CURLOPT_LOW_SPEED_TIME, 8);
curl_exec($ch);
$error = curl_error($ch);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$content_type = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
$end_size = $begin_size + curl_getinfo($ch, CURLINFO_SIZE_DOWNLOAD);
Log::info('end_size='.$end_size);
curl_close($ch);
fclose($fp);
You can add /use PHP DOMXPath function to parsing your scrape result.
add this following script right after your code
$dom = new DOMDocument();
#$dom->loadHTML($result);
$xpath = new DOMXPath($dom);
//get all images
$images = $xpath->query ('//img/#src');
$img = array();
foreach ( $images as $image) {
$img[] = $image->nodeValue;
}
print_r($img);
** Edited Part
Try to change your CURL code with this one
$url = 'https://en.wikipedia.org/wiki/Wikipedia:Picture_of_the_day';
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10');
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0); //untuk https
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);//untuk https
//curl_setopt($curl, CURLOPT_ENCODING , 'gzip');
$html = curl_exec($curl);
if(curl_error($curl)){
echo 'Curl error: ' . curl_error($curl);
$result = ''; //return empty if error
}
else {
$result = $html;
}
Im trying to fetch content of a website about a tourament. I want to display the results on a temporary page.
I'm tying to fetch this page:
http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1
I want to return the content of this page, and afterwards fetch the specific table with the fixtures.
The script i'm using returns a 404 Not found error, while the url is present.
My script:
function nxs_cURLTest($url, $msg, $testText){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSLVERSION,3); // Apparently 2 or 3
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_close($ch);
echo "Testing ... ".$url." - ".$cInfo['url']."<br />";
if (stripos($response, $testText)!==false)
echo "....".$msg." - OK<br />";
else
{
echo "....<b style='color:red;'>".$msg." - Problem</b><br /><pre>";
print_r($errmsg);
print_r($cInfo);
print_r(htmlentities($response));
echo "</pre>There is a problem with cURL. You need to contact your server admin or hosting provider.";
}
}
nxs_cURLTest("http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1", "HTTPS to Toernooi.nl", 'link rel="canonical" href="http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1"');
Can anyone help me on this one?
I used the following on xampp win7 with chrome and it returned data
$url = 'http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1';
echo curl_scrap($url);
function curl_scrap($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
It did not return the same source as just using chrome to hit url, as function doesn't set anything like user agent field etc, but returned the data you want, just a quick and dirty test.