I'm trying to get content of a page through curl call. But I'm getting only an empty array. Here is my code
print_r(get_data('https://www.realestate.com.au/sold/in-9%2f20%3b/list-1'));
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
// the url to fetch
curl_setopt($ch, CURLOPT_URL, $url);
// return result as a string rather than direct output
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
// set max time of cURL execution
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I have also tried different method like file_get_contents function but always getting empty page.
given function is to complicated I think its a copy paste function, as I can see from
return $result; } which is not needed and you should get syntax error.
probably that code will work without that bracket.
I cleared and shortened code, and giving a working and (tested) solution.
NOTE: You can add the code I removed back
$url 'your url';
//Usaqe of function I recomend to use parse_url();
echo get_data($url);
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Related
I am creating a Wordpress Plugin that would create a meeting after some transaction but I don't know how to get the token. I made some tests but doesn't seem to work.
Tried doing some post CURL.
$consumerKey = "consumer_key_here";
$url = "https://api.getgo.com/oauth/v2/authorize?client_id={$consumerKey}&response_type=code";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/32.0.1700.107 Chrome/32.0.1700.107 Safari/537.36');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
$test = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch);
}
print $test;
This should be redirected to the desired redirect url with a response code along the parameter.
To encode these values, open an encoding site and paste in the consumerKey, add a colon (:), and then paste in the consumerSecret. No spaces, no quotes, no brackets. Submit the values and an encoded value is returned that will look something like: ZXhhbXBsZV9jbGllbnRfaWQ6ZXhhbXBsZV9jbGllbnRfc2VjcmV0==.
$ch = curl_init();
if($ch) {
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$token="for this you need to use documentation ";
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-www-form-urlencoded','Authorization: Basic '.$token,'Accept:application/json'));
$curlulr="https://api.getgo.com/oauth/v2/token";
curl_setopt($ch, CURLOPT_URL, $curlulr);
curl_setopt($ch, CURLOPT_POST, true);
$data = 'grant_type=authorization_code'.
'&code={responseKey}';
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$rmv_results = curl_exec($ch);
if($rmv_results === false) {
echo('Error performing rmv lookup: '.curl_errno($ch).' - '.curl_error($ch).' (Message ID: '.basename(__FILE__).'-'.__LINE__.')');
} // end if errors
// echo "3";
curl_close($ch);
print_r($rmv_results);
echo ">><br>". $rmv_results;
}
I'm trying to download with curl from this url:
http://utorrent.en.softonic.com/download
When I select the alternative download link with my php script, I get this url:
http://utorrent.en.softonic.com/download-tracker?th=1%2F6CH9aeXedl4L8u%2BBHNJXWTW%2BLP1LFlnGQpxqjlxAMoVKXinh0rsXkMa8DN%2Bu6VBtArq5kfHl7Y2RJqLz%2FQUFTYOfbBsY4c9edz4d0uBydqkfciVN7cTFTjgf%2BVtTnzLV%2F4y3COCfojCKefPsPrRevbDp7d8SrrFDue95ZjhblKzCHHqpNk5gpQMihnZewg%2Fyc95fQne0DfkIfh%2BifbqX%2FCBkTDiekrGjfGamf9P%2BEll7oSfXnb9SLZg9H76ancy%2BinY3u8GE7HAJTsmqcaD4WPGon5rZYcnS8aljq7CPWLxysaKFOBu0VTr99hDNv%2Fd6rsepVexoymYrP2Hg6pe3strNwvWWePZXBsmekvAGt76yLP8R3mo5nqoBKu497L%2Fz4yQ%2BQCySXk9eqDKSE0MjByPorUPuqXZ9EMmi1KtSqX6HFvg6moZrG9Eg46eNZAY%2F9jhm6NhCV51pF6kWsnBtg4UfBJLKpQFLElRSt5vYfOQ7GE44TTPD7DpmF4XAZioF0ceMqr%2BEzZ6b3N1LY1Pwjsg7EYiTIyUmRtzKsI8WS7HB51H3bJqzF2HhUdVYVJt5U33G9VF0%2FwykMd7eJ%2FY5p6Ia6MmkpoWbe29UB3TzbVPUc%2F0t4wc8R2qkZK7Cqx5vvm3rCPViTpOQiq4ItJNTkM7DYamDyIpice8VB%2Fm7SBM9SOboRDewuQTNEDrIw3yGjfakXSGvFTeAS6j35ikg%3D%3D
but curl is downloading empty files. At the beginning I thought it was a bad config but I've tryed the url in the browser and it doesn't work, so finally I've figured they are using some kind of codification for the url but I have no idea which one because I'm very stuck in this domain. I've been checking the traffic with httpfox and there is a 302 redirect, curl is configured to follow redirects and after reading a lot of posts about curl config here, I think everything is ok on this side.
$ch = curl_init();
$output_filename = "test.exe";
curl_setopt($ch, CURLOPT_URL, $downloadUrl);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, false);
curl_setopt($ch, CURLOPT_REFERER, $previousUrl);
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$result = curl_exec($ch);
$dl=curl_error($ch);
curl_close($ch);
$fp = fopen($output_filename, 'w');
fwrite($fp, $result);
fclose($fp);
die('download complete');
So my question is does anybody know what kind of codification is using this url string and how can I decode it?
Tested and working:
<?php
$url = 'http://utorrent.en.softonic.com/download';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIEFILE, '');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2450.0 Iron/46.0.2450.0 Safari/537.36');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$result = curl_exec($ch);
libxml_use_internal_errors(true);
$doc = new DOMDocument();
$doc->loadHtml($result);
$xpath = new DOMXPath($doc);
$link = $xpath->query('//*[#id="download-button"]');
if ($link->length == 0) {
die('Failed to find download link on page');
}
$url = $link->item(0)->attributes->getNamedItem('href')->nodeValue;
curl_setopt($ch, CURLOPT_URL, $url);
$result = curl_exec($ch);
// TODO: check HTTP response code and result for errors
file_put_contents('/tmp/utorrent.exe', $result);
When running on my localhost or my server I can’t load a particular external a page. However, if I load the page in my browser it loads or using Postman it loads fine.
How can I fix this and how is Spotify preventing this?
The URL of the content I want to load is this.
<?php
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
$page = file_get_contents($url);
echo $page; //returns nothing
$page = get_data($url);
echo $page; //returns nothing with a http code of 0
$url="https://www.google.com";
$page = file_get_contents($url);
echo $page; //returns google
$page = get_data($url);
echo $page; //returns google with a http code of 200
/* gets the data from a URL */
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return $data;
}
?>
Try setting CURLOPT_POSTFIELDS to true & set the URL parameters in CURLOPT_POSTFIELDS like this. Note the URL changes as a result since the parameters are now in CURLOPT_POSTFIELDS. I set the params as an array called $post_fields since I find that to be an easier way to read that way when debugging.
UPDATE: The post params didn’t work. But adding CURLOPT_SSL_VERIFYHOST set to false as well as CURLOPT_SSL_VERIFYPEER set to false seems to do the trick on my side.
Here is my cleaned up version of your code. Removed your tests & commented out the post param stuff I thought would help before:
// Set the `url`.
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
// Set the `post` fields.
$post_fields = array();
// $post_fields['uri'] = 'spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI';
// Set the `post` fields.
$page = get_data($url, $post_fields);
// Echo the output.
echo $page;
// Gets the data from a `url`.
function get_data($url, $post_fields) {
$curl_timeout = 5;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
// curl_setopt($ch, CURLOPT_POST, true);
// curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $curl_timeout);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
// echo curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $data;
}
Im trying to fetch content of a website about a tourament. I want to display the results on a temporary page.
I'm tying to fetch this page:
http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1
I want to return the content of this page, and afterwards fetch the specific table with the fixtures.
The script i'm using returns a 404 Not found error, while the url is present.
My script:
function nxs_cURLTest($url, $msg, $testText){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSLVERSION,3); // Apparently 2 or 3
$response = curl_exec($ch);
$errmsg = curl_error($ch);
$cInfo = curl_getinfo($ch);
curl_close($ch);
echo "Testing ... ".$url." - ".$cInfo['url']."<br />";
if (stripos($response, $testText)!==false)
echo "....".$msg." - OK<br />";
else
{
echo "....<b style='color:red;'>".$msg." - Problem</b><br /><pre>";
print_r($errmsg);
print_r($cInfo);
print_r(htmlentities($response));
echo "</pre>There is a problem with cURL. You need to contact your server admin or hosting provider.";
}
}
nxs_cURLTest("http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1", "HTTPS to Toernooi.nl", 'link rel="canonical" href="http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1"');
Can anyone help me on this one?
I used the following on xampp win7 with chrome and it returned data
$url = 'http://www.tournamentsoftware.com/sport/draw.aspx?id=600CA297-99CA-4420-AE1A-698BA10C39B0&draw=1';
echo curl_scrap($url);
function curl_scrap($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
It did not return the same source as just using chrome to hit url, as function doesn't set anything like user agent field etc, but returned the data you want, just a quick and dirty test.
Alright I am still learning my functions in php but this particular piece of code has me stuck. Credit goes to http://www.barattalo.it/2010/08/29/php-bot-to-get-wikipedia-definitions/ for the wikipedia query segment. I thought this looked interesting and am trying to use the code provided to echo data from the function. This is what I have:
<?php
function wikidefinition($s) {
$url = "http://wikipedia.org/w/api.php?action=opensearch&search=".urlencode($s)."&format=xml&limit=1";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_POST, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_NOBODY, FALSE);
curl_setopt($ch, CURLOPT_VERBOSE, FALSE);
curl_setopt($ch, CURLOPT_REFERER, "");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_MAXREDIRS, 4);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 6.1; he; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8");
$page = curl_exec($ch);
$xml = simplexml_load_string($page);
if((string)$xml->Section->Item->Description) {
return array((string)$xml->Section->Item->Text, (string)$xml->Section->Item->Description, (string)$xml->Section->Item->Url);
} else {
return "blank";
}
}
$define = wikidefinition("test");
echo $define;
?>
however this simply echos "array"
I know the code is getting to the if/else statement because if you change the input in "$define = wikidefinition("test");" to some random key combination such as "qoigenqnge" it will echo "blank" I cannot figure out why it is only echoing "array" instead of the data inside the array. Probably something stupid, but I have been reading up on arrays for a while and cant find any info. Any help will be great thanks!
Have you tried:
print_r($define);
instead of echo ?