I'm trying to get all the urls of the shows that are listed https://api.tvmaze.com/schedule?country=US&date=2021-03-24 here but im getting just a blank page and I don't have any ideea why.
function request($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 3);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$api_url = 'https://api.tvmaze.com/schedule?country=US&date=2021-03-23';
$data = request($api_url);
$final_data = json_decode($data, TRUE);
for ($x = 0; $x < count($final_data); $x++)
{
echo $finaldata[$x]['show']['url']."\n";
}
Related
So I found a solution to a problem I was having, (how to determine a domain protocol) How to find the domain is whether HTTP or HTTPS (with or without WWW) using PHP?
Below are two versions of my code;
The first doesn't work as expected it only echoes out my domains.
<?php
$url_list = file('urls.txt');
foreach($url_list as $url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_exec($ch);
$real_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
echo $real_url;
}
?>
The Second version of my code gives me an error of how I am supplied my foreach statement...
<?php
$fn = fopen("urls.txt","r");
while(! feof($fn)) {
$url_list = fgets($fn);
foreach($url_list as $url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_exec($ch);
$real_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
echo $real_url;
#echo $result;
fclose($fn);
}
}
?>
What can I be doing wrong??
Expected results is this, but when reading the domains from a file;
code: reference --> How to find the domain is whether HTTP or HTTPS (with or without WWW) using PHP?
<?php
$url_list = ['facebook.com','google.com'];
foreach($url_list as $url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_exec($ch);
$real_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
echo $real_url;//add here your db commands
}
?>
Output:
test#linux: php domain-fuzzer.php
https://www.facebook.com/http://www.google.com/#
I found the solution, just incase anyone would find such an issue here is the code:
<?php
$fn = file_get_contents("urls.txt");
$url_list = explode(PHP_EOL, $fn);
foreach($url_list as $url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_exec($ch);
$real_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
echo nl2br("$real_url \n");
}
?>
By default fgets returns a single current line from the file not an array of all lines of the file.
So, your code will be:
<?php
$fn = fopen("urls.txt","r");
while(! feof($fn)) {
$url = fgets($fn);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_exec($ch);
$real_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
echo $real_url;
#echo $result;
fclose($fn);
}
?>
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 days ago.
Improve this question
I have been using the code below to log into my Amazon account for several years and it has suddenly stopped working.
It throws me back into the login page.
Can someone tell me what the problem is and how I can fix it?
<?php
$email = 'email#hotmail.com';
$password = 'password';
$URL = 'https://www.amazon.com/ap/signin?_encoding=UTF8&openid.assoc_handle=usflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fyourstore%2Fhome%3Fie%3DUTF8%26action%3Dsign-out%26path%3D%252Fgp%252Fyourstore%252Fhome%26ref_%3Dnav_youraccount_signout%26signIn%3D1%26useRedirectOnSuccess%3D1';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/amazoncookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/amazoncookie.txt');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$page = curl_exec($ch);
if (!preg_match('/<form name="signIn".*?<\/form>/is', $page, $form)) {
die('Failed to find log in form!');
}
$form = $form[0];
if (!preg_match('/action="([^"]+)"/i', $form, $action)) {
die('Failed to find login form url');
}
$URL2 = $action[1];
$count = preg_match_all('/<input type="hidden"\s*name="([^"]*)"\s*value="([^"]*)"/i', $form, $hiddenFields);
for ($i = 0; $i < $count; ++$i) {
$postFields[$hiddenFields[1][$i]] = $hiddenFields[2][$i];
}
$postFields['email'] = $email;
$postFields['create'] = 0;
$postFields['password'] = $password;
$post = '';
foreach($postFields as $key => $value) {
$post .= $key . '=' . urlencode($value) . '&';
}
$post = substr($post, 0, -1);
curl_setopt($ch, CURLOPT_URL, $URL2);
curl_setopt($ch, CURLOPT_REFERER, $URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
$page = curl_exec($ch);
?>
There is a known PHP bug related to CURLOPT_FOLLOWLOCATION.
Looking at the interaction of CURL elements it could be that your code has not changed in years, but PHP has.
ive tested your code to singup amazon and it's work good right.
her your code a little modified :
<?php
function getRandomUserAgent()
{
$userAgents=array(
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)",
"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)",
"Opera/9.20 (Windows NT 6.0; U; en)",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; en) Opera 8.50",
"Mozilla/4.0 (compatible; MSIE 6.0; MSIE 5.5; Windows NT 5.1) Opera 7.02 [en]",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; fr; rv:1.7) Gecko/20040624 Firefox/0.9",
"Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/48 (like Gecko) Safari/48"
);
$random = rand(0,count($userAgents)-1);
return $userAgents[$random];
}
//============================================================================
$randIP = "".mt_rand(0,255).".".mt_rand(0,255).".".mt_rand(0,255).".".mt_rand(0,255);
$email = 'tictac#gmail.com';
$password = 'pass1234325';
$customerName = 'body';
$passwordCheck = 'pass1234325';
// initial login page which redirects to correct sign in page, sets some cookies
$URL = 'https://www.amazon.com/ap/register?openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2Fgp%2Fyourstore%2Fhome%3Fie%3DUTF8%26ref_%3Dnav_ya_signin&prevRID=QQTDRH3T7WP058F3RJBC&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.assoc_handle=usflex&openid.mode=checkid_setup&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&prepopulatedLoginId=&failedSignInCount=0&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&pageId=usflex&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $URL);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'amazoncookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'amazoncookie.txt');
curl_setopt($ch, CURLOPT_USERAGENT,getRandomUserAgent());
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("REMOTE_ADDR: ".$randIP."", "HTTP_X_FORWARDED_FOR: ".$randIP.""));
//curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, fopen('php://stdout', 'w'));
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$page = curl_exec($ch);
//var_dump($page);exit;
// try to find the actual login form
if (!preg_match('/<form id="ap_register_form" name="register".*?<\/form>/is', $page, $form)) {
die('Failed to find log in form!');
}
$form = $form[0];
// find the action of the login form
if (!preg_match('/action=(?:\'|")?([^\s\'">]+)/i', $form, $action)) {
die('Failed to find login form url');
}
$URL2 = $action[1]; // this is our new post url
// find all hidden fields which we need to send with our login, this includes security tokens
$count = preg_match_all('/<input type="hidden"\s*name="([^"]*)"\s*value="([^"]*)"/i', $form, $hiddenFields);
$postFields = array();
// turn the hidden fields into an array
for ($i = 0; $i < $count; ++$i) {
$postFields[$hiddenFields[1][$i]] = $hiddenFields[2][$i];
}
// add our login values
$postFields['email'] = $email;
$postFields['password'] = $password;
$postFields['customerName'] = $customerName;
$postFields['passwordCheck'] = $passwordCheck;
$post = '';
// convert to string, this won't work as an array, form will not accept multipart/form-data, only application/x-www-form-urlencoded
foreach($postFields as $key => $value) {
$post .= $key . '=' . urlencode($value) . '&';
}
$post = substr($post, 0, -1);
// set additional curl options using our previous options
curl_setopt($ch, CURLOPT_URL, $URL2);
curl_setopt($ch, CURLOPT_REFERER, $URL);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("REMOTE_ADDR: ".$randIP."", "HTTP_X_FORWARDED_FOR: ".$randIP.""));
$page = curl_exec($ch); // make request
var_dump($page); // should be logged in
?>
When running on my localhost or my server I can’t load a particular external a page. However, if I load the page in my browser it loads or using Postman it loads fine.
How can I fix this and how is Spotify preventing this?
The URL of the content I want to load is this.
<?php
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
$page = file_get_contents($url);
echo $page; //returns nothing
$page = get_data($url);
echo $page; //returns nothing with a http code of 0
$url="https://www.google.com";
$page = file_get_contents($url);
echo $page; //returns google
$page = get_data($url);
echo $page; //returns google with a http code of 200
/* gets the data from a URL */
function get_data($url) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
echo curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return $data;
}
?>
Try setting CURLOPT_POSTFIELDS to true & set the URL parameters in CURLOPT_POSTFIELDS like this. Note the URL changes as a result since the parameters are now in CURLOPT_POSTFIELDS. I set the params as an array called $post_fields since I find that to be an easier way to read that way when debugging.
UPDATE: The post params didn’t work. But adding CURLOPT_SSL_VERIFYHOST set to false as well as CURLOPT_SSL_VERIFYPEER set to false seems to do the trick on my side.
Here is my cleaned up version of your code. Removed your tests & commented out the post param stuff I thought would help before:
// Set the `url`.
$url="https://embed.spotify.com/?uri=spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI";
// Set the `post` fields.
$post_fields = array();
// $post_fields['uri'] = 'spotify:user:spotify:playlist:4hOKQuZbraPDIfaGbM3lKI';
// Set the `post` fields.
$page = get_data($url, $post_fields);
// Echo the output.
echo $page;
// Gets the data from a `url`.
function get_data($url, $post_fields) {
$curl_timeout = 5;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
// curl_setopt($ch, CURLOPT_POST, true);
// curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $curl_timeout);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
// echo curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $data;
}
I am scraping content from a url that is hosted in the UK using curl. When i view the site in my browser from the US it shows the product pricing in dollars but when i use curl to retrieve content it returns in Euros. I need it to return in US dollars as if you were viewing it from a browser in the US. Below is the code I am using
function LoadCURLPage($url, $agent = "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-us; rv:1.4)
Gecko/20030624 Netscape/7.1 (ax)",
$cookie = '', $referer = '', $post_fields = '', $return_transfer = 1,
$follow_location = 1, $ssl = '', $curlopt_header = 1)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
if($ssl)
{
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
}
curl_setopt ($ch, CURLOPT_HEADER, $curlopt_header);
curl_setopt ($ch, CURLOPT_HTTPHEADER,array('User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16', 'Accept-language: en-us,en;q=0.7,bn;q=0.3', 'Accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'));
if($agent)
{
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
}
if($post_fields)
{
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
}
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
if($referer)
{
curl_setopt($ch, CURLOPT_REFERER, $referer);
}
if($cookie)
{
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
}
$result = curl_exec ($ch);
curl_close ($ch);
return $result;
}
// the url
$url = "http://us.asos.com/Adidas-Honey-Silver-Mid-Sneakers/ysrqb/?iid=2212284";
//the function
echo LoadCURLPage($url);
It's in a cookie. So either visit the page that sets that cookie, or edit your cookiejar file.
Does anyone know how to download a file using curl and PHP and supports resume. and link to a tutorial or code demonstration will be good. I'm using this code now but it doesn't support resume, and i don't know how to handle headers. help will be appreciated
$start_range = 0;
$url = URL to remote file
$end_range = $filesize;
while($start_range <= $end_range) {
if(($start_range + 9999999) > $end_range) $range = $start_range.'-';
else $range = $start_range.'-'.($start_range + 9999999);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_REFERER, $refurl);
curl_setopt($ch, CURLOPT_RANGE,$range);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1');
curl_exec($ch);
curl_close($ch);
$start_range +=10000000;
flush();
ob_flush()
}