CURL not working with link in PHP, but in command line it does - php

I've got a problem with my PHP script using CURL. I want to download file from another server (Probably with hotlink protection). The URL is like this: example.com/script.extension?id=12345&type=xxx&another=qwerty
When i hit that link from my website, It sends 'Referer:' in headers, so the download is not starting. (without the header it works)
I have to modify the file's name, so i've used CURL to download that to my server. When I'm doing that with that script, the file is created, but it is empty:
function my_function($the_url) {
$fp = fopen ('tmp_file.extension', 'w+');
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_VERBOSE => 1,
CURLOPT_URL => $the_url,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POST => false,
CURLOPT_USERAGENT => $user_agent,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_AUTOREFERER => false,
CURLOPT_REFERER => "http://example.com/",
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init();
curl_setopt_array($ch, $options);
$my_exec = curl_exec($ch);
fwrite($fp, $my_exec);
curl_close($ch);
fclose($fp);
}
But it works well with file which is hosted on my site (eg. mysite.com/file.zip)
Also, when I run command through the shell:
$ curl -L -o file.tmp --referer http://example.com/ full url here
It downloaded the file properly.
What's the reason I cannot do it with my PHP function?

Related

php script login to td ameritrade

I need to log into the TD Ameritrade website with my account. Note, I am not talking about the official API, I'm talking about their actual UI. I need to scrape from the site itself, unfortunately. I'm using curl in PHP, but any language you can get to work is probably fine. I've started with postman, which I'm usually able to get to work for this kind of thing, but it isn't returning anything when I copy the transactions from Chrome Dev Tools and importing into Postman. For example, it either loops back to the login site, or if I play with the settings it will flat out not return anything. The current issue with my php is that I get Maximum (10) redirects followed, it's an infinite loop where the login forwards back to itself. Unfortunately, for obvious reasons, I cannot post my username and password, so you would need an account to test. They are free, but not the easiest to set up. In any case, if you have any ideas that would be greatly appreciated.
<?php
header('Content-type: text/javascript');
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'https://auth.tdameritrade.com/oauth?client_id=MOBI%40AMER.OAUTHAP&response_type=code&redirect_uri=https%3A%2F%2Finvest.ameritrade.com%2Fgrid%2Fp%2Fsite&code_challenge=xNghkUhiiE_DSuVXLcSF1ufBVSPTQRa1ITybIyw2USc&code_challenge_method=S256',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'POST',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_HEADER => 1, // needed for getting cookies
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1",
CURLOPT_POSTFIELDS => 'su_username=' . $username . '&su_password=' . $password,
CURLOPT_VERBOSE => true,
CURLOPT_POST => true,
CURLOPT_AUTOREFERER => true,
// CURLOPT_HTTPHEADER => array(
// 'Content-Type: application/x-www-form-urlencoded'
// ),
));
//curl_setopt($curl, CURLOPT_USERPWD, $username . ":" . $password);
// $cookie = dirname(__FILE__) . "/cookie.txt";
// curl_setopt($curl, CURLOPT_COOKIEFILE, $cookie);
// curl_setopt($curl, CURLOPT_COOKIEJAR, $cookie);
$res = (object)array(
'response' => curl_exec($curl),
'info' => (object)curl_getinfo($curl),
'errors' => curl_error($curl)
);
print_r($res);

Download Whatsapp Business Media Image API

I am Using Whatsapp Media Api to get Image send from user to business number using webhooks and i also get the Image ID and url and Image as well in Postman but when I use that curl request made from Postman it alwasy shows facebook error:
Sorry, something went wrong.
We're working on it and we'll get it fixed as soon as we can.
But its working fine in Postman what am i doing wrong?
<?php
$ua = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \
Chrome/24.0.1304.0 Safari/537.16';
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => 'Image_URL_from_Api',
CURLOPT_USERAGENT => $ua,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_HTTPHEADER => array(
'Authorization: Bearer My_Token'
),
));
$response = curl_exec($curl);
curl_close($curl);
echo $response;
OK, I've got the answer I needed to give the user_agent value which is in postman it is automatically putting CURLOPT_USERAGENT => 'PostmanRuntime/7.26.10' in the header but for curl, I've added CURLOPT_USERAGENT => 'curl/7.64.1', and it worked.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent

curl request works from one server and give 403 on another

I am using curl to hit a url and get contents from it and below is the code
$url = 'http://uber.com';
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.16 (KHTML, like Gecko) \ Chrome/24.0.1304.0 Safari/537.16',
CURLOPT_CONNECTTIMEOUT => 180,
CURLOPT_TIMEOUT => 180 + 60,
CURLOPT_COOKIE => 'lang=en;'
));
$response = curl_exec($ch);
var_dump($response);
It gives me the content when I execute the script on local server but while executing this on staging and production server, situated in US, it give 403-forbidden error. I also tested the same from another server in US and it works fine.
Any help is appreciated.

PHP cURL login fails

There is a website http://www.tremorgames.com and I want to do a cURL login. I have a code which I use on my server http://shalva97.x10.mx/tremorlogin/ but it just does not work, but after a lot of testing I decided to install Apache on my PC and tested the exactly same code and it worked.
Here is the code :
<?php
$url="http://www.tremorgames.com/index.php";
$postdata = "loginuser=shalva&loginpassword=shalva&Submit=&saveme=1";
$ch = curl_init();
$f = fopen('request.txt', 'w');
curl_setopt_array($ch, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_VERBOSE => 1,
CURLOPT_STDERR => $f,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0',
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $postdata,
CURLOPT_HEADER => 1,
CURLOPT_HTTPHEADER => array("Accept-Language: en-US;q=0.6,en;q=0.4", "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Connection: keep-alive"),
CURLOPT_COOKIEFILE => "cookies.txt", //Save cookies
CURLOPT_COOKIEJAR => "cookies.txt", //Cookies located
CURLOPT_REFERER => 'http://www.tremorgames.com/index.php',
CURLOPT_ENCODING => 'gzip,deflate'
));
echo curl_exec($ch);
fclose($f);
curl_close($ch);
?>
Executing this same code on both my PC and server, I get different headers, on PC it receives multiple "Set-Cookie" header, but on server it is only one.
Why it does not work on the server? Why it works on my PC?
well after a lot of thinking i found out that it is not only headers is sent but the ip and my location. The site does not allow logins from other countries thats why it didnot loged in, the code is correct and it was all about server location. i hope this will help others who also try to do same thing.

CURLOPT_TIMEOUT not working at all (php)

I am using curl and setting all the parameters correctly (as far as I know) but CURLOPT_TIMEOUT is being ignored and allowing for an infinite loop. Here is the configuration for my Curl Request:
$user_agent = 'Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_CUSTOMREQUEST => "GET", //set request type post or get
CURLOPT_POST => false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE => dirname(__FILE__)."/cookie.txt", //set cookie file
CURLOPT_COOKIEJAR => dirname(__FILE__)."/cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_SSL_VERIFYPEER, FALSE, //ignore ssl
CURLOPT_PROXY =>$proxy['ip'],
CURLOPT_PROXYPORT =>$proxy['port'],
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referrer on redirect
CURLOPT_CONNECTTIMEOUT => 20, // timeout on connect
CURLOPT_TIMEOUT => 10, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
I am not the best at debugging so I'm not sure what the problem could be. Please help me.
The curl_setopt($connection, CURLOPT_TIMEOUT, $seconds) should be called just right before curl_exec() function.
It wasn't working for me when I called it sooner
If your are using CloudFlare note that :
Enterprise customers can increase the 524 timeout up to 6000 seconds
using the proxy_read_timeout API endpoint. If you regularly run HTTP
requests that take over 100 seconds to complete (for example large
data exports), move those processes behind a subdomain not proxied
(grey clouded) in the Cloudflare DNS app.

Categories