change http ref using cURL and PHP - php

I am using cURL to change the HTTP ref for a site that only allow you to see their content if it's coming from search engins:
I was able to do that!
But the problem is: the IP address of the visitor the site get is not mine! it's the IP address of the site I'm using to change the ref !, here is the code:
echo geturl('http://example.com', 'http://referring-site.com');
function geturl($url, $referer) {
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg,text/html,application/xhtml+xml';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$useragent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)';
$process = curl_init($url);
curl_setopt($process, CURLOPT_HTTPHEADER, $headers);
curl_setopt($process, CURLOPT_HEADER, 0);
curl_setopt($process, CURLOPT_USERAGENT, $useragent);
curl_setopt($process, CURLOPT_REFERER, $referer);
curl_setopt($process, CURLOPT_TIMEOUT, 30);
curl_setopt($process, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($process, CURLOPT_FOLLOWLOCATION, 1);
$return = curl_exec($process);
curl_close($process);
return $return;
}
Let say I use that code on
mysite.com
So example.com will si referring-site.com as an HTTP ref, but it thinks that the visitors IP is the same as mysite.com !!!
Ho can I get it to get the real IP address of the visitor in stead of the site's IP I use the code on?
I tried to replace
return $return;
with
return "<?php
header( 'Location: http://example.com' ) ;
?>";
or
echo '<META HTTP-EQUIV='Refresh' Content='0; URL=http://example.com'>';
But it doesn't work!

What you are doing is proxying the request and lying about the referer. Since the request is coming from your server, it is under your control so that sort of underhand deception is possible.
There is no way for a website to induce a browser into making that sort of lie.

Related

Final Effective URL - PHP (Proxy)

I apologize in advance for my English. I have small problem.
I want to get Final Effective URL from page
streamuj.tv/video/00e276bf5841bf77c8de?streamuj=original&authorize=ac13bb77d3d863ca362315b9b4dcdf3e
When you put a link into the browser gives me to .flv file
But when I put it through PHP gives me s3.streamuj.tv/unauthorized.flv
When I try it through this: getlinkinfo.com/info?link=http%3A%2F%2Fwww.streamuj.tv%2Fvideo%2F00e276bf5841bf77c8de%3Fstreamuj%3Doriginal%26authorize%3Dac13bb77d3d863ca362315b9b4dcdf3e&x=49&y=11
So everything is fine indicates that
s4.streamuj.tv:8080/vid/d0fe77e1020b6414a16aa5316c759add/58aaf1dd/00e276bf5841bf77c8de_hd.flv?start=0
My PHP CODE:
<?php
session_start();
include "simple_html_dom.php";
$proxy = array("189.3.93.114:8080");
$proxyNum = 0;
$proxy = explode(':', $proxy[$proxyNum]);
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://www.streamuj.tv/video/00e276bf5841bf77c8de?streamuj=original&authorize=ac13bb77d3d863ca362315b9b4dcdf3e');
curl_setopt($curl, CURLOPT_FILETIME, true);
curl_setopt($curl, CURLOPT_NOBODY, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($curl, CURLOPT_PROXY, $proxy[0]);
curl_setopt($curl, CURLOPT_PROXYPORT, $proxy[1]);
$header = curl_exec($curl);
$info = curl_getinfo($curl);
curl_close($curl);
$u1 = $info['url'];
echo "u1: $u1</br>";
$u2 = str_replace("flv?start=0","flv",$u1);
echo $u2;
?>
Where is the problem? Why it makes unauthorized.flv?
Solution
Server was checking client legitimacy via user-agent HTTP header parameter.
Using custom user-agent solved the problem.
curl_setopt($curl, CURLOPT_HTTPHEADER, array( 'user-agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2950.0 Iron Safari/537.36' ));
Original post:
Most likely the generated flv URL is not pointing to static place. It
probably uses sessionID + cookie / verifies IP (one of these, or
both).
Without knowing what header you have to request with via CURL, you
probably won't get a relevant response.

Using Curl from SSL server to download xml feed?

When i try this code on some other server it works properly, but when i run it on server where is SSL "installed" i get empty string from var_dump.
$feedUrl = 'https://api.pinnaclesports.com/v1/feed?sportid=29&leagueid=1980-1977-1957-1958-1983-2421-2417-2418-2419-1842-1843-2436-2438-2196-2432-2036-2037-1928-1817-2386-2592-2081';
// Set your credentials here, format = clientid:password from your account.
$credentials = base64_encode("password");
// Build the header, the content-type can also be application/json if needed
$header[] = 'Content-length: 0';
$header[] = 'Content-type: application/xml';
$header[] = 'Authorization: Basic ' . $credentials;
// Set up a CURL channel.
$httpChannel = curl_init();
// Prime the channel
curl_setopt($httpChannel, CURLOPT_URL, $feedUrl);
curl_setopt($httpChannel, CURLOPT_RETURNTRANSFER, true);
curl_setopt($httpChannel, CURLOPT_HTTPHEADER, $header);
curl_setopt($httpChannel, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)' );
// Unless you have all the CA certificates installed in your trusted root authority, this should be left as false.
curl_setopt($httpChannel, CURLOPT_SSL_VERIFYPEER, false);
// This fetches the initial feed result. Next we will fetch the update using the fdTime value and the last URL parameter
$initialFeed = curl_exec($httpChannel);
//var_dump($initialFeed);
I already have script on this ssl server who downloads csv files from an other url and it works normally, so i think that problem is in my header, but how it works on other servers, same code?
Try this
Basically says to do:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/CAcerts/BuiltinObjectToken-EquifaxSecureCA.crt");
Or try this

PHP cURL fails to fetch images from website

I've written a small PHP script for grabbing images with curl and saving them locally.
It reads the urls for the images from my db, grabs it and saves the file to a folder.
Tested and works on a couple other websites before, fails with a new one I'm trying it with.
I did some reading around, modified the script a bit but still nothing.
Please suggest what to look out for.
$query_products = "SELECT * from product";
$products = mysql_query($query_products, $connection) or die(mysql_error());
$row_products = mysql_fetch_assoc($products);
$totalRows_products = mysql_num_rows($products);
do {
$ch = curl_init ($row_products['picture']);
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20110319 Firefox/4.0');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$rawdata = curl_exec ($ch);
$http_status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close ($ch);
if($http_status==200){
$fp = fopen("images/products/".$row_products['productcode'].".jpg", 'w');
fwrite($fp, $rawdata);
fclose($fp);
echo ' -- Downloaded '.$newname.' to local: '.$newname.'';
} else {
echo ' -- Failed to download '.$row_products['picture'].'';
}
usleep(500);
} while ($row_products = mysql_fetch_assoc($products));
Your target website may require/check a combination of things. In order:
Location. Some websites only allow the referer to be a certain value (either their site or no referer, to prevent hotlinking)
Incorrect URL
Cookies. Yes, this can be checked
Authentication of some sort
The only way to do this is to sniff what a normal request looks like and to mimic it. Your MSIE user-agent string looks different from a genuine MSIE UA, however, and I'd consider changing it to an exact copy of a real one if I were you.
Could you get curl to output to a file (using the setopt for output stream) and telling us what error code you are getting, along with the URL of an image? This will help me be more precise.
Also, 0 isn't a success - it's a failure

Save image from URL with cURL

I need to save an image from url directly to my server, i've tried many methods but all seems doesn't work properly. file_put_contents($file_location, file_get_contents($image_url)); keeps me getting no file directory found error. Simple fopen and fwrite keeps returning corrupted image. This one worked, but it keeps returning html file instead of jpg file.
function getimg($url) {
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$user_agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
$process = curl_init($url);
curl_setopt($process, CURLOPT_HTTPHEADER, $headers);
curl_setopt($process, CURLOPT_HEADER, 0);
curl_setopt($process, CURLOPT_USERAGENT, $user_agent);
curl_setopt($process, CURLOPT_TIMEOUT, 30);
curl_setopt($process, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($process, CURLOPT_FOLLOWLOCATION, 1);
$return = curl_exec($process);
curl_close($process);
return $return;
}
$imgurl = 'http://some/url/to/image.jpg';
$imagename= basename($imgurl);
if(file_exists('./image/'.$imagename)){continue;}
$image = getimg($imgurl);
file_put_contents('image/'.$imagename,$image);
Something is missing?
Thanks.
Your code works correct. It downloads the image from the given url.
Your issue will be in the path where the image is stored.
if(file_exists('./image/'.$imagename)){continue;}
$image = getimg($imgurl);
file_put_contents('image/'.$imagename,$image);
In the above code check the path ./image/ and give the path as in the file_put_contents path.
This method works:
<?php
file_put_contents("/var/www/test/test.png", file_get_contents("http://www.google.com/intl/en_com/images/srpr/logo3w.png"));
?>
You need to enable allow_url_fopen and it's the simplest method. See http://php.net/manual/en/features.remote-files.php

how can i get data from mediwiki

Hi I am using following api to get the data from mediawiki. When I copy this url and paste it into a browser, an xml response appears.
http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main_Page&rvprop=timestamp|user|comment|content
but when I try to do with curl it gives me the error "Scripts should use an informative User-Agent string with contact information, or they may be IP-blocked without notice. ".
I am using following code for this. Can any one trace my error?
$url='http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main_Page&rvprop=timestamp|user|comment|content';
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
//curl_setopt($curl, CURLOPT_TIMEOUT, 1);
$objResponse = curl_exec($curl);
curl_close($curl);
echo $objResponse;die;
this will work to bypass there referrer user agent checks:
<?php
function getwiki($url="", $referer="", $userAgent="") {
if($url==""||$referer==""||$userAgent=="") { return false;};
$headers[] = 'Accept: image/gif, image/x-bitmap, image/jpeg, image/pjpeg';
$headers[] = 'Connection: Keep-Alive';
$headers[] = 'Content-type: application/x-www-form-urlencoded;charset=UTF-8';
$user_agent = $userAgent;
$process = curl_init($url);
curl_setopt($process, CURLOPT_HTTPHEADER, $headers);
curl_setopt($process, CURLOPT_HEADER, 0);
curl_setopt($process, CURLOPT_USERAGENT, $user_agent);
curl_setopt($process, CURLOPT_REFERER, $referer);
curl_setopt($process, CURLOPT_TIMEOUT, 30);
curl_setopt($process, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($process, CURLOPT_FOLLOWLOCATION, 1);
$return = curl_exec($process);
curl_close($process);
return $return;
}
//edited to include Adam Backstrom's sound advice
echo getwiki('http://en.wikipedia.org/w/api.php?action=query&prop=revisions&titles=API|Main_Page&rvprop=timestamp|user|comment|content', 'http://en.wikipedia.org/', 'Mozilla/5.0 (compatible; YourCoolBot/1.0; +http://yoursite.com/botinfo)');
?>
From the MediaWiki API:Quick start guide:
Pass a User-Agent header that properly identifies your client: don't use the default User-Agent from your client library, but use a custom one including the name of your client and the version number, something like MyCuteBot/0.1.
On Wikimedia wikis, failing to supply a User-Agent header or supplying an empty or generic one will cause the request to fail with an HTTP 403 error. See meta:User-Agent policy. Other MediaWiki wikis may have similar policies.
From meta:User-Agent policy:
If you run a bot, please send a User-Agent header identifying the bot and supplying some way of contacting you, e.g.: User-Agent: MyCoolTool (+http://example.com/MyCoolToolPage/)

Categories