Get content from url after redirected - php

I am trying to set up a tool to get someone's Twitter ID without using twitter's API.
In my example, my username is quixthe2nd.
So if you enter someone's twitter username in this link:
https://twitter.com/quixthe2nd/profile_image?size=original
It would redirect to:
https://pbs.twimg.com/profile_images/1116692743361687553/0P-dk3sF.jpg
The ID is 1116692743361687553. It is listed after https://pbs.twimg.com/profile_images/.
My code is:
$url = "https://twitter.com/quixthe2nd/profile_image?size=original";
function get_redirect_target($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$headers = curl_exec($ch);
curl_close($ch);
// Check if there's a Location: header (redirect)
if (preg_match('/^Location: (.+)$/im', $headers, $matches))
return trim($matches[1]);
// If not, there was no redirect so return the original URL
// (Alternatively change this to return false)
return $url;
}
function get_redirect_final_target($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // follow redirects
curl_setopt($ch, CURLOPT_AUTOREFERER, 1); // set referer on redirect
curl_exec($ch);
$target = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
if ($target)
return $target;
return false;
}
$s = $target;
if(preg_match('/profile_images\/\K[^\/]+/',$s,$matches)) {
print($matches[0]);
}
My code doesn't output anything. How would I be able grab that via PHP?

You can use this regex and grab your value from group1,
profile_images\/([^\/]+)
Regex Demo
Alternatively, you can use \K so your full match is your intended text.
profile_images\/\K[^\/]+
Regex Demo using \K
PHP Code Demo
$s = "https://pbs.twimg.com/profile_images/1116692743361687553/0P-dk3sF.jpg";
if(preg_match('/profile_images\/\K[^\/]+/',$s,$matches)) {
print($matches[0]);
} else {
echo "Didn\'t match";
}
Prints,
1116692743361687553

Related

PHP Faking and Following a Form POST from a PHP Page....cURL

I am creating a checkout process in PHP and have 4 stages/pages. Each page collects different information about the user etc and the 4th stage/page is the secure checkout page which is hosted externally and accepts a POST form submission (from stage 3).
All would be fine however I need to validate the data in stage 3 before I send the user on to the external stage 4 so I looked into this and found this article on cURL...
http://www.html-form-guide.com/php-form/php-form-submit.html
All looked great but it only seems to post the data to the external 4th page but I need the user to actually be taken there at the same time so they see the 4th page. Ive tried...
header('Location: http://externalURLLink');
...straight after the cURL connection is closed but it didn't work.
The obvious way is to have a page that basically says "Now click here to go to our secure payment page but I would rather not do that.
Any suggestions?
Thanks
You could try the following function which may work for you:
function curl_redir_exec($ch,$test = false)
{
static $curl_loops = 0;
static $curl_max_loops = 20;
if ($curl_loops++>= $curl_max_loops)
{
$curl_loops = 0;
return FALSE;
}
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
list($header, $data) = explode("\n\n", $data, 2);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($http_code == 301 || $http_code == 302)
{
$matches = array();
preg_match('/Location:(.*?)\n/', $header, $matches);
$url = #parse_url(trim(array_pop($matches)));
if (!$url){
//couldn't process the url to redirect to
$curl_loops = 0;
return $data;
}
$last_url = parse_url(curl_getinfo($ch, CURLINFO_EFFECTIVE_URL));
if (!$url['scheme'])
$url['scheme'] = $last_url['scheme'];
if (!$url['host'])
$url['host'] = $last_url['host'];
if (!$url['path'])
$url['path'] = $last_url['path'];
$new_url = $url['scheme'] . '://' . $url['host'] . $url['path'] . ($url['query']?'?'.$url['query']:'');
curl_setopt($ch, CURLOPT_URL, $new_url);
return $this->curl_redir_exec($ch);
} else {
$curl_loops=0;
if($test){
return curl_getinfo($ch, CURLINFO_EFFECTIVE_URL).'<br />'.$http_code.'<br />'.$data;
}else{
return curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
}
}
}
You would use it like this:
$curl_session = curl_init($DESTINATION_URL);
curl_setopt($curl_session, CURLOPT_URL, $DESTINATION_URL);
curl_setopt($curl_session, CURLOPT_COOKIESESSION, 1);
curl_setopt($curl_session, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($curl_session, CURLOPT_HTTPHEADER, array('X-Forwarded-For: '.$_SERVER['REMOTE_ADDR']));
curl_setopt($curl_session, CURLOPT_VERBOSE, 1);
curl_setopt($curl_session, CURLOPT_POST, 1);
curl_setopt($curl_session, CURLOPT_POSTFIELDS, $POST_DATA);
curl_setopt($curl_session, CURLOPT_TIMEOUT, 30);
curl_setopt($curl_session, CURLOPT_SSL_VERIFYPEER, FALSE);
$redirect_url = $shop->curl_redir_exec($curl_session);
if(curl_errno($curl_session))
{
echo '<p>An error has occurred, please take note of the information below and contact support.</p>';
echo "<br>Errno : ".curl_errno($curl_session) ."<br>";
echo "<br>Error : ".curl_error($curl_session) ."<br>";
die();
}
curl_close($curl_session);
header("location:$redirect_url");
Hope this is useful.
Try curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

Grab web pages from redirected URL using CURL

I have an url http://www.example.com/all.php?f=1. When I open the url, it redirect to http://www.example.com/all.php with value of f (where f = 1). I try to grab the redirected URL using curl but it still fails. I try to search anything in this website but almost all of the question is about how to get the url, not how to grab it.
I try with this function :
function get_url_content($url,$timeout) {
$ch = curl_init($url); // initialize curl with given url
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]); // set useragent
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // write the response to a variable
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); // follow redirects if any
curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout); // max. seconds to execute
curl_setopt($ch, CURLOPT_FAILONERROR, 1); // stop when it encounters an error
$output = curl_exec($ch);
curl_close($ch);
if (!$output) {
return -1;
}
return $output;
}
But when I run it :
$url = 'http://www.example.com/all.php?f=1';
$me = get_url_content($url);
echo $me; //it should grab the page http://www.example.com/all.php
It returns -1, which mean there are no output. I confused about it. Any solution?
Change it to
$url = 'http://www.example.com/';
$me = file_get_content($url);
echo $me;

PHP: Get link address from redirected URL

How can I get the link address after a URL has been redirected?
Take for example this URL: http://www.boligsiden.dk/viderestilling/992cff55882a40f79e64b0a25e847a69
How can I make a PHP script echo the final URL? (http://www.eltoftnielsen.dk/default.aspx?side=sagsvisning&AutoID=125125&DID=140 in this case)
Note: The following solution isn't ideal for high traffic situations.
$url = 'http://www.boligsiden.dk/viderestilling/992cff55882a40f79e64b0a25e847a69';
file_get_contents($url);
preg_match('/(Location:|URI:)(.*?)\n/', implode("\n", $http_response_header), $matches);
if (isset($matches[0]))
{
echo $matches[0];
}
Here's what happens: file_get_contents() redirects and downloads the target website but writes the original response header into $http_response_header.
the preg_match tries to find the first "Location: x" match and returns it.
use this
<?php
$name="19875379";
$url = "http://www.ikea.co.il/default.asp?strSearch=".$name;
$ch = curl_init();
$timeout = 0;
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$header = curl_exec($ch);
$redir = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
//print_r($header);
$x = preg_match("/<script>location.href=(.|\n)*?<\/script>/", $header, $matches);
$script = $matches[0];
$redirect = str_replace("<script>location.href='", "", $script);
$redirect = "http://www.ikea.co.il" . str_replace("';</script>", "", $redirect);
echo $redirect;
?>
enter link description here

file_get_contents() how to fix error "Failed to open stream", "No such file"

I'm getting the following error when I try to run my PHP script:
failed to open stream: No such file or directory in C:\wamp\www\LOF\Data.php on line 3
script:
My code is as follows:
<?php
$json = json_decode(file_get_contents('prod.api.pvp.net/api/lol/euw/v1.1/game/by-summoner/20986461/recent?api_key=*key*'));
print_r($json);
?>
Note: *key* is a replacement for a string in the URL (my API key) and has been hidden for privacy reasons.
I removed the https:// from the URL to get one error to disappear.
Am I doing something wrong here? Maybe the URL?
The URL is missing the protocol information. PHP thinks it is a filesystem path and tries to access the file at the specified location. However, the location doesn't actually exist in your filesystem and an error is thrown.
You'll need to add http or https at the beginning of the URL you're trying to get the contents from:
$json = json_decode(file_get_contents('http://...'));
As for the following error:
Unable to find the wrapper - did you forget to enable it when you configured PHP?
Your Apache installation probably wasn't compiled with SSL support. You could manually try to install OpenSSL and use it, or use cURL. I personally prefer cURL over file_get_contents(). Here's a function you can use:
function curl_get_contents($url)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Usage:
$url = 'https://...';
$json = json_decode(curl_get_contents($url));
Why don't you use cURL ?
$yourkey="your api key";
$url="https://prod.api.pvp.net/api/lol/euw/v1.1/game/by-summoner/20986461/recent?api_key=$yourkey";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
$auth = curl_exec($curl);
if($auth)
{
$json = json_decode($auth);
print_r($json);
}
}
You may try using this
<?php
$json = json_decode(file_get_contents('./prod.api.pvp.net/api/lol/euw/v1.1/game/by-summoner/20986461/recent?api_key=*key*'));
print_r($json);
?>
The "./" allows to search url from current directory.
You may use
chdir($_SERVER["DOCUMENT_ROOT"]);
to change current working directory to root of your website if path is relative from root directory.
I just solve this by encode params in the url.
URL may be: http://abc/dgdc.php?p1=Hello&p2=some words
we just need to encode the params2.
$params2 = "some words";
$params2 = urlencode($params2);
$url = "http://abc/dgdc.php?p1=djkl&p2=$params2"
$result = file_get_contents($url);
just to extend Shankars and amals answers with simple unit testing:
/**
*
* workaround HTTPS problems with file_get_contents
*
* #param $url
* #return boolean|string
*/
function curl_get_contents($url)
{
$data = FALSE;
if (filter_var($url, FILTER_VALIDATE_URL))
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
$data = curl_exec($ch);
curl_close($ch);
}
return $data;
}
// then in the unit tests:
public function test_curl_get_contents()
{
$this->assertFalse(curl_get_contents(NULL));
$this->assertFalse(curl_get_contents('foo'));
$this->assertTrue(strlen(curl_get_contents('https://www.google.com')) > 0);
}
We can solve this issue by using Curl....
function my_curl_fun($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$feed = 'http://................'; /* Insert URL here */
$data = my_curl_fun($feed);
The actual problem of this error has nothing to do with file_get_content, the problem is the requested url if the url is not throwing content of the page and redirecting the request to some where else file_get_content says "Failed to open stream", just before file_get_contents check whether the url is working and not redirecting, here is the code:
function checkRedirect404($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
$out = curl_exec($ch);
// line endings is the wonkiest piece of this whole thing
$out = str_replace("\r", "", $out);
// only look at the headers
$headers_end = strpos($out, "\n\n");
if( $headers_end !== false ) {
$out = substr($out, 0, $headers_end);
}
$headers = explode("\n", $out);
foreach($headers as $header) {
if( substr($header, 0, 10) == "Location: " ) {
$target = substr($header, 10);
//echo "Redirects: $target<br>";
return true;
}
}
return false;
}
I hope below solution will work for you all as I was having the same problem with my websites...
For : $json = json_decode(file_get_contents('http://...'));
Replace with below query
$Details= unserialize(file_get_contents('http://......'));

get the last redirected url in curl php

Hi I know its a very common topic on StackOverFlow.
I have already spent my entire week to search it out.
I have a url : abc.com/default.asp?strSearch=19875379
this further redirect to this url: abc.com/default.asp?catid={170D4F36-39F9-4C48-88EB-CFC8DDF1F531}&details_type=1&itemid={49F6A281-8735-4B74-A170-B6110AF6CC2D}
I have made my effort to get the final url in my php code using Curl but can't make it.
here is my code:
<?php
$name="19875379";
$url = "http://www.ikea.co.il/default.asp?strSearch=".$name;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$a = curl_exec($ch);
curl_close( $ch );
// the returned headers
$headers = explode("\n",$a);
// if there is no redirection this will be the final url
$redir = $url;
// loop through the headers and check for a Location: str
$j = count($headers);
for($i = 0; $i < $j; $i++){
// if we find the Location header strip it and fill the redir var
//print_r($headers);
if(strpos($headers[$i],"Location:") !== false){
$redir = trim(str_replace("Location:","",$headers[$i]));
break;
}
}
// do whatever you want with the result
echo $redir;
?>
it gives me url "abc.com/default.asp?strSearch=19875379" instead of this url "abc.com/default.asp?catid={170D4F36-39F9-4C48-88EB-CFC8DDF1F531}&details_type=1&itemid={49F6A281-8735-4B74-A170-B6110AF6CC2D}"
Thanks in advance for your kind help :)
Thank you everyone for helping me in my situation.
Actually I want to develop a scraper in php for ikea website used in Israel (in Hebrew).
After putting a lot of hours I recognize that there is no server side redirection in url which I put to get the redirected url. It may be javascript redirection.
I have now implemented the below code and it works for me.
<?php
$name="19875379";
$url = "http://www.ikea.co.il/default.asp?strSearch=".$name;
$ch = curl_init();
$timeout = 0;
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$header = curl_exec($ch);
$redir = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
//print_r($header);
$x = preg_match("/<script>location.href=(.|\n)*?<\/script>/", $header, $matches);
$script = $matches[0];
$redirect = str_replace("<script>location.href='", "", $script);
$redirect = "http://www.ikea.co.il" . str_replace("';</script>", "", $redirect);
echo $redirect;
?>
Thanks again everyone :)
The accepted answer is applicable to a very specific scenario. So, most of us will be better off having a more general answer. Though you can extract the more general answer from within the accepted answer, separately having that part may be more helpful.
So, if you just want to get the last redirected URL, this code will help.
<?php
function redirectedUrl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']); // set browser info to avoid old browser warnings
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // allow url redirects
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); // get the return value of curl execution as a string
$html = curl_exec($ch);
// store last redirected url in a variable before closing the curl session
$lastUrl = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
curl_close($ch);
return $lastUrl;
}
First of all, I didn't see any redirection while I have given a run on your code. Anyway, here are few things you can do for this(keeping your approach intact):
First of all, make sure that the header will be returned to your curl output(in this case at $a).
curl_setopt($ch, CURLOPT_HEADER, true);
Now, separates only the header portion from the whole http response.
// header will be at 0 index, and html will be at 1 index.
$header = explode("\n\r",$a);
Explode the header string into headers array.
$headers = explode("\n", $header[0]);
You can use curl_getinfo() ...
http://php.net/manual/en/function.curl-getinfo.php

Categories