How to generate Facebook User Access Token using curl - php

I am doing a project where I need to import Facebook Page feeds. For accessing Facebook page feeds, I need a page_access_token and to generate page_access_token I need User access token.
Here my question is
1.How to generate this User_access_token using CURL ? Most of the solution requires APP_KEY & APP_SECRET. Is it not possible to get user_access_token without any APP ?
Once I get the User_access_token how do I use it to get Page access Token using CURL.

You can´t get ANY Token without an App, but you don´t need to program anything in order to get a User Token. These articles explain everything in detail:
https://developers.facebook.com/docs/facebook-login/access-tokens
http://www.devils-heaven.com/facebook-access-tokens/
https://developers.facebook.com/docs/facebook-login/
For example, you can use the API Explorer to select your App and generate User Tokens.

Wrong question, No tokens are needed
I just tried it and it took me less than 5 minutes, never having scraped FB in the past. I saved the page to my Server and brought up the page using my URL and it looked just like if I were on FB.
If a Browser can load the page with JavaScript disabled, then you can too.
You have to use https://m.facebook.com/, JavaScript is not required on their mobile site.
What you want to do, is not difficult at all.
Just go there in your Browser and copy the cookies key values into the Cookie: HTTP request header. Mine are x'ed out.
<?php
$request = array();
$request[] = 'Host: m.facebook.com';
$request[] = 'User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:39.0) Gecko/20100101 Firefox/39.0';
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
$request[] = 'Accept-Language: en-US,en;q=0.5';
$request[] = 'Accept-Encoding: gzip, deflate';
$request[] = 'DNT: 1';
$request[] = 'Cookie: datr=x; fr=x; lu=x s=xx; csm=x; xs=xx; c_user=x; p=-2; act=x; presence=x; noscript=1';
$request[] = 'Connection: keep-alive';
$request[] = 'Pragma: no-cache';
$request[] = 'ache-Control: no-cache';
$url = 'https://m.facebook.com/';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FILETIME, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 100);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_TIMEOUT,100);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$responseHeader = substr($data,0,$skip);
$data= substr($data,$skip);
$info = curl_getinfo($ch);
$info = var_export($info,true);
}
while(true){ // get cookies from response header
$s = strpos($head,'Set-Cookie: ',$e);
if (!$s){break;}
$s += 12;
$e = strpos($head,';',$s);
$cookie = substr($head,$s,$e-$s) ;
$s = strpos($cookie,'=');
$key = substr($cookie,0,$s);
$value = substr($cookie,$s);
$cookies[$key] = $value;
}
$cookie = ''; // format cookies for next request header
$show = '';
$head = '';
$delim = '';
foreach ($cookies as $k => $v){
$cookie .= "$delim$k$v";
$delim = '; ';
}
$fp = fopen("fb.html",'w');
fwrite($fp,"$data\n$info\n$responseHeader");
fclose($fp);
readfile('fb.html');

Related

PHP - Loop cURL with increase id of URL

with the following code a name with an ID (each name in the gnd can be addressed by an ID) is received via the GND interface. This works.
Now i want to get many names at the same time with a cURL loop. The ID of the URL must always be increased by one and the cURL request must loop. How can I do this?
With this Code i receive for example names from the GND database
<?php
header('Content-type: text/html; charset=utf-8');
$User_Agent = 'Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0';
$url = "http://hub.culturegraph.org/entityfacts/118600001";
$request_headers = [];
$request_headers[] = 'Accept: application/json';
$request_headers[] = 'charset=utf-8';
$request_headers[] = 'Content-Type: application/json; charset=utf-8';
$request_headers[] = 'Accept-Encoding: gzip, deflate, identity';
$request_headers[] = 'Accept-Language: de,en-US;q=0.7,en;q=0.3';
for ($i = 1; $i <= 10; $i++)
{
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, $User_Agent);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_ENCODING, "");
$result = curl_exec($ch);
$code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
}
curl_close($ch);
$data = json_decode($result, true);
if ($code == 200) {
$data = json_decode($result, true);
echo 'Erfolg';
} else {
$error = $data['Error'];
echo 'Beim anfordern der GND-Daten ist ein Fehler aufgetreten: Fehlercode ' . $error;
echo ' Zurueck<br />';
}
var_dump($data['preferredName']);
Result for URL with ID 118600001 = Gerhardt von Reutern
But how must the code be adapted, so that also the names of the URL's 118600002, 118600003 and so on are output? So as often as it is specified. Be it 100 or 1000 times.
change your $url variable to something like $base_url which is the url without the ID
$base_url = "http://hub.culturegraph.org/entityfacts/";
Then in your for loop you do:
$ch = curl_init($base_url . (118600001 + $i));
this rule can be removed, this is not necessary:
curl_setopt($ch, CURLOPT_URL, $url);
You should also handle the response inside the for loop, else you will only see the last person's name, after an incredibly long load time.

Misspelled RegEx in PHP function preg_match()

This is a question about PHP, cURL, preg_match(), RegEx, and Instagram.
I'm trying to write a cURL PHP code that logs-in to Instagram, and then access content only available to a validated (logged-in) user. In this question there's an interesting code just for that.
I copy-pasted the code, which is the following:
<?php
$username = "yourname";
$password = "yourpass";
$useragent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36";
$cookie=$username.".txt";
#unlink(dirname(__FILE__)."/".$cookie);
$url="https://www.instagram.com/accounts/login/?force_classic_login";
$ch = curl_init();
$arrSetHeaders = array(
"User-Agent: $useragent",
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.5',
'Accept-Encoding: deflate, br',
'Connection: keep-alive',
'cache-control: max-age=0',
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $arrSetHeaders);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$page = curl_exec($ch);
curl_close($ch);
// try to find the actual login form
if (!preg_match('/<form method="POST" id="login-form" class="adjacent".*?<\/form>/is', $page, $form)) {
die('Failed to find log in form!');
}
$form = $form[0];
// find the action of the login form
if (!preg_match('/action="([^"]+)"/i', $form, $action)) {
die('Failed to find login form url');
}
$url2 = $action[1]; // this is our new post url
// find all hidden fields which we need to send with our login, this includes security tokens
$count = preg_match_all('/<input type="hidden"\s*name="([^"]*)"\s*value="([^"]*)"/i', $form, $hiddenFields);
$postFields = array();
// turn the hidden fields into an array
for ($i = 0; $i < $count; ++$i) {
$postFields[$hiddenFields[1][$i]] = $hiddenFields[2][$i];
}
// add our login values
$postFields['username'] = $username;
$postFields['password'] = $password;
$post = '';
// convert to string, this won't work as an array, form will not accept multipart/form-data, only application/x-www-form-urlencoded
foreach($postFields as $key => $value) {
$post .= $key . '=' . urlencode($value) . '&';
}
$post = substr($post, 0, -1);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $page, $matches);
$cookieFileContent = '';
foreach($matches[1] as $item)
{
$cookieFileContent .= "$item; ";
}
$cookieFileContent = rtrim($cookieFileContent, '; ');
$cookieFileContent = str_replace('sessionid=; ', '', $cookieFileContent);
$oldContent = file_get_contents(dirname(__FILE__)."/".$cookie);
$oldContArr = explode("\n", $oldContent);
if(count($oldContArr))
{
foreach($oldContArr as $k => $line)
{
if(strstr($line, '# '))
{
unset($oldContArr[$k]);
}
}
$newContent = implode("\n", $oldContArr);
$newContent = trim($newContent, "\n");
file_put_contents(
dirname(__FILE__)."/".$cookie,
$newContent
);
}
$arrSetHeaders = array(
'origin: https://www.instagram.com',
'authority: www.instagram.com',
'upgrade-insecure-requests: 1',
'Host: www.instagram.com',
"User-Agent: $useragent",
'content-type: application/x-www-form-urlencoded',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.5',
'Accept-Encoding: deflate, br',
"Referer: $url",
"Cookie: $cookieFileContent",
'Connection: keep-alive',
'cache-control: max-age=0',
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $arrSetHeaders);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
sleep(5);
$page = curl_exec($ch);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $page, $matches);
$cookies = array();
foreach($matches[1] as $item) {
parse_str($item, $cookie1);
$cookies = array_merge($cookies, $cookie1);
}
var_dump($page);
curl_close($ch);
?>
You must, of course, change the $username's value from yourname to an actual username, and $password's value from yourpass to the corresponding password of the username. Also, I don't think this code will work if 2-step authentication is enabled in the account of the user (ie., for example, a MMS is sent to the linked phone number, or an email is sent to the linked email), therefore, 2-step authentication must be disabled in the user's account settings.
However, when I run this PHP script in Chrome, I get the message Failed to find log in form!, meaning that, on line 35, the pattern (written as a RegEx) /<form method="POST" id="login-form" class="adjacent".*?<\/form>/is wasn't found in the variable $page, which at the same time is basically the HTML of the classic Instagram login page.
It's weird that such pattern is not being found, because there's indeed a HTML named <form method="POST" id="login-form" class="adjacent" (to check it yourself, go to the login page, press Ctrl+U, then Ctrl+F, and then paste the previous pattern). So, I assume this is because the RegEx is somehow misspelled. Is that correct?
Even weirder, I tested it on the page RegEx101.com, and the test was successful. So, why is PHP saying the pattern wasn't found?
The idea is that the pattern can be found in the code of the original question, so that I can log-in automatically/programmatically. Keep in mind that more preg_match() functions are used later in the code, eg. on lines 42, 48 (preg_match_all()), 70 (preg_match_all()), and 137 (preg_match_all()).
EDIT: I realized that if I add var_dump($page); just before line 35, I get bool(false) (in other words, the HTML output is bool(false) Failed to find log in form!). I think this is the reason why the code returns the message Failed to find log in form!. However, why is the value of $page returning that?

Instagram API retrieve the code using PHP

I try to use the Instagram API but it's really not easy.
According to the API documentation, a code must be retrieved in order to get an access token and then make requests to Instagram API.
But after few try, I don't succeed.
I already well-configured the settings in https://www.instagram.com/developer
I call the url api.instagram.com/oauth/authorize/?client_id=[CLIENT_ID]&redirect_uri=[REDIRECT_URI]&response_type=code with curl, but I don't have the redirect uri with the code in response.
Can you help me please ;)!
I would recommend you use one of the existing PHP instagram client libraries like this https://github.com/cosenary/Instagram-PHP-API
I did this not too long ago, here's a good reference:
https://auth0.com/docs/connections/social/instagram
Let me know if it helps!
I've made this code, I hope it doesnt have error, but i've just made it for usecase like you wantedHere is the code, I'll explain it below how this code works.
$authorization_url = "https://api.instagram.com/oauth/authorize/?client_id=".$instagram_client_id."&redirect_uri=".$your_website_redirect_uri."&response_type=code";
$username='ig_username';
$password='ig_password';
$_defaultHeaders = array(
'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.5',
'Accept-Encoding: ',
'Connection: keep-alive',
'Upgrade-Insecure-Requests: 1',
'Cache-Control: max-age=0'
);
$ch = curl_init();
$cookie='/application/'.strtoupper(VERSI)."instagram_cookie/instagram.txt";
curl_setopt( $ch, CURLOPT_POST, 0 );
curl_setopt( $ch, CURLOPT_HTTPGET, 1 );
if($this->token!==null){
array_push($this->_defaultHeaders,"Authorization: ".$this->token);
}
curl_setopt( $ch, CURLOPT_HTTPHEADER, $this->_defaultHeaders);
curl_setopt( $ch, CURLOPT_HEADER, true);
curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt( $ch, CURLOPT_COOKIEFILE,getcwd().$cookie );
curl_setopt( $ch, CURLOPT_COOKIEJAR, getcwd().$cookie );
curl_setopt($this->curlHandle,CURLOPT_URL,$url);
curl_setopt($this->curlHandle,CURLOPT_FOLLOWLOCATION,true);
$result = curl_exec($this->curlHandle);
$redirect_uri = curl_getinfo($this->curlHandle, CURLINFO_EFFECTIVE_URL);
$form = explode('login-form',$result)[1];
$form = explode("action=\"",$form)[1];
// vd('asd',$form);
$action = substr($form,0,strpos($form,"\""));
// vd('action',$action);
$csrfmiddlewaretoken = explode("csrfmiddlewaretoken\" value=\"",$form);
$csrfmiddlewaretoken = substr($csrfmiddlewaretoken[1],0,strpos($csrfmiddlewaretoken[1],"\""));
//finish getting parameter
$post_param['csrfmiddlewaretoken']=$csrfmiddlewaretoken;
$post_param['username']=$username;
$post_param['password']=$password;
//format instagram cookie from vaha's answer https://stackoverflow.com/questions/26003063/instagram-login-programatically
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $result, $matches);
$cookieFileContent = '';
foreach($matches[1] as $item)
{
$cookieFileContent .= "$item; ";
}
$cookieFileContent = rtrim($cookieFileContent, '; ');
$cookieFileContent = str_replace('sessionid=; ', '', $cookieFileContent);
$cookie=getcwd().'/application/'.strtoupper(VERSI)."instagram_cookie/instagram.txt";
$oldContent = file_get_contents($cookie);
$oldContArr = explode("\n", $oldContent);
if(count($oldContArr))
{
foreach($oldContArr as $k => $line)
{
if(strstr($line, '# '))
{
unset($oldContArr[$k]);
}
}
$newContent = implode("\n", $oldContArr);
$newContent = trim($newContent, "\n");
file_put_contents(
$cookie,
$newContent
);
}
// end format
$useragent = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0";
$arrSetHeaders = array(
'origin: https://www.instagram.com',
'authority: www.instagram.com',
'upgrade-insecure-requests: 1',
'Host: www.instagram.com',
"User-Agent: $useragent",
'content-type: application/x-www-form-urlencoded',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.5',
'Accept-Encoding: deflate, br',
"Referer: $redirect_uri",
"Cookie: $cookieFileContent",
'Connection: keep-alive',
'cache-control: max-age=0',
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__)."/".$cookie);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, $arrSetHeaders);
curl_setopt($ch, CURLOPT_URL, $this->base_url.$action);
curl_setopt($ch, CURLOPT_REFERER, $redirect_uri);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post_param));
sleep(5);
$page = curl_exec($ch);
preg_match_all('/^Set-Cookie:\s*([^;]*)/mi', $page, $matches);
$cookies = array();
foreach($matches[1] as $item) {
parse_str($item, $cookie1);
$cookies = array_merge($cookies, $cookie1);
}
var_dump($page);
Step 1:
We need to get to the login page first.
We can access it using curl get, with CURLOPT_FOLLOWLOCATION set to true so that we will be redirected to the login page, we access our application instagram authorization url
$authorization_url = "https://api.instagram.com/oauth/authorize/?client_id=".$instagram_client_id."&redirect_uri=".$your_website_redirect_uri."&response_type=code";
$username='ig_username';
This is step one from this Instagram documentation here
Now the result of the first get curl we have the response page and its page uri that we store at $redirect_uri, this must be needed and placed on referer header when we do http post for login.
After get the result of login_page, we will need to format the cookie, I know this and use some code from vaha answer here vaha's answer
Step 2:
After we get the login_page we will extract the action url , extract csrfmiddlewaretoken hidden input value.
After we get it, we will do a post parameter to login.
We must set the redirect uri, and dont forget the cookiejar, and other header setting like above code.After success sending the parameter post for login, Instagram will call your redirect uri, for example https://www.yourwebsite.com/save_instagram_code at there you must use or save your instagram code to get the access token using curl again ( i only explain how to get the code :D)
I make this in a short time, I'll update the code which I have tested and work if i have time, Feel free to suggest an edit of workable code or a better explanation.

Why can I not scrape the title off this site?

I'm using simple-html-dom to scrape the title off of a specified site.
<?php
include('simple_html_dom.php');
$html = file_get_html('http://www.pottermore.com/');
foreach($html->find('title') as $element)
echo $element->innertext . '<br>';
?>
Any other site I've tried works, apple.com for example.
But if I input pottermore.com, it doesn't output anything. Pottermore has flash elements on it, but the home screen I'm trying to scrape the title off of has no flash, just html.
This works for me :)
$url = 'http://www.pottermore.com/';
$html = get_html($url);
file_put_contents('page.htm',$html);//just to test what you have downloaded
echo 'The title from: '.$url.' is: '.get_snip($html, '<title>','</title>');
function get_html($url)
{
$ch = curl_init();
$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: "; //browsers keep this blank.
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows;U;Windows NT 5.0;en-US;rv:1.4) Gecko/20030624 Netscape/7.1 (ax)');
curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 20);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE);
curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE);
$result = curl_exec ($ch);
curl_close ($ch);
return($result);
}
function get_snip($string,$start,$end,$trim_start='1',$trim_end='1')
{
$startpos = strpos($string,$start);
$endpos = strpos($string,$end,$startpos);
if($trim_start!='')
{
$startpos += strlen($start);
}
if($trim_end=='')
{
$endpos += strlen($end);
}
return(substr($string,$startpos,($endpos-$startpos)));
}
Just to confirm what others are saying, if you don't send a user agent string this site sends 403 Forbidden.
Adding this worked for me:
User-Agent: Mozilla/5.0 (Windows;U;Windows NT 5.0;en-US;rv:1.4) Gecko/20030624 Netscape/7.1 (ax)
The function file_get_html uses file_get_contents under the covers. This function can pull data from a URL, but to do so, it sends a User Agent string.
By default, this string is empty. Some webservers use this fact to detect that a non-browser is accessing its data and opt to forbid this.
You can set user_agent in php.ini to control the User Agent string that gets sent. Or, you could try:
ini_set('user_agent','UA-String');
with 'UA-String' set to whatever you like.

file_get_contents script works with some websites but not others

I'm looking to build a PHP script that parses HTML for particular tags. I've been using this code block, adapted from this tutorial:
<?php
$data = file_get_contents('http://www.google.com');
$regex = '/<title>(.+?)</';
preg_match($regex,$data,$match);
var_dump($match);
echo $match[1];
?>
The script works with some websites (like google, above), but when I try it with other websites (like, say, freshdirect), I get this error:
"Warning: file_get_contents(http://www.freshdirect.com) [function.file-get-contents]: failed to open stream: HTTP request failed!"
I've seen a bunch of great suggestions on StackOverflow, for example to enable extension=php_openssl.dll in php.ini. But (1) my version of php.ini didn't have extension=php_openssl.dll in it, and (2) when I added it to the extensions section and restarted the WAMP server, per this thread, still no success.
Would someone mind pointing me in the right direction? Thank you very much!
It just requires a user-agent ("any" really, any string suffices):
file_get_contents("http://www.freshdirect.com",false,stream_context_create(
array("http" => array("user_agent" => "any"))
));
See more options.
Of course, you can set user_agent in your ini:
ini_set("user_agent","any");
echo file_get_contents("http://www.freshdirect.com");
... but I prefer to be explicit for the next programmer working on it.
$html = file_get_html('http://google.com/');
$title = $html->find('title')->innertext;
Or if you prefer with preg_match and you should be really using cURL instead of fgc...
function curl($url){
$headers[] = "User-Agent:Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13";
$headers[] = "Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$headers[] = "Accept-Language:en-us,en;q=0.5";
$headers[] = "Accept-Encoding:gzip,deflate";
$headers[] = "Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$headers[] = "Keep-Alive:115";
$headers[] = "Connection:keep-alive";
$headers[] = "Cache-Control:max-age=0";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
curl_setopt($curl, CURLOPT_ENCODING, "gzip");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
$data = curl_exec($curl);
curl_close($curl);
return $data;
}
$data = curl('http://www.google.com');
$regex = '#<title>(.*?)</title>#mis';
preg_match($regex,$data,$match);
var_dump($match);
echo $match[1];
Another option: Some hosts disable CURLOPT_FOLLOWLOCATION so recursive is what you want, also will log into a text file any errors. Also a simple example of how to use DOMDocument() to extract the content, obviously its not extensive but something you could build appon.
<?php
function file_get_site($url){
(function_exists('curl_init')) ? '' : die('cURL Must be installed. Ask your host to enable it or uncomment extension=php_curl.dll in php.ini');
$curl = curl_init();
$header[0] = "Accept: text/xml,application/xml,application/xhtml+xml,";
$header[0] .= "text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0 Firefox/5.0');
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_REFERER, $url);
curl_setopt($curl, CURLOPT_ENCODING, 'gzip,deflate');
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_TIMEOUT, 60);
$html = curl_exec($curl);
$status = curl_getinfo($curl);
curl_close($curl);
if($status['http_code']!=200){
if($status['http_code'] == 301 || $status['http_code'] == 302) {
list($header) = explode("\r\n\r\n", $html, 2);
$matches = array();
preg_match("/(Location:|URI:)[^(\n)]*/", $header, $matches);
$url = trim(str_replace($matches[1],"",$matches[0]));
$url_parsed = parse_url($url);
return (isset($url_parsed))? file_get_site($url):'';
}
$oline='';
foreach($status as $key=>$eline){$oline.='['.$key.']'.$eline.' ';}
$line =$oline." \r\n ".$url."\r\n-----------------\r\n";
$handle = #fopen('./curl.error.log', 'a');
fwrite($handle, $line);
return FALSE;
}
return $html;
}
function get_content_tags($source,$tag,$id=null,$value=null){
$xml = new DOMDocument();
#$xml->loadHTML($source);
foreach($xml->getElementsByTagName($tag) as $tags) {
if($id!=null){
if($tags->getAttribute($id)==$value){
return $tags->getAttribute('content');
}
}
return $tags->nodeValue;
}
}
$source = file_get_site('http://www.freshdirect.com/about/index.jsp');
echo get_content_tags($source,'title'); //FreshDirect
echo get_content_tags($source,'meta','name','description'); //Online grocer providing high quality fresh......
?>

Categories