I have this function:
function file_get_contents_curl($url) {
$ch = curl_init();
$ua = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)';
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, $ua);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Now I have a strange issue: I;m trying to search in Google
using the "inurl:example.com", like this:
$email = $_GET["email"];
$in = "inurl:example.com+" . $email;
$in = str_replace(' ','+',$in);
$url = 'http://www.google.com/search?hl=en&q='.$in;
and every time i get 0 results.
However, when i get rid from the "inurl:" and replace it with some other
condition, like this:
$in = "example.com+" . $email;
it works well and i see the results.
So something about the COLON ("inurl:") causes trouble for the file_get_contents_curl function.
I tried the same inurl: string in file_get_html function (instead of file_get_contents_curl)
$html = file_get_html($url);
with no problem at all.
Any ideas?
EDIT:
$in = "+inurl:pastebin.com";
$in = str_replace("%3A",":",$in);
(NOT) solved the problem.
EDIT:
Sometimes it works and sometimes not.
So it's something related to Google blocking/rate-limit
and stuff like that.
Thanks anyway.
Related
I m trying to integrate a payment method on a website, the first thing I did, I tried a curl code to test it using git console and it works just fine, then I tried to execute the curl command using PHP. I created a file then I used this code:
<?php
$endpoint_url = 'https://secure.payinspect.com';
$params = [
'action'=>'SALE',
'order_id'=>'ORDER12345',
'order_amount'=>'1.99',
'order_currency'=>'USD',
'order_description'=>'Product',
'card_number'=>'4111111111111111',
'card_exp_month'=>'05',
'card_exp_year'=>'2020',
'card_cvv2'=>'000',
'payer_first_name'=>'John',
'payer_last_name'=>'Doe',
'payer_address'=>'BigStreet',
'payer_country'=>'US',
'payer_state'=>'CA',
'payer_city'=>'City',
'payer_zip'=>'123456',
'payer_email'=>'doe#example',
'payer_phone'=>'199999999',
'payer_ip'=>'123.123.123.123',
'term_url_3ds'=>'http://client.site.com/return.php',
'recurring_init'=>'N',
'hash'=>'e3dd86f469f40a5cfedf96a82ff257af'
];
$buff = [];
foreach ($params as $k => $v) {
array_push($buff, "{$k}={$v}");
}
$url = $endpoint_url . implode('&', $buff);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_close($ch);
if ($result===false){ print curl_error($curl); }
$response = json_decode($result, true);
echo $result;
?>
but I got this error :
Forbidden
You don't have permission to access
I googled for this error and I tried to add this line
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36');
$result = curl_exec($ch);
but i still got the same error . so what causes this problem and how cauld i fix it
Assuming everything else is correct - this might fix the problem.
<?php
$endpoint_url = 'https://secure.payinspect.com';
$params = [
'action'=>'SALE',
'order_id'=>'ORDER12345',
'order_amount'=>'1.99',
'order_currency'=>'USD',
'order_description'=>'Product',
'card_number'=>'4111111111111111',
'card_exp_month'=>'05',
'card_exp_year'=>'2020',
'card_cvv2'=>'000',
'payer_first_name'=>'John',
'payer_last_name'=>'Doe',
'payer_address'=>'BigStreet',
'payer_country'=>'US',
'payer_state'=>'CA',
'payer_city'=>'City',
'payer_zip'=>'123456',
'payer_email'=>'doe#example',
'payer_phone'=>'199999999',
'payer_ip'=>'123.123.123.123',
'term_url_3ds'=>'http://client.site.com/return.php',
'recurring_init'=>'N',
'hash'=>'e3dd86f469f40a5cfedf96a82ff257af'
];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $endpoint_url);
// -- this sets the request method to POST ----
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
// --- end ----
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_close($ch);
if ($result===false) { print curl_error($curl); }
$response = json_decode($result, true);
echo $result;
I can't say for certain (which means this isn't a great answer) but by suffixing the parameters to the URL, in the way you're doing currently, you're creating a GET request rather than a POST one.
It's quite likely that the receiving service is expecting to see your hash value (and everything else) in the POST data - and as it doesn't see it there, it rejects your request completely.
I am trying to scrape some data from Yahoo, but the xpath query is returning me length 0 when I var_dump this. Here's a portion of my scraping code.
error_reporting(0);
function curl($url) {
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; MSIE 7.0; Windows NT 6.0; en-US)');
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_AUTOREFERER, false);
curl_setopt($curl, CURLOPT_FRESH_CONNECT, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 200);
return curl_exec($curl);
}
$page = curl('https://www.yahoo.com');
$dom = new DOMDocument();
$dom->loadHTML($page);
$xpath = new DOMXPath($dom);
$link = $xpath->query('//li[#style="background-color:#fafaff;"]/div/div/div/h3/a');
foreach ($link as $links) {
$get_title[] = $links->nodeValue;
$get_link[] = $links->getAttribute('href');
}
This code has no syntax errors, but there is a logical error.
Your code is working correctly. The problem is that the HTML returned by Yahoo.com simply doesn't contain any li elements that match your selector. You can see this by looking at the contents of $page.
I check each n every thing . but at last i found another solution . this code is not working . so it’s rubbish . Thanks . The Exact way to scrape data from yahoo is so simple . Using Ajax you can easily scrape data . first load yahoo page and then with the help of ajax scrape anything .
Thanks To all who respond on my question .
For what i'm trying to do, i use PHP5 in CLI, and cURL extension.
I'm trying to download a file from youtube's server, it works fine with any navigator,
the link is something like that;
`http://youtube.com/get_video_info?video_id=VIDEO_ID
exemple: http://youtube.com/get_video_info?video_id=9pQxmD6Bhd
When i access this file trough my navigator, it prompt me with a download box for the file
'get_video_info', when downloaded the file content some data, ..
The problem is to get this file with cURL, i keep getting this error message;
status=fail&errorcode=2&reason=Invalid+parameters.
This is the code ( i tried to change some option, but i'm not familliar with cURL, so i'm stuck.
$c = curl_init();
curl_setopt($c, CURLOPT_URL, "http://youtube.com/get_video_info?video_id=9pQxmD6Bhd");
curl_setopt($c, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1");
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($c, CURLOPT_HEADER, false);
$output = curl_exec($c);
if($output === false)
{
trigger_error('Erreur curl : '.curl_error($c),E_USER_WARNING);
}
else
{
var_dump($output);
}
curl_close($c);
I tried to use some curl_setopt options, like CURLOPT_TRANSFERTEXT with no success.
I definitely need help !
Thanks for answers, and sorry if i did something that dont respect the rules here, it's my first post.
EDIT
Here is the code to download youtube video ( .ogg ) with php in cli.
<?php
/*Youtube URL and ID*/
$youtube_video = "http://www.youtube.com/watch?v=Ftud51NhY2I";
$yt_id = explode("=", $youtube_video);
$id = $yt_id[1];
/*
Functions
*/
function get_link($raw){
$url = rawurldecode(rawurldecode($raw));
$url = explode("&qual", $url);
return $url[0];
}
/*
Here we go
Query video token
*/
$c = curl_init();
curl_setopt($c, CURLOPT_URL, $youtube_video);
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_HEADER, false);
$output = curl_exec($c);
if($output === false)
{
trigger_error('Erreur curl : '.curl_error($c),E_USER_WARNING);
}
else{}
curl_close($c);
/*
Get Video infos
*/
$c = curl_init();
curl_setopt($c, CURLOPT_URL, "http://youtube.com/get_video_info?video_id=".$id);
curl_setopt($c, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1");
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
curl_setopt($c, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($c, CURLOPT_HEADER, false);
$output = curl_exec($c);
if($output === false){trigger_error('Erreur curl : '.curl_error($c),E_USER_WARNING);}
else{}
curl_close($c);
/*Get RAW link*/
$temp = explode("url_encoded_fmt_stream_map=url%3D", $output);
$url = explode("=", $temp[1]);
$url = get_link($url[0]);
/*Get Video name*/
$temp = "";
$temp = explode("title=", $output);
$title = explode("&", $temp[1]);
$title = rawurldecode(rawurldecode($title[0]));
$replace = array(':', '+', '\\', '/', '"', '<', '>', '|', '(', ')', '\'');
$title = str_replace($replace, ' ',$title);
//echo $title;
/*
Download Video
*/
$url = $url;
$path = $title.'.ogg';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$data = curl_exec($ch);
curl_close($ch);
file_put_contents($path, $data);
echo "Done... \r\n";
?>
You get error message because the video_id parameter isn't valid.
Try changing that ID and it should work correctly.
http://www.youtube.com/watch?v=9pQxmD6Bhd - does not exist
youtube has changed their system. now it is working only with the real IP who use the get_video_info system. when you try with cURL it sends the server IP to Youtube, then you have to download videos with the servers IP, because youtube system creates the direct video download urls with given IP.
Using Wikiepdia API link to get some basic informations about some world known characters.
Example : (About Dave Longaberger)
This would show as following
Now my question
I'd like to parse the xml to get such basic informations between <extract></extract> to show it.
Here is my idea but failed (I/O warning : failed to load external entity)
<?PHP
$url = 'http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Dave Longaberger&format=xml&exintro=1';
$xml = simplexml_load_file($url);
// get extract
$text=$xml->pages[0]->extract;
// show title
echo $text;
?>
Another idea but also failed (failed to open stream: HTTP request failed!)
<?PHP
function get_url_contents($url){
$crl = curl_init();
$timeout = 5;
curl_setopt ($crl, CURLOPT_URL,$url);
curl_setopt ($crl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($crl, CURLOPT_CONNECTTIMEOUT, $timeout);
$ret = curl_exec($crl);
curl_close($crl);
return $ret;
}
$url = "http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Dave Longaberger&format=xml&exintro=1";
$text = file_get_contents($url);
echo $text;
?>
so any idea how to do it. ~ Thanks
Update (after added urlencode or rawurlencode still not working)
$name = "Dave Longaberger";
$name = urlencode($name);
$url = 'http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles='.$name.'&format=xml&exintro=1';
$text = file_get_contents($url);
Also not working
$url = 'http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Dave Longaberger&format=xml&exintro=1';
$url = urlencode($url);
$text = file_get_contents($url);
nor
$url = 'http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles='.rawurlencode('Dave Longaberger').'&format=xml&exintro=1';
$text = file_get_contents($url);
Well so i really don't know looks like it is impossible by somehow.
Set the User Agent Header in your curl request, wikipedia replies with error 403 forbidden otherwise.
<?PHP
$url = "http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Dave+Longaberger&format=xml&exintro=1";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
$xml = curl_exec($ch);
curl_close($ch);
echo $xml;
?>
Alternatively:
ini_set("user_agent","Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
$url = "http://en.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Dave+Longaberger&format=xml&exintro=1";
$xml = simplexml_load_file($url);
$extracts = $xml->xpath("/api/query/pages/page/extract");
var_dump($extracts);
Look at the note in this php man page
http://php.net/manual/en/function.file-get-contents.php
If you're opening a URI with special characters, such as spaces, you need to encode the URI with urlencode().
A similar question has been posted at but i could not find the solution there
Curl error Could not resolve host: saved_report.xml; No data record of requested type"
<?php
$url="http://en.wikipedia.org/wiki/Pakistan";
$ch = curl_init(urlencode($url));
echo $ch;
// used to spoof that coming from a real browser so we don't get blocked by some sites
$useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 4);
curl_setopt($ch, CURLOPT_TIMEOUT, 8);
curl_setopt($ch, CURLOPT_LOW_SPEED_TIME, 10);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$content = curl_exec($ch);
$info = curl_getinfo($ch);
if ($content === false || $info['http_code'] != 200) {
$content = "No cURL data returned for $url [". $info['http_code']. "]";
if (curl_error($ch))
$content .= "\n". curl_error($ch);
}
else {
// 'OK' status; format $output data if necessary here:
echo "...";
}
echo $content;
curl_close($ch);
?>
when i paste the same address in browser i am able to access the webpage. but when i run this script i get the error message. Can anyone please help me.
Thanks
Remove the urlencode call.
remove the urlencode($url) it should be:
$ch = curl_init($url);
Well.
If you remove urlencode() with instantiating your $ch-var, you go just fine. urlencode() is definitely wrong here.
Good:
$ch = curl_init($url);
Bad:
$ch = curl_init(urlencode($url));
$ch = curl_init($url);
instead of
$ch = curl_init(urlencode($url));