Hello i have problem with loading page via curl. Page html looks like this
<!--showModel--><!DOCTYPE html>
<html>
<head>
...
and then curl load page i get only
<!--showModel-->
how can i load full page?
php code looks like this
$agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
Related
I am trying to get string from external website but i am getting 403 error. I have tried using custom user agent and same error. Can you please check and help me to get it fixed.
Code :
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$homepage = file_get_contents_curl("https://www.transunion.com/");
echo $homepage;
Help me fix this.
i want to get web page of this url "http://namnak.com" in my server but this url blocked curl request from my server this is my code :
<?php
$proxy = "138.68.173.29 :8080";
$proxy = explode(':', $proxy);
$url = "http://namnak.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
You must define the type of proxy you want to use, for example, SOCKS5:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURL_SOCKS5);
curl_setopt($ch, CURLOPT_HEADER, 1);
Start by trying that and we'll figure it out from there!
EDIT: In some cases, SOCKS5 isn't defined in curl, you can use its value which is equal to 7
I´m trying to get google trends data by parsing this csv file:
https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120
I tried it with php and curl, doesn´t work. I get an error page of google. Telling me "Bad Request". Alternatively I could download it with a cronjob but also this doesn´t work for me. Any chance to get/download this file?
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120");
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
?>
It's because there's a literal space in your URL. Here's a working example:
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$ch = curl_init();
$url = "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120";
$url = str_replace(" ", '%20', $url);
curl_setopt($ch, CURLOPT_URL, $url);
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
print_r($output);
?>
But you really don't need to use CURL here:
<?php
$url = "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120";
$url = str_replace(" ", '%20', $url);
$result = file_get_contents($url);
print_r($result);
?>
i am trying to get web page content with curl from some websites but they return 400 bad request ( file_get_contents return empty ) here's the function i am using :
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Put error_reporting(E_ALL); line at the top file where you are calling this function.
It will generate the cause of an error.
I have sort php code using curl for get a web page.
$url = "http://google.com";
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_LOW_SPEED_LIMIT, 1);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
exit;
and here is the full reponse from google.
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
But now i don't want CURL get full page from google, i just want get response from <HTML> to <H1>301 Moved</H1> and stop curl response and close connection.
Anyone can help me to do this?
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); will have cURL follow any 301/302 redirects.