How do I add a custom user agent in PHP cURL [duplicate] - php

This question already has answers here:
Reference: What is variable scope, which variables are accessible from where and what are "undefined variable" errors?
(3 answers)
PHP cURL how to add the User Agent value OR overcome the Servers blocking cURL requests?
(6 answers)
Closed 19 days ago.
This post was edited and submitted for review 19 days ago and failed to reopen the post:
Original close reason(s) were not resolved
I am trying to pass a custom browser user_agent with cURL in PHP.
Error I am getting:
Undefined variable $url in index-curl.php on line XX
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$homepage = file_get_contents_curl("https://www.example.com");

Is this code working for you?
<?php
function file_get_contents_curl($curl_url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $curl_url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$homepage = file_get_contents_curl("https://example.com");
echo $homepage;
?>

Related

403 Error with PHP cURL from external website

I am trying to get string from external website but i am getting 403 error. I have tried using custom user agent and same error. Can you please check and help me to get it fixed.
Code :
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$homepage = file_get_contents_curl("https://www.transunion.com/");
echo $homepage;
Help me fix this.

php curl proxy request blocked

i want to get web page of this url "http://namnak.com" in my server but this url blocked curl request from my server this is my code :
<?php
$proxy = "138.68.173.29 :8080";
$proxy = explode(':', $proxy);
$url = "http://namnak.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_HEADER, 1);
$exec = curl_exec($ch);
echo curl_error($ch);
print_r(curl_getinfo($ch));
echo $exec;
You must define the type of proxy you want to use, for example, SOCKS5:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
curl_setopt($ch, CURLOPT_PROXY, $proxy[0]);
curl_setopt($ch, CURLOPT_PROXYPORT, $proxy[1]);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURL_SOCKS5);
curl_setopt($ch, CURLOPT_HEADER, 1);
Start by trying that and we'll figure it out from there!
EDIT: In some cases, SOCKS5 isn't defined in curl, you can use its value which is equal to 7

Parse google trends (csv file)

I´m trying to get google trends data by parsing this csv file:
https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120
I tried it with php and curl, doesn´t work. I get an error page of google. Telling me "Bad Request". Alternatively I could download it with a cronjob but also this doesn´t work for me. Any chance to get/download this file?
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120");
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
?>
It's because there's a literal space in your URL. Here's a working example:
<?php
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
error_reporting(E_ALL);
$ch = curl_init();
$url = "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120";
$url = str_replace(" ", '%20', $url);
curl_setopt($ch, CURLOPT_URL, $url);
$agent= 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15); //time out of 15 seconds
$output = curl_exec($ch);
curl_close($ch);
print_r($output);
?>
But you really don't need to use CURL here:
<?php
$url = "https://trends.google.com/trends/api/widgetdata/comparedgeo/csv?req={'geo':{'country':'US'},'comparisonItem':[{'time':'2017-10-06 2018-10-06','complexKeywordsRestriction':{'keyword':[{'type':'BROAD','value':'travel'}]}}],'resolution':'REGION','locale':'de','requestOptions':{'property':'','backend':'IZG','category':0}}&token=APP6_UEAAAAAW7o-Y1_D87AoOXJJqulrVGiPmc3Cz6_Z&tz=-120";
$url = str_replace(" ", '%20', $url);
$result = file_get_contents($url);
print_r($result);
?>

Why curl return 400 bad request when i try to get page content?

i am trying to get web page content with curl from some websites but they return 400 bad request ( file_get_contents return empty ) here's the function i am using :
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)');
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Put error_reporting(E_ALL); line at the top file where you are calling this function.
It will generate the cause of an error.

Loading full document via curl

Hello i have problem with loading page via curl. Page html looks like this
<!--showModel--><!DOCTYPE html>
<html>
<head>
...
and then curl load page i get only
<!--showModel-->
how can i load full page?
php code looks like this
$agent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);

Categories