Curl doesn't send cookie from dev server, but when i run script from another server, it's works well. I can't understand what's wrong with dev server.
$curl=curl_init($request);
//$cook = './cook/1.txt';
//curl_setopt($curl, CURLOPT_COOKIEJAR, $cook);
//curl_setopt($curl, CURLOPT_COOKIEFILE, $cook)
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36");
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER,true);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
//curl_setopt($curl, CURLOPT_ENCODING, 'UTF-8');
curl_setopt($curl, CURLOPT_VERBOSE,1);
curl_setopt($curl, CURLINFO_HEADER_OUT,1);
curl_setopt($curl, CURLOPT_COOKIE, "departureCity=2; path=/;");
$out = curl_exec($curl);
$info = curl_getinfo($curl);
curl_close($curl);
$info:
Array
(
[request_header] => GET /hotels/greece/showhotel/12_islands_villas_26 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36
Host: SITE
Accept: */*
Referer: SITE
)
there is now cookie data in header
Related
I use the PHP cURL code from this answer https://stackoverflow.com/a/46834320/12616388. When I run the script on localhost I get the desired output. If I run it from my web server, I retrieve a captcha to verify that I am not a bot. I am new to this topic and would like to know the cause. My code:
$request = array();
//$request[] = 'host:www.amazon.com';
$request[] = 'Connection: keep-alive';
$request[] = 'Pragma: no-cache';
$request[] = 'Cache-Control: no-cache';
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8';
$request[] = 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:98.0) Gecko/20100101 Firefox/98.0';//Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36';
$request[] = 'DNT: 1';
$request[] = 'Accept-Encoding: gzip, deflate';
$request[] = 'Accept-Language: en-US,en;q=0.8';
$url = 'https://www.amazon.de/Wenn-Dunkeln-Sterne-funkeln-Puste-Licht-Buch/dp/3480236529/ref=sr_1_3?keywords=buch&qid=1670662644&sr=8-3';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
$output = curl_exec($ch);
EDIT:
I slightly modified the code (random user agent string and multiple cURL requests in a loop) but the problems are the same: on localhost no problems on the webserver I get the captchas).
$user_agents = array('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.67 Safari/537.36', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:89.0) Gecko/20100101 Firefox/89.0', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (K HTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36', 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0', 'Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0', 'Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0');
foreach ($products as $key => $value) {
$request = array();
$request[] = 'Connection: keep-alive';
$request[] = 'Pragma: no-cache';
$request[] = 'Cache-Control: no-cache';
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8';
$request[] = 'User-Agent: ' . $user_agents[array_rand($user_agents)];
$request[] = 'DNT: 1';
$request[] = 'Accept-Encoding: gzip, deflate';
$request[] = 'Accept-Language: en-US,en;q=0.8';
$url = $value['url'];
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
$output = curl_exec($ch);
...
}
Since it only gets triggered when you're on the server, the captcha probably tracks IP addresses. Any chance it's a Recaptcha?
Whatever the captcha is, one thing that could help is solving the captcha from the webserver's IP address.
If the webserver has a desktop environment, connect via VNC (or whatever you ususally use for connecting), open a browser and solve the captcha.
If it does not, try setting up a VPN server on the webserver (this one seems easy enough), connect to the VPN from your computer (and thus get the same IP address as your webserver), open a browser and solve the captcha.
Another option is creating a proxy server which will achieve similar result to VPN.
Sadly you'll have to do it from time to time because that's exactly what captcha is for - preventing automated scrapping of websites via bots.
To fix this, you can try to include additional headers or cookies in your cURL request to make it appear more like a real user. For example, you could include the User-Agent header to specify the browser and operating system that your cURL request is coming from, and also you could include the Cookie header to include cookies that are typically sent by a real user.
For example:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Include additional headers
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36',
'Cookie: __cfduid=<cookie-data-goes-here>'
));
$response = curl_exec($ch);
I tried to fetch some data using php curl_setopt in my code. But it went through 2 minutes of loading, then got 504 Gateway Timeout error. Here's the code:
function sendRequest($url, $data, $token = '') {
$header[] = "Connection: keep-alive";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HTTP_VERSION, 'CURL_HTTP_VERSION_1_1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36");
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
Strangely, I succeeded fetching the data instantly using command line cURL with the same options:
curl --url "http://x.x.x.x" --header "Connection: keep-alive" --http1.1 --verbose --request "POST" -d "data1=value1" -d "data2=value2" --user-agent "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
The server that hosts the code and the destination is in the same network and firewall is not activated in both servers. What could be the cause of the failure?
No clue what $data is. If it's an associative array, try
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));
why php curl blocked but not blocked if I use Fiddler proxy (there are no special setup), it is because carriage return \n vs \r\n or curl certificate?
without fiddler I Got response
<title>Please Wait... | Cloudflare</title>
and with Fiddle proxy enabled
<title>Subscene - Passionate about good subtitles</title>
and the script
<?php
$ch = curl_init();
//curl_setopt($ch, CURLOPT_PROXY, '127.0.0.1:8888'); // uncomment to enable fiddler proxy
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_URL, "https://subscene.com/");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$html = curl_exec($ch);
curl_close($ch);
header("Content-Type: text/plain");
echo $html;
I'm trying to make this to work, but it only works if I remove the two lines that are commented out. If I enable it stops working with a Error 404 (Not Found). Why?
Thanks in advance.
$headers = [
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
'Accept-Encoding: gzip, deflate, br',
'Accept-Language: pt-PT,pt;q=0.9,en-US;q=0.8,en;q=0.7',
'Cache-Control: no-cache',
'Content-Type: application/x-www-form-urlencoded; charset=utf-8',
'Host: httpbin.org',
'Referer: http://www.google.com',
'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
'X-MicrosoftAjax: Delta=true',
'X-Amzn-Trace-Id: Root=1-60e1dc05-44eb1f0a7cff152139d79c76'
];
$url = $_POST['URL'];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
//curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36');
curl_setopt($ch, CURLOPT_REFERER, 'http://www.google.com');
//curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$get = curl_exec($ch);
$url = "http://localhost/dss/dss.asmx/AreaCategory_list";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36');
$xmlstr = curl_exec($ch);
curl_close($ch);
var_dump( $xmlstr);
cannot show data
$json2 = "http://localhost/dss/dss.asmx/AreaCategory_list";
$array = json_decode($json2);
echo $array->Remark;