Curl spoofing url impossible after some retriction - php

I was scraping redirection result with curl spoofing. After almost a year getting what i want without problem, suddenly it stopped working. On browser, its working perfectly, but with curl I only get a url with 404 error on another server. I am getting redirected to the wrong url. Here is my code.
function curl_spoofred($url)
{
$curl = curl_init();
//set some headers if you want
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
curl_setopt($curl, CURLOPT_URL, $url);
//Spoof the agent
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36');
//Spoof the Referer
curl_setopt($curl, CURLOPT_REFERER, 'https://wwv.example.com');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_COOKIESESSION, true);
curl_setopt($curl, CURLOPT_FAILONERROR, true);
curl_setopt($curl, CURLOPT_ENCODING, 'gzip,deflate,br');
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
if (!$html = curl_exec($curl))
{
}
return curl_getinfo($curl,CURLINFO_EFFECTIVE_URL );
curl_close($curl);
}
After checking the network console in Chrome, it gave me this :
Request URL: https://www.example.com/video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
Request Method: GET
Status Code: 301
Remote Address: 104.26.5.130:443
Referrer Policy: no-referrer-when-downgrade
cache-control: max-age=3600
cf-ray: 51b45707fb1969aa-CDG
date: Tue, 24 Sep 2019 11:15:20 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expires: Tue, 24 Sep 2019 12:15:20 GMT
location: https://www.example.com/video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
server: cloudflare
status: 301
vary: Accept-Encoding
:authority: www.example.com
:method: GET
:path: /video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7,ar;q=0.6
cookie: __cfduid=d948bdac17ae9dca8577dffc6dc3509cd1565743125; _ga=GA1.2.1798570479.1565743125; HstCfa2982759=1565743132096; __dtsu=3DD172A73239535D5B772D7602A61C9A; HstCmu2982759=1568475393523; HstCla2982759=1568892578738; HstPn2982759=1; HstPt2982759=96; HstCnv2982759=18; HstCns2982759=34; _gid=GA1.2.1495236200.1569313316; _gat_gtag_UA_138212094_1=1
referer: https://wwv.example.com/some-article.htm
sec-fetch-mode: nested-navigate
sec-fetch-site: same-site
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
p: 2
c: V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09
id: 631
I hope you can help me with this. Regards.

Related

CURL gives HTTP/1.1 302 Moved Temporarily

CURL gives HTTP/1.1 302 Moved Temporarily
$ua = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://...........");
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $ua);
curl_setopt($ch1, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch1, CURLOPT_POST, true);
curl_setopt($ch1, CURLOPT_HTTPHEADER, array('Accept: Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11'));
$result = curl_exec($ch1);
Output:
HTTP/1.1 302 Moved Temporarily Content-Length: 0 Connection: keep-alive Date: Mon, 03 Jul 2017
12:23:55 GMT
Set-Cookie:AWSALB=ZBxF0OO7JHWtHtsbAUVLo1a2Y7GQ1KTxIaMml+1UFTDK2UL7V6Xm8FIU1mWeCDZDtYvQ0emHLPLOtWpli
+6pqFbYe+ulWPA8SBqeM/n5S9x0ScxYTGVsUeDixyuf; Expires=Mon, 10 Jul 2017 12:23:55 GMT; Path=/ Server: nginx/1.8.0 Location:
https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx X-Cache: Miss
from cloudfront Via: 1.1
ea5d711fdce750991b725e09c07704b0.cloudfront.net (CloudFront)
X-Amz-Cf-Id: rVytkGKHVM628Zh3WFSohHlVdb-6XXZX-cXlQkvtRSeDCZN95s9Guw==

Cookie issue in CURL (PHP) - Cookie information not attached in CURL headers

I have this code
<?php
$mLoginUrl = "https://www.test.com/login";
$mCookieFile = dirname(__FILE__).'/tmpCookies/cookie'.rand().'.txt';
define('USER_AGENT', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36');
define('COOKIE_FILE', $mCookieFile);
define('LOGIN_FORM_URL', $mLoginUrl);
define('LOGIN_ACTION_URL', $mLoginUrl);
$postValues = array(
'user[email]' => "mymail#email.com",
'user[password]' => "mypassword"
);
$headers = Array(
"Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
"Cache-Control: max-age=0",
"Connection: keep-alive",
"Keep-Alive: 300",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Accept-Language: en-us,en;q=0.5",
"Pragma: "
);
$curl = curl_init();
curl_setopt($curl, CURLOPT_VERBOSE, TRUE);
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);
curl_setopt($curl, CURLOPT_POST, TRUE);
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($postValues));
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER,FALSE);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt($curl, CURLOPT_COOKIEFILE, COOKIE_FILE);
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_REFERER, LOGIN_FORM_URL);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($curl, CURLOPT_HTTPHEADER,$headers) ;
curl_setopt($curl, CURLINFO_HEADER_OUT, TRUE);
$res = curl_exec($curl);
$info = curl_getinfo($curl);
print_r($info['request_header']);
exit;
?>
This works fine on my local computer and one of my servers and shows following output
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36
Host: www.test.com
Referer: https://www.test.com/login
Cookie: _property_session=WjROWEYvTHNYaE5Zb29jVk04WGM0Z3FybmhmY1ZIeVdBc2N6d2N3UmViaXlZdFNhR1dSbUN4QVh6aFFSRjFPYktybmdnRGlXNG0yWWcremEzcklKTnE1ZE1lTTM0eUQrSG90SVhRRzhvYW5rWmFQTVhBMjVCWjBtb1FSc0RrTEh2RjhHSFI3aHkwa3U4N3Y3czJhTzJuN2ZGbWRRN0Nra2Z6OTR4aHhvVG42bVVRS3kwTExUL1hMN2JoZ0xRd2g3VVdIMC81cGhLQzJjOTJvc2RYajIwakE0VjZqRnhTeHBleFltTGF4Z3hpUGJEb0E3Nlo2S3BwMElqNnVkaWhDVS0tc0pCRlozSVE5bXRHQXlHWE1IbTl4UT09--67cf6e056b84b4cae4d275507f544927802eb78d
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Cache-Control: max-age=0
Connection: keep-aliveKeep-Alive: 300 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Language: en-us,en;q=0.5
which means that cookie was created and attached in headers of CURL (as we can see in above header print of CURL.) The cookie file is created as well at mentioned location.
But on one of my server the code do not work as per expectations and gives following output
POST /users/sign_in? HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36 Host: www.test.com Referer: https://www.test.com/login Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Cache-Control: max-age=0 Connection: keep-alive Keep-Alive: 300 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Language: en-us,en;q=0.5 Content-Length: 76 Content-Type: application/x-www-form-urlencoded
Means cookie information is not attached in headers of CURL. I have checked that cookie file is created in this (problematic) server too and is having cookie contents but still cookie contents are not included in CURL headers. Temporary cookie directory mentioned is having full rights/permissions (777) for all users.
PHP version is 5.4.19 and CURL version is 7.19.7 on problematic server.
If anybody can help? I have tried all of the solutions found on internet.
Thanks in advance.
$mCookieFile = dirname(__FILE__).'/tmpCookies/cookie'.rand().'.txt';
remove rand() ... to be a static file
Thanks all for your help. I have solved issue. The issue was open_basedir path. I set this to "none" on server which fixed the problem.

Curl login not working

I am trying to login to a website in PHP and then store a cookie.However I am not able to get pass the login for some reason.Here is my code so far
First i get the cookie and XSRF-TOKEN .You can consider $cookie_file_path to exist but empty
$ch = curl_init("https://ahrefs.com/user/login");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path );
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_REFERER, "");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT , 120);
curl_setopt($ch, CURLOPT_TIMEOUT , 0);
$result = curl_exec($ch);
At this point my cookie file at location $cookie_file_path contain ahrefs_cookie and XSRF-TOKEN
Now i will send everything via post
$ch = curl_init("https://ahrefs.com/user/login");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36');
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding: gzip, deflate",
"Accept-Language: en-US,en;q=0.8",
"Content-Type: application/x-www-form-urlencoded"
));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($curl_parameters));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path );
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path );
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT , 120);
curl_setopt($ch, CURLOPT_TIMEOUT , 0);
$result = curl_exec($ch);
Here are the post data i am sending
$curl_parameters = array(
"_token" => $token, # I get the token by just downloading the page with curl and extract the token value.This value is refresh on each post
"email" => xxxxx#gmail.com,
"password" => xxxxx,
"return_to" => "/", #
"remember_me" => 1
);
Here is what happen on a successful login
POST /user/login HTTP/1.1
Host: ahrefs.com:443
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Content-Type: application/x-www-form-urlencoded
Cookie: _vwo_uuid_v2=9788FEFE2A8FD2A0894175F5AA899CC9|0edcdb3cdbf14c6dbb9bd958d78f6023; ajs_anonymous_id=%224084d228-28d6-42b7-8129-3faf51983652%22; _vis_opt_s=1%7C; _vis_opt_test_cookie=1; __insp_slim=1453358056412; __insp_wid=88855823; __insp_nv=true; __insp_ref=d; __insp_targlpu=https%3A%2F%2Fahrefs.com%2F; __insp_targlpt=Ahrefs%20SEO%20Tools%3A%20Backlink%20Checker%20%26%20SEO%20Report; km_ai=jRUMYEFwlZb6sEf5ikr900Rt24E%3D; km_uq=; __insp_norec_sess=true; _vwo_uuid=9788FEFE2A8FD2A0894175F5AA899CC9; _vis_opt_exp_12_combi=5; PHPSESSID=mf02jth4fd9nnj626p7i4lqs11; XSRF-TOKEN=eyJpdiI6Iit1MWxXUmZPZ0lpQkx3b1pObUtDU2c9PSIsInZhbHVlIjoiWGtuSW1sRXRBTWpKY2FIaExBXC8wVzIxZFNOalNWRTI2WXBWdkxLbG5cL3k4VWZqVWllOVJPNHpwZGZcL0xMdFVDOE1WXC8xZUplMkk1SzNqbXFSbXRzZUNnPT0iLCJtYWMiOiIzZDVkMDQzODNiMDAwNzU0NGRkOWM2Y2Y4YmFiNTkxZmFmN2QxNmY2ZWE1ZjkxOTBiOGE0YjhjNTVkNGRkNzE2In0%3D; ahrefs_cookie=eyJpdiI6IkZjb2NzY0U2Q3d0d1wvQ0tKZTBuN1JnPT0iLCJ2YWx1ZSI6IjBEWDBSZ1dFbVNLMlF4RTJhK1lmZ1Q2MFllOG9sUERXTGtPTmFOXC9RQk1YSUdiaG5lVGRmZk1mOG1KUWJMdWxzSEQ1elJBdUVVMk9MalhpcmcrVVcwdz09IiwibWFjIjoiNDM0YzRlM2ZhZWFmMGNiYjA2MDdmMmI2N2E2ZGUyMzRjOGE2OGJkNDFkYWZhODdiYWJjNGRkNGQwNTNjY2E2ZCJ9; _gat=1; ajs_user_id=null; ajs_group_id=null; mp_462869d58108d4904e778d9b2b8fbead_mixpanel=%7B%22distinct_id%22%3A%20%2215262e404c33be-09e7d85a5-424f072e-384000-15262e404c48f2%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; kvcd=1453360373661; km_vs=1; km_lv=1453360374; wooTracker=RMJYZZqFbCxy; intercom-id=05c5d5fa-ccaa-4744-8c69-4fdb89bfa409; _ga=GA1.2.274586625.1453358055; _gat_elevioTracker=1; _gali=login_form
Origin: https://ahrefs.com
Referer: https://ahrefs.com/user/login
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36
X-DevTools-Emulate-Network-Conditions-Client-Id: 4E0199DA-8C20-44C9-BAAA-5B5AD8C56232
HTTP/1.1 302 Found
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 316
Content-Type: text/html; charset=UTF-8
Date: Thu, 21 Jan 2016 07:13:48 GMT
Location: https://ahrefs.com
Server: nginx/1.6.2
Set-Cookie: remember_82e5d2c56bdd0811318f0cf078b78bfc=eyJpdiI6IktHdUtuXC9peTJtTXRnMitmWldnVm5RPT0iLCJ2YWx1ZSI6ImxybTExSHZ5M2NIVXBpXC8wV2lRNmhtS2VKdm02VzhCUWtcL1gyb3FSb2hsVGZvcTAxclwvOUJzcTVjVjd1QjR0OGRpYjZCSEVwdDlCWUVmS05mOFBpRTR2SGNaYWl6Q2NCSWJGRlBwbndHVTgwPSIsIm1hYyI6ImZhMGYzZGM2ZjVkOWI3NDRmYTYzY2Q1OWI5OWRlOWUwZWI4OTU2NzQyYTlmNWM3YjlkZDA1M2FkYzU4OTJiNjEifQ%3D%3D; expires=Tue, 19-Jan-2021 07:13:47 GMT; Max-Age=157679999; path=/; domain=.ahrefs.com; httponly
Set-Cookie: XSRF-TOKEN=eyJpdiI6IkxVbkVZUVlZTGUyWGE5R0ZFSVwveTd3PT0iLCJ2YWx1ZSI6Ilk3VlRFQ1dvaTFZS1huVFIwZ2k4bXZTSEtRSzFNVlh0QkFkQTNIZnBRc011akJpTlVJblFVVVlGcXBNcExRTThpUmkyTHo5aVpjQzhxeWhROGQ5RzlBPT0iLCJtYWMiOiI5ZTE0ZDUyMjdmMjYwZDY5NmI4YzBiNmQyOTdkNWRhZWNiYTFhYzhmZWMxMzBiZTUwODEyZWJkNDc4ZmZiMGNkIn0%3D; expires=Thu, 21-Jan-2016 09:13:48 GMT; Max-Age=7200; path=/; domain=.ahrefs.com
Set-Cookie: ahrefs_cookie=eyJpdiI6IjFSU2d2Wm9KRjFycHJkaktUazM3dHc9PSIsInZhbHVlIjoiXC9xd0I1TWNvMTl4UHN5UnRNUmxYd014dnp6TXNYbnlDQ1hyRTdXTFVUeWZYM1VWWlVHbTNcL0tNa3NlN3lVQmgzTTNrOVdIckVHV0JQNmhielVLMEpmUT09IiwibWFjIjoiMzZkNDU2OWRjMmM4NTMzYjAyNWZhNWFkMDgzYjc4ZWYwOTQ3M2E3Mzg5YzRlNjJhYzk0ZDI5YzE0ZjNkOTIwZiJ9; expires=Mon, 21-Mar-2016 07:13:48 GMT; Max-Age=5184000; path=/; domain=.ahrefs.com; httponly
X-Powered-By: PHP/5.6.14-0+deb8u1
What i am missing here , why i can't login via post ?
Edit
If i set a wrong username/password you get the following
POST /user/login HTTP/1.1
Host: ahrefs.com:443
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Content-Type: application/x-www-form-urlencoded
Cookie: ajs_anonymous_id=%22e915c496-7e0b-46fc-8393-ec3c135edeac%22; _vis_opt_s=1%7C; km_ai=8tTicdoOSGYV7D%2F0bUXMtSQ%2BUhc%3D; _vwo_uuid=2E9A1BBEAC643740726E45FEDA088BB0; _vis_opt_exp_12_combi=3; PHPSESSID=mf02jth4fd9nnj626p7i4lqs11; remember_82e5d2c56bdd0811318f0cf078b78bfc=82e5d2c56bdd0811318f0cf078b78bfc=eyJpdiI6IktHdUtuXC9peTJtTXRnMitmWldnVm5RPT0iLCJ2YWx1ZSI6ImxybTExSHZ5M2NIVXBpXC8wV2lRNmhtS2VKdm02VzhCUWtcL1gyb3FSb2hsVGZvcTAxclwvOUJzcTVjVjd1QjR0OGRpYjZCSEVwdDlCWUVmS05mOFBpRTR2SGNaYWl6Q2NCSWJGRlBwbndHVTgwPSIsIm1hYyI6ImZhMGYzZGM2ZjVkOWI3NDRmYTYzY2Q1OWI5OWRlOWUwZWI4OTU2NzQyYTlmNWM3YjlkZDA1M2FkYzU4OTJiNjEifQ%3D%3D; _vwo_uuid_v2=2E9A1BBEAC643740726E45FEDA088BB0|b8caeecf523e81382e01b6691b0f508b; _vis_opt_test_cookie=1; _gat=1; __insp_slim=1453365450618; __insp_wid=88855823; __insp_nv=true; __insp_ref=d; __insp_targlpu=https%3A%2F%2Fahrefs.com%2F; __insp_targlpt=Ahrefs%20SEO%20Tools%3A%20Backlink%20Checker%20%26%20SEO%20Report; km_lv=x; km_uq=; __insp_norec_sess=true; _gat_elevioTracker=1; XSRF-TOKEN=eyJpdiI6ImZ3TGkraHJOVmRQMkhKb3NoZHdYNUE9PSIsInZhbHVlIjoiTThcL2hnY2VGZytGNTZsTXpseWVXb1lvMjVveEpLbFZJUzQ4cHRuNUFWWnpwVVpBSTZKVFdlKzZZN3hLdlpmMHdONGd2cGluTk91RkQ2Q0VJckt1ZWtnPT0iLCJtYWMiOiJjNjE5NWJmOTMyYzI4OWE4ZGFmZThiYjc1MDllYzVkMTBjMDUxNDg4MzQ4YmE5ZjMyOTRhNWM0YTk3ZjM4YmY1In0%3D; ahrefs_cookie=eyJpdiI6IndOUE1BZ1RDOW9VcE1uSU8wRmJwb1E9PSIsInZhbHVlIjoiWTZiaTdETG1ZVE8zWjdDNVo5bEc0NmhOU0VjNHFoZTdpTVVNdVhWMEtESUd6ZHQyTjhyY1JXWGdINkFQQmZ6WnlWb2hucUJqSnFoNmZpYjVtZ0txSWc9PSIsIm1hYyI6IjZlNzY5N2VlNzY0NzY1NGU4OWE5YzVlYTE4MmYzYTI4NmEwZDQwNjBjZDBkZGM3MGRjYjRkOGViOGRhYWM5YjYifQ%3D%3D; ajs_user_id=null; ajs_group_id=null; wooTracker=ZRa7vf7TNNp5; mp_462869d58108d4904e778d9b2b8fbead_mixpanel=%7B%22distinct_id%22%3A%20%22152630d4df88d7-087e038c3-424f072e-384000-152630d4df98b5%22%2C%22%24initial_referrer%22%3A%20%22%24direct%22%2C%22%24initial_referring_domain%22%3A%20%22%24direct%22%7D; kvcd=1453365507959; km_vs=1; _ga=GA1.2.959678603.1453360761; intercom-id=08650ca0-7181-47ec-8ecb-0a6b3c045f19; _gali=login_form
Origin: https://ahrefs.com
Referer: https://ahrefs.com/user/login?return_to=%2F
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36
HTTP/1.1 302 Found
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 416
Content-Type: text/html; charset=UTF-8
Date: Thu, 21 Jan 2016 08:39:16 GMT
Location: https://ahrefs.com/user/login?return_to=%2F
Server: nginx/1.6.2
Set-Cookie: XSRF-TOKEN=eyJpdiI6IkRLcHlIS1Z3YTV3RWtrZG1EUklXS1E9PSIsInZhbHVlIjoiZU1MaDJYSmp4K2VhaTRoMHp2eklJcWNSQjRONGtNcFA3ek5ubmhpXC9SNUlLVG0yVEZxMGJBUHMxTlBCbW9RM3RweEFGUENzbzFXXC9wRFZtQmlHQlBZdz09IiwibWFjIjoiMDRkMzNkZDQwMGU5MGJkZDJmOTk4MTgxMjA1NWViYzZhMmFhY2QyMjU0NjJjZmE2ODk1YjA3OTI1MmEzOTQ3ZSJ9; expires=Thu, 21-Jan-2016 10:39:16 GMT; Max-Age=7200; path=/; domain=.ahrefs.com
Set-Cookie: ahrefs_cookie=eyJpdiI6ImZIdEJGUHJSUmo3V2liZjhWaFdwM1E9PSIsInZhbHVlIjoiM3pyYkdSMDUxa3VZSXRScXZOb3VTbnBodndkS280SkJ6NzJ0Zm94UHo1ZHF0RVA3MVBKMThyOFJaWTdHTzdRajFycjREWE1rMkhPN1J6Ynl2ajJcL2FBPT0iLCJtYWMiOiIyZmFlMTdiOGFkNjg5YWQ0YTlmMDU3OGZhMDg5ODljNmE5NWIyOWY0NTNhN2FhZDJjYzVjNTViODY1MGYxMGYyIn0%3D; expires=Mon, 21-Mar-2016 08:39:16 GMT; Max-Age=5184000; path=/; domain=.ahrefs.com; httponly
X-Powered-By: PHP/5.6.14-0+deb8u1
Regards

php curl emulate a browser?

we are have next headers when we send post:
POST http://www.autonavigator.ru/dispatcher.pl HTTP/1.1
Host: www.autonavigator.ru
Connection: keep-alive
Content-Length: 55
Origin: http://www.autonavigator.ru
X-Request: JSON
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36
Content-type: application/x-www-form-urlencoded; charset=UTF-8
Accept: application/json
X-Requested-With: XMLHttpRequest
Referer: http://www.autonavigator.ru/my/offer_add/
Accept-Encoding: gzip,deflate
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: region_id=45; city_id=22; user_name=rora%40gmail.com; user_type=user; user_offer_count=1; user_message_count=0; user_no_confirm=1; session_id=WR9q4d41DgD7biTOOsMzgtXfJm83VFQn; USession=WR9q4d41DgD7biTOOsMzgtXfJm83VFQn; _ym_visorc_5781676=b
class=list&method=make&show_all=1&vehicle=car&type=used
I would like emulate browser with curl.
For this i use next code:
$ch = curl_init('http://www.autonavigator.ru/dispatcher.pl');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36");
$headers = array
(
'Accept: application/json',
'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
'Accept-Encoding: gzip,deflate',
'Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7'
);
curl_setopt($ch, CURLOPT_HTTPHEADER,$headers);
curl_setopt($ch, CURLOPT_REFERER, "http://www.autonavigator.ru/my/offer_add/");
curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=make&show_all=1&vehicle=car&type=used');
$result = curl_exec($ch);
curl_close($ch);
var_dump($result);
But in result we get some errors(http://i.stack.imgur.com/zWkdP.png):
Tell me please where error in code an how will be right ?
Most likely like the content is gzipped, so you just need to do:
curl_setopt($ch,CURLOPT_ENCODING , "gzip");

Simulating a POST with PHP & cURL

I'm trying to simulate a POST to a website based on what I see coming from Live HTTP headers in Firefox.
Here's a copy/paste of the log from the Firefox plugin:
POST /context?tab=login HTTP/1.1
Host: website User-Agent:
Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.9.2.13) Gecko/20101206
Ubuntu/10.10 (maverick)
Firefox/3.6.13 Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset:
ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115 Connection:
keep-alive Referer: referer
Cookie: fontsize=2;
JSESSIONID=0000pXE_BK7TjZFzEyNHqOKzXz2:-1
Content-Type:
application/x-www-form-urlencoded
Content-Length: 46
loginid=password&password=password&login=Login
And the response that follows immediately after the POST:
HTTP/1.1 302 Found Location:
website/context?tab=p00689
Content-Language: en-US
Set-Cookie:
JSESSIONID=0000oaKlIeeDRWkX5YCiJu5v1lM:-1;
Path=/ Transfer-Encoding:
chunked Date: Mon, 07 Feb 2011
14:15:21 GMT Server: WebSphere
Application Server/6.1 Expires:
Thu, 01 Dec 1994 16:00:00 GMT
Cache-Control: no-cache="set-cookie,
set-cookie2"
Based on my testing, a response that redirects to
website/context?tab=p00689
Means that the user was authenticated and everything worked properly.
However, when trying to accomplish this via PHP & cURL, I'm being redirected to a page that informs the user that their session has timed out.
Here's the code:
// Provider only likes Firefox
$agent = "User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13";
ini_set("user_agent", $agent);
// Cookie
$cookie = tempnam("/tmp", "curlcookie");
// Post everything that was posted to me.
$fields = $_POST;
foreach($fields as $key=>$value)
{
$fields_string .= "$key=$value&";
}
$fields_string = substr($fields_string, 0, strlen($fields_string) - 1);
// Custom headers
$headers = array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-us,en;q=0.5",
"Accept-Encoding: gzip,deflate",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Keep-Alive: 115",
"Connection: keep-alive");
// cURL options
$ch = curl_init("website");
curl_setopt($ch, CURLOPT_REFERER, "referer");
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields_string);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_MAXREDIRS, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$output = curl_exec($ch);
$header = curl_getinfo($ch);
curl_close($ch);
// Debugging junk
echo nl2br($header["request_header"]);
echo "<br/><br/>Output:<br/><br/>$output";
The output from that script is as follows:
POST /context?tab=login HTTP/1.1
User-Agent: User-Agent: Mozilla/5.0
(X11; U; Linux i686; en-US;
rv:1.9.2.13) Gecko/20101206
Ubuntu/10.10 (maverick)
Firefox/3.6.13 Host: website
Pragma: no-cache Referer:
referer Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset:
ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115 Connection:
keep-alive Content-Length:
46 Content-Type:
application/x-www-form-urlencoded
loginid=username&password=password&login=Login
Output: HTTP/1.1 302
Found
Location:website/context?tab=p00697
Content-Language: en-US Set-Cookie:
JSESSIONID=0000Tl8NL1Hg2dbNv_PEnq-bbvr:-1;
Path=/ Set-Cookie:
JSESSIONID=0000Zue58y1tXg3tt4XjB8exXw6:-1;
Path=/ Transfer-Encoding: chunked
Date: Mon, 07 Feb 2011 19:18:20 GMT
Server: WebSphere Application
Server/6.1 Expires: Thu, 01 Dec 1994
16:00:00 GMT Cache-Control:
no-cache="set-cookie,
set-cookie2"
Based on what I've posted, is there anything obvious that I'm missing? What should I try next? The requests look semantically the same; I'm not sure what I could be doing incorrectly.
The one thing that stands out is the following line of code:
$cookie = tempnam("/tmp", "curlcookie");
Now if this fails to create the file then tempnam would return false, meaning that the following lines of code:
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
are as good as not being set at all, you should keep the cookie file within the same directory as the executing script.
the next thing is:
$fields = $_POST;
foreach($fields as $key=>$value)
{
$fields_string .= "$key=$value&";
}
$fields_string = substr($fields_string, 0, strlen($fields_string) - 1);
You do not need to do this as CURLOPT_POSTFIELDS accepts an array so you should be able to do:
curl_setopt($ch, CURLOPT_POSTFIELDS, $_POST);
This will make sure that the entities are correctly parsed.
I also think you can remove the ini_set as that's for native functions such as file_get_contents and fopen streams etc, so double check the line:
ini_set("user_agent", $agent);
Also I would check to see if there is a cookie already set from the main page, such as index.php as the site may block requests from sources that have come directly to the login page with data.

Categories