PHP Curl return 403 - php

I have an url need to get using CURL PHP. When I tried to setup PHP curl as below, I got 403 error
$url = "http://r3---sn-ogueln76.googlevideo.com/videoplayback?initcwndbps=22007500&signature=111AEFD82398870EA1643F228D98E7D5834FB654.1CA4909304F857122728B00FC24CD06C751FC181&dur=3812.809&nh=IgpwcjAzLm5ydDE5KgkxMjcuMC4wLjE&sver=3&sparams=dur,id,initcwndbps,ip,ipbits,itag,lmt,mime,mm,mn,ms,mv,nh,pl,ratebypass,source,upn,expire&lmt=1460861599174672&mime=video/mp4&ip=45.32.11.179&mm=31&source=youtube&pl=25&mn=sn-ogueln76&mt=1461253133&mv=m&id=o-AHFL53t2W9hKDUR3w8WBd5DOA-1Joopi7CtioNWE1jG4&ms=au&expire=1461274970&key=yt6&upn=zmxYuJKDITA&ratebypass=yes&ipbits=0&itag=18&fexp=9405185,9408350,9416126,9416891,9420452,9422542,9422596,9425077,9426927,9427773,9428398,9430014,9430031,9431012,9431522,9433094,9433097,9433653,9434126,9434765&signature=";
$xx = curl_init();
$head[] = "Connection: keep-alive";
$head[] = "Keep-Alive: 300";
$head[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$head[] = "Accept-Language: en-us,en;q=0.5";
curl_setopt($xx, CURLOPT_URL, "$url");
curl_setopt($xx, CURLOPT_BINARYTRANSFER, true);
curl_setopt($xx, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36');
curl_setopt($xx, CURLOPT_HEADER, true);
curl_setopt($xx, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($xx, CURLOPT_TIMEOUT, 60);
curl_setopt($xx, CURLOPT_CONNECTTIMEOUT, 60);
curl_setopt($xx, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($xx, CURLOPT_HTTPHEADER, $head);
curl_setopt($xx, CURLOPT_VERBOSE, true);
$verbose = fopen('tmp.log', 'w+');
curl_setopt($xx, CURLOPT_STDERR, $verbose);
var_dump(curl_exec($xx));
Here is the error:
string(223) "HTTP/1.1 403 Forbidden Last-Modified: Wed, 02 May 2007 10:26:10 GMT Content-Type: text/plain Content-Length: 0 Connection: close X-Content-Type-Options: nosniff Date: Thu, 21 Apr 2016 16:15:25 GMT Server: gvs 1.0 "
But when I try to request same url which CURL shell (command prompt). I got 302 code
I am suppesting that php curl encoded my request so that the server don't allow and return 403.
If you have any idea, please help. Thank you very much.

Related

Soap Curl request error - 400 Bad Request

I am making soap curl request and here is the code:
$headers = array(
"Content-type: text/xml;charset=\"utf-8\"",
"Accept: text/xml",
"Cache-Control: no-cache",
"Pragma: no-cache",
"SOAPAction: \"http://tempuri.org/Save_ExMsgInfo2\"",
"Content-length: ".strlen($xml_post_string),
);
$ch= curl_init();
curl_setopt($ch, CURLOPT_URL, "https://xxx.yyy.com:517/zzz.asmx" );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, $xml_post_string); // the SOAP request
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
if(curl_exec($ch) === false)
{
$err = 'Curl error: ' . curl_error($ch);
curl_close($soap_do);
return $err;
}
else
{
curl_close($ch);
return 'Operation completed without any errors';
}
But getting below error
HTTP/1.1 400 Bad Request
Cache-Control: private
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Wed, 01 Jul 2020 16:42:35 GMT
Content-Length: 0
Any soln?

Curl spoofing url impossible after some retriction

I was scraping redirection result with curl spoofing. After almost a year getting what i want without problem, suddenly it stopped working. On browser, its working perfectly, but with curl I only get a url with 404 error on another server. I am getting redirected to the wrong url. Here is my code.
function curl_spoofred($url)
{
$curl = curl_init();
//set some headers if you want
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
$header[] = "Cache-Control: max-age=0";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: 300";
curl_setopt($curl, CURLOPT_URL, $url);
//Spoof the agent
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36');
//Spoof the Referer
curl_setopt($curl, CURLOPT_REFERER, 'https://wwv.example.com');
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_COOKIESESSION, true);
curl_setopt($curl, CURLOPT_FAILONERROR, true);
curl_setopt($curl, CURLOPT_ENCODING, 'gzip,deflate,br');
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
if (!$html = curl_exec($curl))
{
}
return curl_getinfo($curl,CURLINFO_EFFECTIVE_URL );
curl_close($curl);
}
After checking the network console in Chrome, it gave me this :
Request URL: https://www.example.com/video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
Request Method: GET
Status Code: 301
Remote Address: 104.26.5.130:443
Referrer Policy: no-referrer-when-downgrade
cache-control: max-age=3600
cf-ray: 51b45707fb1969aa-CDG
date: Tue, 24 Sep 2019 11:15:20 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expires: Tue, 24 Sep 2019 12:15:20 GMT
location: https://www.example.com/video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
server: cloudflare
status: 301
vary: Accept-Encoding
:authority: www.example.com
:method: GET
:path: /video.php?p=2&c=V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09&id=631
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7,ar;q=0.6
cookie: __cfduid=d948bdac17ae9dca8577dffc6dc3509cd1565743125; _ga=GA1.2.1798570479.1565743125; HstCfa2982759=1565743132096; __dtsu=3DD172A73239535D5B772D7602A61C9A; HstCmu2982759=1568475393523; HstCla2982759=1568892578738; HstPn2982759=1; HstPt2982759=96; HstCnv2982759=18; HstCns2982759=34; _gid=GA1.2.1495236200.1569313316; _gat_gtag_UA_138212094_1=1
referer: https://wwv.example.com/some-article.htm
sec-fetch-mode: nested-navigate
sec-fetch-site: same-site
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36
p: 2
c: V1RKa2RHTlRhXbTFhUbmtvMVRWYzRQUT09
id: 631
I hope you can help me with this. Regards.

Cookie issue in CURL (PHP) - Cookie information not attached in CURL headers

I have this code
<?php
$mLoginUrl = "https://www.test.com/login";
$mCookieFile = dirname(__FILE__).'/tmpCookies/cookie'.rand().'.txt';
define('USER_AGENT', 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36');
define('COOKIE_FILE', $mCookieFile);
define('LOGIN_FORM_URL', $mLoginUrl);
define('LOGIN_ACTION_URL', $mLoginUrl);
$postValues = array(
'user[email]' => "mymail#email.com",
'user[password]' => "mypassword"
);
$headers = Array(
"Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
"Cache-Control: max-age=0",
"Connection: keep-alive",
"Keep-Alive: 300",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Accept-Language: en-us,en;q=0.5",
"Pragma: "
);
$curl = curl_init();
curl_setopt($curl, CURLOPT_VERBOSE, TRUE);
curl_setopt($curl, CURLOPT_URL, LOGIN_ACTION_URL);
curl_setopt($curl, CURLOPT_POST, TRUE);
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($postValues));
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER,FALSE);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($curl, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt($curl, CURLOPT_COOKIEFILE, COOKIE_FILE);
curl_setopt($curl, CURLOPT_USERAGENT, USER_AGENT);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_REFERER, LOGIN_FORM_URL);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($curl, CURLOPT_HTTPHEADER,$headers) ;
curl_setopt($curl, CURLINFO_HEADER_OUT, TRUE);
$res = curl_exec($curl);
$info = curl_getinfo($curl);
print_r($info['request_header']);
exit;
?>
This works fine on my local computer and one of my servers and shows following output
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36
Host: www.test.com
Referer: https://www.test.com/login
Cookie: _property_session=WjROWEYvTHNYaE5Zb29jVk04WGM0Z3FybmhmY1ZIeVdBc2N6d2N3UmViaXlZdFNhR1dSbUN4QVh6aFFSRjFPYktybmdnRGlXNG0yWWcremEzcklKTnE1ZE1lTTM0eUQrSG90SVhRRzhvYW5rWmFQTVhBMjVCWjBtb1FSc0RrTEh2RjhHSFI3aHkwa3U4N3Y3czJhTzJuN2ZGbWRRN0Nra2Z6OTR4aHhvVG42bVVRS3kwTExUL1hMN2JoZ0xRd2g3VVdIMC81cGhLQzJjOTJvc2RYajIwakE0VjZqRnhTeHBleFltTGF4Z3hpUGJEb0E3Nlo2S3BwMElqNnVkaWhDVS0tc0pCRlozSVE5bXRHQXlHWE1IbTl4UT09--67cf6e056b84b4cae4d275507f544927802eb78d
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Cache-Control: max-age=0
Connection: keep-aliveKeep-Alive: 300 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Language: en-us,en;q=0.5
which means that cookie was created and attached in headers of CURL (as we can see in above header print of CURL.) The cookie file is created as well at mentioned location.
But on one of my server the code do not work as per expectations and gives following output
POST /users/sign_in? HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.2309.372 Safari/537.36 Host: www.test.com Referer: https://www.test.com/login Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5 Cache-Control: max-age=0 Connection: keep-alive Keep-Alive: 300 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Accept-Language: en-us,en;q=0.5 Content-Length: 76 Content-Type: application/x-www-form-urlencoded
Means cookie information is not attached in headers of CURL. I have checked that cookie file is created in this (problematic) server too and is having cookie contents but still cookie contents are not included in CURL headers. Temporary cookie directory mentioned is having full rights/permissions (777) for all users.
PHP version is 5.4.19 and CURL version is 7.19.7 on problematic server.
If anybody can help? I have tried all of the solutions found on internet.
Thanks in advance.

$mCookieFile = dirname(__FILE__).'/tmpCookies/cookie'.rand().'.txt';
remove rand() ... to be a static file

Thanks all for your help. I have solved issue. The issue was open_basedir path. I set this to "none" on server which fixed the problem.

PHP cURL results in 403 Forbidden

I am trying to cURL a remote site that results in a 403 Forbidden error. From the same server I can run the following two commands via Terminal. The first fails while the second works. How can I get my PHP code to match up to the second terminal command?
This Terminal command results in nothing returned.
curl http://www.barneys.com
This Terminal command results in a proper result (the HTML of the page)
curl -L http://www.barneys.com
My PHP Code:
$ch = curl_init('http://www.barneys.com');
$http_headers = array(
'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)' . 'Gecko/20100101 Firefox/40.0',
'Accept: */*',
'X-Requested-With: XMLHttpRequest',
'Referer: http://www.barneys.com', # IMPORTANT
'Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3',
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.barneys.com');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$response = curl_exec($ch);
$redirectURL = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL);
curl_close($ch);
echo $response;
EDIT: Here is the log from the cURL request via PHP:
* About to connect() to www.barneys.com port 80 (#0)
* Trying 23.204.27.110... * connected
* Connected to www.barneys.com (23.204.27.110) port 80 (#0)
> GET / HTTP/1.1
Host: www.barneys.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:40.0)Gecko/20100101 Firefox/40.0
Accept: */*
X-Requested-With: XMLHttpRequest
Referer: http://www.barneys.com
Accept-Language: pt-BR,en-US;q=0.7,en;q=0.3
Connection: keep-alive
< HTTP/1.1 403 Forbidden
< Server: AkamaiGHost
< Mime-Version: 1.0
< Content-Type: text/html
< Content-Length: 265
< Expires: Mon, 11 Apr 2016 23:22:16 GMT
< Date: Mon, 11 Apr 2016 23:22:16 GMT
< Connection: close
<
* Closing connection #0

There is no request header included in your php cURL request. To fix that, add the following line right before your setting the CURLOPT_HTTPHEADER option:
curl_setopt($ch, CURLOPT_HEADER, true);
From the PHP curl docs:
CURLOPT_HEADER TRUE to include the header in the output.
Also, if you add a trailing slash to the url, the URL won't need to be rebuilt by cURL. A lot of your code appears to be unnecessary. Here is a trimmed down working example:
<?php
$ch = curl_init('http://www.barneys.com/');
$http_headers = array(
'User-Agent: Junk', // Any User-Agent will do here
);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $http_headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
echo $response;

PHP cURL refuses to POST to website

I would like to ask for some kind assistance. I've been trying for hours reading different articles and Q&As for a possible cure as to why PHP cURL refuses to POST. But none of the methods I try are giving any results.
The website I am attempting to POST to is "pixiv.net". Here is my current PHP code:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__).'/pixiv-cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__).'/pixiv-cookie.txt');
curl_setopt($ch, CURLOPT_URL, 'http://www.pixiv.net/');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36');
curl_setopt($ch, CURLOPT_REFERER, 'http://www.pixiv.net');
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
echo curl_exec($ch);
if (curl_errno($ch)){
echo 'Curl error: ' . curl_error($ch);
} else {
echo 'Operation completed without any errors<br>'; }
//--------------------------------------------
curl_setopt($ch, CURLOPT_URL, 'https://www.secure.pixiv.net/login.php/');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$httpHeader = array(
'Host: www.pixiv.net',
'Connection: keep-alive',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36',
'DNT: 1',
'Referer: http://www.pixiv.net',
'Accept-Encoding: gzip, deflate, sdch',
'Accept-Language: en-US,en;q=0.8,ja;q=0.6,zh-CN;q=0.4,zh;q=0.2,fr-FR;q=0.2,fr;q=0.2'
);
curl_setopt($ch, CURLOPT_HTTPHEADER, $httpHeader);
curl_setopt($ch, CURLOPT_POST, 1);
$postData='
mode=login
&return_to=%2F
&pixiv_id=#####
&pass=#####
&skip=1
';
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
echo curl_exec($ch);
curl_close($ch);
unset($ch);
?>
On the website, the HTML portion of the login area is as follows:
<form action="https://www.secure.pixiv.net/login.php" method="post" data-time="1429509525" data-text-confirm="...">
<input type="hidden" name="mode" value="login">
<input type="hidden" name="return_to" value="/">
<div class="id"><input type="text" class="js-placeholder" id="login_pixiv_id" name="pixiv_id" value="" maxlength="255" placeholder="pixiv ID or Email"></div>
<div class="pass-signin"><input type="password" class="js-placeholder" id="login_password" name="pass" value="" maxlength="32" placeholder="Password"><input type="submit" id="login_submit" value="Login" class="ui-button"></div>
<div class="save-forgotpass"><label><input name="skip" type="checkbox" checked="" value="1">Remember me</label>
<div>Forgot ID or password?
Secure sign-in through SSL (https)</div>
</div></form>
So far, the GET works fine in saving the cookie from the website. I set the HTTP Headers to be the exact same as my current browser's, as well as some other things to help spoof the connection. I blanked out my login details here, but I assure you that they're correct.
The header returned by the website is:
HTTP/1.1 200 OK
Server: nginx
Date: Mon, 20 Apr 2015 18:59:09 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Host-Time: 113
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
HTTP/1.1 403 Forbidden
Server: nginx
Date: Mon, 20 Apr 2015 19:03:25 GMT
Content-Type: text/html
Content-Length: 564
Connection: keep-alive
Does anybody have an idea as to what I could be doing wrong? If it helps, the website does load (GET) some of their own images, which show as 403's in my browser's console (probably because the website detected that it's not a same-domain request). This PHP page is being run locally on XAMPP, and I'm not getting any errors from cURL.

Update 2
First request
Request header (http://www.pixiv.net/)
GET / HTTP/1.1
Accept-Encoding: deflate, gzip
Host: www.pixiv.net
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Response header (http://www.pixiv.net/)
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 21 Apr 2015 01:12:07 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Host-Time: 113
Set-Cookie: PHPSESSID=c40dbb30394766ec885c61f7e08e00fb; expires=Tue, 21-Apr-2015 02:12:07 GMT; Max-Age=3600; path=/; domain=.pixiv.net
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: p_ab_id=2; expires=Tue, 21-Apr-2020 01:12:07 GMT; Max-Age=157852800; path=/; domain=.pixiv.net
Set-Cookie: p_ab_id=2; expires=Tue, 21-Apr-2020 01:12:07 GMT; Max-Age=157852800; path=/; domain=.pixiv.net
X-Frame-Options: SAMEORIGIN
Content-Encoding: gzip
Second request
Request header (https://www.secure.pixiv.net/login.php)
This has a cookie from previous response.
POST /login.php HTTP/1.1
Accept-Encoding: deflate, gzip
Host: www.secure.pixiv.net
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Cookie: PHPSESSID=c40dbb30394766ec885c61f7e08e00fb; p_ab_id=2
Content-Length: 56
Content-Type: application/x-www-form-urlencoded
Response header (https://www.secure.pixiv.net/login.php)
This redirects to http://www.pixiv.net/ (See Location in headers),
then http://www.pixiv.net/ redirects to http://www.pixiv.net/mypage.php.
It did not alter the cookies so I skipped http://www.pixiv.net/ and
when right to http://www.pixiv.net/mypage.php
HTTP/1.1 302 Found
Server: nginx
Date: Tue, 21 Apr 2015 01:12:08 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Connection: keep-alive
X-Host-Time: 62
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
P3P: CP="THIS IS NOT P3P"
Set-Cookie: PHPSESSID=509719_83607aedd2945280c7879fab036c5fb0; expires=Tue, 21-Apr-2015 02:12:08 GMT; Max-Age=3600; path=/; domain=.pixiv.net
Set-Cookie: device_token=d14a832ce615d42daae571775215d712; expires=Thu, 21-May-2015 01:12:08 GMT; Max-Age=2592000; path=/; domain=.pixiv.net
Location: http://www.pixiv.net/
NOTICE: Last line of Response Header Location: http://www.pixiv.net/
This is the URL the Browser is being redirected to.
but I skipped Location: http://www.pixiv.net/ because it did not set new cookies.
http://www.pixiv.net/ just created another 302 redirect to http://www.pixiv.net/mypage.php
Third request
Request header (http://www.pixiv.net/mypage.php)
GET /mypage.php HTTP/1.1
Accept-Encoding: deflate, gzip
Host: www.pixiv.net
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Cookie: PHPSESSID=509719_83607aedd2945280c7879fab036c5fb0; p_ab_id=2; device_token=d14a832ce615d42daae571775215d712
Response header (http://www.pixiv.net/mypage.php)
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 21 Apr 2015 01:12:08 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Host-Time: 62
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: module_orders_mypage=%5B%7B%22name%22%3A%22everyone_new_illusts%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22spotlight%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22featured_tags%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22contests%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22following_new_illusts%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22mypixiv_new_illusts%22%2C%22visible%22%3Atrue%7D%2C%7B%22name%22%3A%22booth_follow_items%22%2C%22visible%22%3Atrue%7D%5D; expires=Wed, 20-Apr-2016 01:12:08 GMT; Max-Age=31536000; path=/; domain=.pixiv.net
X-Frame-Options: SAMEORIGIN
Content-Encoding: gzip
FINAL PHP CODE
Get new cookies from a 302 redirect. Makes third request with the 302 URL.
$request = array();
$request[] = "Host: www.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$url = 'http://www.pixiv.net/';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
$info = var_export(curl_getinfo($ch),true);
$fp = fopen('pixiv.html','w');
fwrite($fp,"$data\n\n$info");
fclose($fp);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$head = substr($data,0,$skip);
$e = 0;
while(true){
$s = strpos($head,'Set-Cookie: ',$e);
if (!$s){break;}
$s += 12;
$e = strpos($head,';',$s);
$cookie = substr($head,$s,$e-$s) ;
$s = strpos($cookie,'=');
$key = substr($cookie,0,$s);
$value = substr($cookie,$s);
$cookies[$key] = $value;
}
$cookie = '';
$delim = '';
foreach ($cookies as $k => $v){
$cookie .= "$delim$k$v";
$delim = '; ';
}
}
$request = array();
$request[] = "Host: www.secure.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$request[] = "Cookie: $cookie";
$postData = 'mode=login&return_to=%2F&pixiv_id=username&pass=password';
$url = 'https://www.secure.pixiv.net/login.php';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$info = var_export(curl_getinfo($ch),true);
$fp = fopen('pixivLogin.html','w');
fwrite($fp, "\n------------------$data\n\n$info");
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$head = substr($data,0,$skip);
$e = 0;
while(true){
$s = strpos($head,'Set-Cookie: ',$e);
if (!$s){break;}
$s += 12;
$e = strpos($head,';',$s);
$cookie = substr($head,$s,$e-$s) ;
$s = strpos($cookie,'=');
$key = substr($cookie,0,$s);
$value = substr($cookie,$s);
$cookies[$key] = $value;
}
$cookie = '';
$delim = '';
foreach ($cookies as $k => $v){
$cookie .= "$delim$k$v";
$delim = '; ';
}
}
$request = array();
$request[] = "Host: www.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$request[] = "Cookie: $cookie";
$url = 'http://www.pixiv.net/mypage.php';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
$info = var_export(curl_getinfo($ch),true);
$fp = fopen('pixiv2.html','w');
fwrite($fp,"$data\n\n$info");
fclose($fp);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$head = substr($data,0,$skip);
$data = substr($data,$skip);
echo $data;
end of update2
UPDATE
This starts fresh every login by getting a new cookie to use for login each time it is executed.
This goes to http://www.pixiv.net/ and gets the cookie for the login.
Worked well for me.
$request = array();
$request[] = "Host: www.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$url = 'http://www.pixiv.net/';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
Gets the response header:
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$head = substr($data,0,$skip);
Finds all Set-Cookie and all the cookies:
$e = 0;
while(true){
$s = stripos($head,'Set-Cookie: ',$e);
if (!$s){break;}
$s += 12;
$e = strpos($head,';',$s);
$cookie = substr($head,$s,$e-$s) ;
$s = strpos($cookie,'=');
$key = substr($cookie,0,$s);
$value = substr($cookie,$s);
$cookies[$key] = $value;
}
Then convert $cookies array back to a string:
$cookie = '';
$delim = '';
foreach ($cookies as $k => $v){
$cookie .= "$delim$k$v";
$delim = '; ';
}
}
Do Login:
$request = array();
$request[] = "Host: www.secure.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
With cookie from first request:
$request[] = "Cookie: $cookie";
$postData = 'mode=login&return_to=%2F&pixiv_id=username&pass=password';
$url = 'https://www.secure.pixiv.net/login.php';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$head = substr($data,0,$skip);
$data = substr($data,$skip);
echo $data;
end of update
I went directly to:
https://www.secure.pixiv.net/login.php
Using the cookie from http://www.pixiv.net/
For sure postData was wrong:
$postData='
mode=login
&return_to=%2F
&pixiv_id=#####
&pass=#####
&skip=1
';
It add all the spaces and newline characters:
mode=login
&return_to=%2F
&pixiv_id=#####
&pass=#####
do it like this:
$postData = 'mode=login&return_to=%2F&pixiv_id=username&pass=password';
I did not use these:
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_REFERER, 'http://www.pixiv.net');
I added the cookie to the HTTP Header:
$request[] = "Cookie:PHPSESSID=b2e3185bd045475a2174fb28fb642569;
These I use just in case there is trouble and are not needed.
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
This Worked
$request = array();
$request[] = "Host: www.secure.pixiv.net";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$request[] = "Cookie:PHPSESSID=b2e3185bd045475a2174fb28fb642569; p_ab_id=3";
$post = 'mode=login&return_to=%2F&pixiv_id=username&pass=password';
$url = 'https://www.secure.pixiv.net/login.php';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$info = rawurldecode(var_export(curl_getinfo($ch),true));
// Get the cookies:
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$requestHeader= substr($data,0,$skip);
$data = substr($data,$skip);
echo $data;
}

To post make sure you have a proper header set. your variable $httpHeader is defined with all the attributes having the same key '0' and therefore are overwritten each other.

You're sending in a custom Host: header which doesn't match the host name of the URL you're using, which is suspicious and most likely wrong.
I'd advice you to not set a custom Host: header but let curl do its own magic for that, which probably is what you want.

Categories