php get html after user is logged with curl - php

Question

I want to get get the contents from a webpage but the user sees different information depending if he is logged or not. I want to send header information with curl to simulate that the use is logged in.
I inspected the network and these are the response headers:
Cache-Control:no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Connection:close
Content-Type:text/html
Date:Tue, 13 Jun 2017 08:08:52 GMT
Expires:Thu, 19 Nov 1981 08:52:00 GMT
Location:http://dims-92.com/ClientNewsPage
Pragma:no-cache
Server:Apache/2.2.3 (CentOS)
Transfer-Encoding:chunked
X-Powered-By:PHP/5.5.30
And there is this request payload:
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="SubmitControlId"
Auto_CAuthenticate_LogIn_LogIn_Standart
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="ParameterInfo"
undefined
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CEShop_SearchControl_SearchInput"
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_UsernameInput"
user
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_PasswordInput"
password
------WebKitFormBoundaryaSWkHLJeD9EymCJb--
I have tried this:
$url = "http://dims-92.com/ClientDisplayProductFolder?param=4553686f703a434e493d3935343b434e494c3d3b5649443d3b543d42473b";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaSWkHLJeD9EymCJb',
'Content-Length: 671',
'------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="SubmitControlId"
Auto_CAuthenticate_LogIn_LogIn_Standart
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="ParameterInfo"
undefined
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CEShop_SearchControl_SearchInput"
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_UsernameInput"
user
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_PasswordInput"
password
------WebKitFormBoundaryaSWkHLJeD9EymCJb--'
));
$content = curl_exec($ch);
echo $content;
but the page I see just says: Bad Request
Your browser sent a request that this server could not understand.
Request header field is missing ':' separator.
------WebKitFormBoundaryaSWkHLJeD9EymCJb

Answer 1

You can't post headers like this, they have to be in an array like so :
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: multipart/form-data',
'Content-Length: 671',
'Content-Disposition: form-data',
....
));
The problem is that you post the entire payload (things like boundary=----WebKitFormBoundaryaSWkHLJeD9EymCJb'), which is invalid.

Answer 2

your code is confusing the requests' HTTP HEADERS and the HTTP BODY
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Length: 671
these are parts of the HTTP REQUEST HEADERS, and indeed goes into the CURLOPT_HTTPHEADER.
Content-Disposition: form-data; name="SubmitControlId"
Auto_CAuthenticate_LogIn_LogIn_Standart
------WebKitFormBoundaryaSWkHLJeD9EymCJb
Content-Disposition: form-data; name="ParameterInfo"
undefined
this is part of the HTTP REQUEST BODY, the body does not go in CURLOPT_HTTPHEADER.
now, contrary to what Julien Lachal says in https://stackoverflow.com/a/44517070/1067003 , you can actually encode the entire request body yourself (using CURLOPT_POST or CURLOPT_INFILE), but when using multipart/form-data or application/x-www-form-urlencoded encoding, its easier, safer, and less error prone, to let curl encode it for you. (the usual reason for encoding it yourself, is when POSTing to a JSON API requiring content-type: application/json, curl does not support encoding to JSON automatically.)
to tell curl to do it for you, simply use CURLOPT_POST and CURLOPT_POSTFIELDS, like this:
curl_setopt_array ( $ch, array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => array (
'SubmitControlId' => 'Auto_CAuthenticate_LogIn_LogIn_Standart',
'ParameterInfo' => 'undefined',
'FC_CEShop_SearchControl_SearchInput' => '',
'FC_CAuthenticate_LogIn_UsernameInput' => 'user',
'FC_CAuthenticate_LogIn_PasswordInput' => 'password'
)
) );
now libcurl will automatically multipart/form-data-encode it, and set the correct content-type, and set the correct content-length header, and the actual HTTP request will look like:
Http Request Headers:
POST / HTTP/1.1
Host: 127.0.0.1:8080
Accept: */*
Content-Length: 686
Expect: 100-continue
Content-Type: multipart/form-data; boundary=------------------------b6890d3827808ee1
Http Request Body:
--------------------------b6890d3827808ee1
Content-Disposition: form-data; name="SubmitControlId"
Auto_CAuthenticate_LogIn_LogIn_Standart
--------------------------b6890d3827808ee1
Content-Disposition: form-data; name="ParameterInfo"
undefined
--------------------------b6890d3827808ee1
Content-Disposition: form-data; name="FC_CEShop_SearchControl_SearchInput"
--------------------------b6890d3827808ee1
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_UsernameInput"
user
--------------------------b6890d3827808ee1
Content-Disposition: form-data; name="FC_CAuthenticate_LogIn_PasswordInput"
password
--------------------------b6890d3827808ee1--
however, note that many websites doesn't support multipart/form-data and/or prefer application/x-www-form-urlencoded encoding. to use that, use http_build_query on the data to CURLOPT_POSTFIELDS, like this:
curl_setopt_array ( $ch, array (
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => http_build_query ( array (
'SubmitControlId' => 'Auto_CAuthenticate_LogIn_LogIn_Standart',
'ParameterInfo' => 'undefined',
'FC_CEShop_SearchControl_SearchInput' => '',
'FC_CAuthenticate_LogIn_UsernameInput' => 'user',
'FC_CAuthenticate_LogIn_PasswordInput' => 'password'
) ),
CURLOPT_URL => 'http://127.0.0.1:8080'
) );
now the actual HTTP request looks like this:
HTTP request headers:
POST / HTTP/1.1
Host: 127.0.0.1:8080
Accept: */*
Content-Length: 204
Content-Type: application/x-www-form-urlencoded
HTTP request body:
SubmitControlId=Auto_CAuthenticate_LogIn_LogIn_Standart&ParameterInfo=undefined&FC_CEShop_SearchControl_SearchInput=&FC_CAuthenticate_LogIn_UsernameInput=user&FC_CAuthenticate_LogIn_PasswordInput=password