Fetch JSON via REST API using PHP cURL - php

I want to retrieve a JSON data via REST API using PHP cURL.
Using the PHP code in this post, I was able to bypass the Spring Security page with the URL:
https://<host>/appserver/j_spring_security_check
However, the JSON exist inside this URL and I want to fetch it:
https://<host>/appserver/portal/api/1.0/apps
This is the PHP code I used to login automatically, I tried using file_get_contents(url) but it failed, only the below code worked to confirm login.
<?php
function login(){
$headers[] = "Accept: */*";
$headers[] = "Connection: Keep-Alive";
$headers[] = "Content-type: application/x-www-form-urlencoded;charset=UTF-8";
$data = "j_username=admin&j_password=demoserver";
$data = rtrim($data, "&");
global $sessionid;
$sessionid = dirname(__FILE__).'/cookie2.txt'
$ch = curl_init();
$options = array(
CURLOPT_REFERER => "https://<host>/appserver/portal/login",
CURLOPT_HTTPHEADER => $headers,
CURLOPT_COOKIEFILE => $sessionid,
CURLOPT_COOKIEJAR => $sessionid,
CURLOPT_URL => "https://<host>/appserver/j_spring_security_check",
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $data,
CURLOPT_HEADER => true,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0');
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
curl_close($ch);
}
login();
$text = file_get_contents($sessionid);
preg_match('/\w{32}/u', $text, $match);
$jsession = implode($match);
$headers1[] = "Accept: */*";
$headers1[] = "Connection: Keep-Alive";
$headers1[] = "Content-type: application/json";
$headers1[] = "Cookie: JSESSIONID=".$jsession;
$url = 'https://<host>/appserver/portal/api/1.0/apps';
$ch2 = curl_init();
curl_setopt($ch2, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_HTTPHEADER, $headers1);
curl_setopt($ch2, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch2, CURLOPT_COOKIEFILE, $sessionid);
curl_setopt($ch2, CURLOPT_COOKIEJAR, $sessionid);
curl_setopt($ch2, CURLOPT_HTTPGET, 1);
$result = curl_exec($ch2);
curl_close($ch2);
// var_dump(json_decode($result, true));
$output = json_decode ( $result );
This is the result after execution of PHP code:
HTTP/1.1 302 Found Date: Fri, 17 Feb 2017 03:59:00 GMT
Server: Apache/2.2.26 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e mod_jk/1.2.37
Set-Cookie: JSESSIONID=DB7139F7B8EE30F868A8392A4BF15523; Path=/appserver/; HttpOnly
Location: https://192.168.100.100:444/appserver/portal/welcome
Content-Length: 0
Access-Control-Allow-Origin: *
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain
How can I fetch the JSON given the above cURL options?

Why are you setting the cookie header manually when you already use cookiefile and cookiejar? That should not be needed.
It is also suspicious that you receive a setcookie header, probably the server is not accepting the session cookie you are sending.
Also, why are you not ignoring invalid SSL on the second request?
Finally, why are you not sending the same user-agent on the second request?
I guess that if you fix these problems you will see the expected behavior.

Related

PHP Login REST API

I'm trying to create a REST API and looking for a way to login using PHP, the documentation provided a login example using Python but I don't have an idea how to do this using PHP. I'm thinking if there's a PHP version of the Python code below.
See below code:
def login():
global sessionID
req = urllib2.Request("https://<host>/appserver/j_spring_security_check")
req.add_data(urllib.urlencode({"j_username" : "admin","j_password" :"demoserver"}))
res = opener.open(req)
sessionID = getCookie("JSESSIONID",cookies)
# Get the value of JSESSIONID cookie
response = res.read()
return
What is the login script (PHP version) that I can use if I need to login to web service using PHP (considering the Python example)?
Additional information:
Logging into the web service requires a JSON object as the request body with user name and password:
Successful execution of the method will return a Cookie session Id
Example request JSON: {"j_username" : "username", "j_password":"*******"}
User needs to parse the cookies and extract cookie with key as JSESSIONID. This JSESSIONID value needs to be added manually in all headers of the Rest calls
“Cookie”: “JSESSIONID=“ + cookieValue
Another example using Python:
//Request for All Apps
global sessionID
sID = "JSESSIONID="+sessionID
uri = "https://<hostname>/appserver/portal/api/1.0/apps"
req = urllib2.Request(uri)
req.add_header("Content-Type", "application/json")
req.add_header("Cookie", sID) # Header
req.get_method = "lambda: GET” # Method Type
res = opener.open(req) # URL Call
response = res.read()
return response
Request headers:
Host: 192.168.100.100:444
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*\/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://192.168.100.100:444/appserver/portal/login;jsessionid=6AD37194D43AB02BB79E26C71554958F
Cookie: JSESSIONID=6AD37194D43AB02BB79E26C71554958F
Connection: keep-alive
Upgrade-Insecure-Requests: 1
----------
When I tried curl using Linux, here's the code.
curl -k -i -H "Content-type: application/x-www-form-urlencoded" -c cookies.txt -X POST https://192.168.100.100:444/appserver/j_spring_security_check -d "j_username=admin&j_password=demoserver"
Here's the result of the linux curl, which I believe has succeed in connecting since I was routed to the welcome page.
HTTP/1.1 302 Found
Date: Thu, 16 Feb 2017 18:41:59 GMT
Server: Apache/2.2.26 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e mod_jk/1.2.37
Set-Cookie: JSESSIONID=358446CC1F87B2D698D48AFECA373691; Path=/appserver/; HttpOnly
Location: https://192.168.100.100:444/appserver/portal/welcome;jsessionid=358446CC1F87B2D698D48AFECA373691
Content-Length: 0
Access-Control-Allow-Origin: *
Content-Type: text/plain
----------
But when I tried using PHP curl with the code, still could not connect though.
<?php
$ch = curl_init();
$url = "https://192.168.100.100:444/appserver/j_spring_security_chec‌​k";
$postData = 'j_username=admin&j_password=demoserver';
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1); // -X
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData); // -d
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'application/x-www-form-urlencoded'
)); // -H
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); // -c
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); // -c
curl_setopt($ch, CURLOPT_HEADER, true); // -i
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // -k
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo curl_exec ($ch);
curl_close ($ch);
This is the resulting header in my browser.
Request URL: http://localhost/curl.php
Request method: GET
Remote address: 127.0.0.1:80
Status code: 200 OK
Version: HTTP
Response header:
Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/\*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Response headers:
Date: Thu, 16 Feb 2017 18:43:52 GMT
Server: Apache/2.2.26 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.1e mod_jk/1.2.37
Content-Language: en-US
Content-Length: 4815
Access-Control-Allow-Origin: *
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html;charset=UTF-8
You need to use CURL
Request to url : "https://<host>/appserver/j_spring_security_check"
And post data : "j_username=admin&j_password=demoserver"
So your code would look like
<?php
$ch = curl_init();
$url = "https://<host>/appserver/j_spring_security_check";
$postData = "j_username=admin&j_password=demoserver";
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec ($ch);
echo $server_output;
curl_close ($ch);
?>
Maybe you need cookie support? Something like this:
<?php
// this is for cookie handling in the session
session_start();
$tmpFname = tempnam(sys_get_temp_dir(),"COOKIE");
if (isset($_SESSION['cookies'])) {
file_put_contents($tmpFname,$_SESSION['cookies']);
}
// the request
$ch = curl_init();
$url = "https://<host>/appserver/j_spring_security_check";
$postData = "j_username=admin&j_password=demoserver";
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/x-www-form-urlencoded'
// you may add more request headers here
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// the next two options are for cookie handling
curl_setopt($ch, CURLOPT_COOKIEJAR, $tmpFname);
curl_setopt($ch, CURLOPT_COOKIEFILE, $tmpFname);
$server_output = curl_exec ($ch);
echo $server_output;
curl_close ($ch);
// this is for cookie handling in the session
$_SESSION['cookies'] = file_get_contents($tmpFname);
unlink($tmpFname);
I hope that helps.
Based on your edited question I think it would be best if you did:
<?php
$ch = curl_init();
$url = "https://<host>/appserver/j_spring_security_check";
$postData = '{"j_username":"admin","j_password":"demoserver"}';
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postData);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Host: 192.168.100.100:444',
'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0',
'Referer: https://192.168.100.100:444/appserver/portal/login;jsessionid=6AD37194D43AB02BB79E26C71554958F',
'Cookie: JSESSIONID=6AD37194D43AB02BB79E26C71554958F'
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); // see comment
$server_output = curl_exec ($ch);
echo $server_output;
curl_close ($ch);
I think this suits your question, in it's current form, better.

PHP curl is not parsing POST data

So i am trying to send a HTTPS POST request to a website but curl is not setting my post data nor my headers.
Here is my code(i changed the domain and values):
<?php
//Create connection
$ch = curl_init();
//Set the headers
$headers = array();
$headers[] = "Connection: Keep-Alive";
$headers[] = "Host: website.com";
$headers[] = "Content-Type: application/x-www-form-urlencoded";
$headers[] = "Something: Something_Data";
//Set the POST data
$headers = array();
$headers[] = "Connection: Keep-Alive";
$headers[] = "Host: website.com";
$headers[] = "Content-Type: application/x-www-form-urlencoded";
$headers[] = "Something: Something_Data";
// Configure the request (HTTPS)
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_URL, "https://website.com/test");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "MyUserAgentHere");
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_POST, 1); //just to be sure...
//Set the POST data and Headers
curl_setopt($ch, CURLOPT_POSTFIELDS, urlencode("DATA1=VAL1&DATA2=VAL2"));
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result=curl_exec($ch);
echo $result;
curl_close($ch);
?>
So basically i'm trying to send a HTTPS POST request to https://website.com/test". The request is sent and i get an answer but for some reasons it doesn't parse the POST data nor the Headers.
Here is my request in fiddler:
POST https://website.com/test HTTP/1.1
Host: website.com
Here is the answer from the request:
HTTP/1.1 400 Bad Request
Date: Fri, 06 Nov 2015 00:57:15 GMT
Server: Apache
Cache-Control: no-store
Pragma: no-cache
Connection: close
Content-Type: application/json;charset=UTF-8
Content-Length: 158
{"error":"unsupported DATA1"}
As you can see, the POST data and the headers are not set in my request for some reason. I have no idea why curl is not setting my headers and my post fields.
Infos: I am using XAMPP with php5.
You shouldn't URL-encode the entire parameter string, that will encode the = and & so they're no longer delimiters. You should just encode the values.
curl_setopt($ch, CURLOPT_POSTFIELDS, 'DATA1=' . urlencode($val1) . '&DATA2' . urlencode($val2));
You can use an array instead, and cURL will do the encoding for you.
curl_setopt($ch, CURLOPT_POSTFIELDS, array('DATA1' => $val1, 'DATA2' => $val2));
If you use the array mechanism, remove the Content-type: application/x-www-form-urlencoded header, because cURL sends it in multipart/form-data format.

PHP - How to Automatically Post Form in Another Website and Parse the Result

I am new to screen-scaraping and curl. I am planning to create a website like what http://www.skyscanner.com.my/ is doing that will allow a user to pull the origin, destination and date from the http://airasia.com website. Then the website return the flight schedule and ticket price to the user. The following is my code so far:
code:
<?php
$post_data['Origin']=$_POST['origin'];
$post_data['Destination']=$_POST['destination'];
$post_data['From']=$_POST['departDate'];
$post_data['To']=$_POST['returnDate'];
foreach ($post_data as $key => $value)
{
$post_items[] = $key . '=' . $value;
}
$post_string = implode ('&', $post_items);
$curl_connection = curl_init('https://booking.airasia.com/search.aspx');
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, False);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
$result = curl_exec($curl_connection);
print_r(curl_getinfo($curl_connection));
echo curl_errno($curl_connection) . '-' .
curl_error($curl_connection);
curl_close($curl_connection);
echo $result;
?>
The above do not return me any result from air asia. So i need some guidance to continue my task. Thank You
UPDATE
This Worked:
$request = array();
$request[] = "Host: mobile.airasia.com";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
$post = 'hash=61582ddd1b6ab8782ad63f1a6c6c1e46&trip-type=round-trip&origin=PEK&destination=SGN&date-depart-d=25&date-depart-my=2015-04&date-return-d=30&date-return-my=2015-04&passenger-count=1&child-count=0&infant-count=0&currency=MYR&depart-sellkey=&return-sellkey=&depart-details-index=&return-details-index=&depart-faretype=&return-faretype=&action=search&btnSearch=Search';
$url = 'https://mobile.airasia.com/en/search';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$data = curl_exec($ch);
if (curl_errno($ch)){
$data .= 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
$info = rawurldecode(var_export(curl_getinfo($ch),true));
// Get the cookies:
$skip = intval(curl_getinfo($ch, CURLINFO_HEADER_SIZE));
$requestHeader= substr($data,0,$skip);
$data = substr($data,$skip);
echo $data
Request Header.
POST /en/search HTTP/1.1
Accept-Encoding: deflate, gzip
Host: mobile.airasia.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
Accept-Language: en-US,en;q=0.5
Connection: keep-alive
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 366
Content-Type: application/x-www-form-urlencoded
Response Header:
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Cache-control: no-cache="set-cookie"
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Date: Mon, 20 Apr 2015 04:02:14 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
P3P: CP="IDC DSP COR ADM DEVi TAIi PSA PSD IVAi IVDi CONi HIS OUR IND CNT"
Server: redishot
Set-Cookie: locale=en; expires=Mon, 20-Apr-2015 05:02:11 GMT; path=/; secure
Set-Cookie: currency=MYR; expires=Mon, 20-Apr-2015 05:02:11 GMT; path=/; secure
Set-Cookie: PHPSESSID=p8mjtiiga4615pnhuu6vl1htiqkqsn7v; path=/; HttpOnly
Set-Cookie: AWSELB=CDFDE3A70C862943856FF6079178A94249700C674BDFF1E117C02BF52443FE13448AB71BEA2EA3F41C01293A39C3579A0A03905034DA565F71B4820BD1807C5558B22ED5E0;PATH=/;MAX-AGE=1800
Vary: Accept-Encoding
transfer-encoding: chunked
Connection: keep-alive
This is the data you need from the returned HTML
Flight No.
Flight Time
Cost
.
Results","position":1}]}}' data-disabled><div class="smallFont farelist no-discount "><div class=flight-no>D7 317</div><div class=flight-time>02:15<br>12:10</div><div class=flight-info><div class=box><div class=total-price>MYR 2,091.42</div>
Results","position":2}]}}' data-disabled><div class="smallFont farelist no-discount "><div class=flight-no>D7 317</div><div class=flight-time>02:15<br>12:55</div><div class=flight-info><div class=box><div class=total-price>MYR 2,106.26</div>
Results","position":3}]}}' data-disabled><div class="smallFont farelist no-discount "><div class=flight-no>D7 317</div><div class=flight-time>02:15<br>15:50</div><div class=flight-info><div class=box><div class=total-price>MYR 2,483.82</div>
end of update
Do not prepend the "?" in the post data.
This is the format in the php manual:
$post = 'key1=value1&key2=value2&key3=value3';
You cannot curl http://booking.airasia.com/search.aspx because it requires javaScript.
You have to use the mobile site. When use a Browser to see the HTTP Request and Response Headers do it with JavaScript disabled on the Browser.
Use:
https://mobile.airasia.com/en/search
The problem is the mobile site is not functioning correctly right now and says to try later. so I could not get any further.
Regarding the post
This is what is being posted by the search:
Content-Type: application/x-www-form-urlencoded
Content-Length: 366
hash=26edce4024c5611451a2a95a74e2bf01
&trip-type=round-trip
&origin=KUL
&destination=OOL&date-depart-d=20
&date-depart-my=2015-04&date-return-d=25
&date-return-my=2015-04
&passenger-count=1
&child-count=0&infant-count=0
&currency=MYR
&depart-sellkey=
&return-sellkey=
&depart-details-index=
&return-details-index=
&depart-faretype=
&return-faretype=
&action=search
&btnSearch=Search
Because their form is application/x-www-form-urlencoded your are almost doing the $post_string correctly. You can use an array for the post data but if value is an array, the Content-Type header will be set to multipart/form-data, which should be OK.
Because it is application/x-www-form-urlencoded you must urlencode $post_string :
$post_string` = urlencode(implode ('&', $post_items));
To get the cookies, you do not need, and possibly never will need:
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, False);
Remove it.
You will be getting a redirect and may need the cookie jar:
curl_setopt($ch,CURLOPT_COOKIEFILE, "/tmp/cookie.txt")
You may need to set the request header to match the Browser Request:
Create an array to put the Request Header Key Values
Fill in the Request array with exactly what is in the Request header of your upload.
EXAMPLE:
$request = array();
$request[] = "Host: www.example.com";
$request[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$request[] = "User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0";
$request[] = "Accept-Language: en-US,en;q=0.5";
$request[] = "Connection: keep-alive";
$request[] = "Cache-Control: no-cache";
$request[] = "Pragma: no-cache";
Add to curl:
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
Checking the the headers in a Browser
Then I use FireFox Inspector or Chrome Development Tool.
I go to the Network Tab
In FireFox I go to Settings and turn on "Enable Persistent logs"
In Chrome I click "Preserve log" on the Network Tab
Then I use the Browser to go wherever I want curl to go.
Now I can see every Request and Response including redirects and compare them with the save headers.
Inspect: Step by Step
right click select Inspect Element
Select the Network tab
Refresh the page
Select Documents (chrome) or HTML (firefox)
Clear the list
Post your upload
Select the upload Request in the list of Requests
I use FireFox with user agent switcher using an old Motorola user agent to retrieve the headers and HTML. Then I use the same user agent in curl's HTTPHEADER:
request[] = 'User-Agent: MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0
It is possible, not likely, the above cased the error when I tried
Your Query String $post_string is right but you are missing to prepend it with ? before sending curl. Try following:
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, "?".$post_string);

curl post from localhost

I'm trying to simulate http post from FireFox's LiveHTTP Replay by using curl. I believe the remote site has some sort of validation that checks where the request is coming from. If the request is coming from their own domain, then its fine. When i try to run a php curl script, i can see from the Live HTTP header that i'm making a GET request rather than a POST. Besides that, the Host, which is expected to be www.aliexpress.com has automatically changed to localhost.
If i use the Live HTTP Replay, it is working fine. So i copied the header data and tried to implement with curl but to no avail. Eg:
http://www.aliexpress.com/cross-domain/shoppingcart/index.html
POST /cross-domain/shoppingcart/index.html HTTP/1.1
Host: www.aliexpress.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.aliexpress.com/store/product/Wireless-N-Wifi-Repeater-802-11N-B-G-Network-Router-Range-Expander-300M-2dBi-Antennas-US/701252_523523529.html
Cookie: ali_apache_id=113.210.130.113.1374818286515.884332.4; ali_apache_track=mt=1|ms=|mid=my1023002521; xman_f=NkltWLLI3tebQbeQzQLiNBd2/KPKX0D81t0DghVMEl/frYuA+aVHnWGevMXWTEPqdLRqlKLbExYQkL61WPSt7Tr0LrdqOLLGM8yY5cBFOvY79qV9R5iTGSd44oPoKZruCpupEK9UBNSiOIf7Go1TN1AiM0ArpkHYTZ4rigCwLp5l2IEPYmFC8UzRnLivCFmLxbDuEewB52ulEop1Y9xtdEr88bjnwci1PldcvTxCmVDiOnm6rRfbnVfMAWaSWIkqQrnOEfwq2B4B/OER9K9IH7EHAMadb9IiOdMo3yavyt4DGWquCAq1izTtU8GE2mRmvi+PZ8WmR+PNOM3zYU4eaWM7uEevjmV2S7kTtlElmJGqxaT5RpSLcxiRxxbYJToejY36QxDf0MIIKTaaJTacVg==; aep_history=product_selloffer%5E%0Aproduct_selloffer%09709591781%091035163509%09523523529; __utma=3375712.263559759.1374818300.1375458795.1375606693.4; __utmz=3375712.1374818300.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); ali_beacon_id=113.210.130.113.1374818286515.884332.4; cna=+/14CsYcs08CAXGC0nFA22WC; xman_us_f=x_locale=en_US&no_popup_today=n&x_user=MY|dexter|wong|ifm|142465266&last_popup_time=1374818565653; u_info=qunLQLajxN+hFYWqPKiksew8tjAelFSu8cj+oG6e7nY=; intl_common_forever=wWZJ4jwSXakD7oylp5nnW9Nwmq8zgJYiqow0UyNV6PEUhc4f0KJghQ==; CoreID6=n; JSESSIONID=F86BD98D5E0CF42A7BE875F648DFA640; ali_apache_tracktmp=W_signed=Y; acs_usuc_t=acs_rt=9af4ec13bf134eb298193f9ac69395dc; acs_t=/iJST0zcbQeKUIQrTj1tDurMpZLQIdanO+zAZzyDlS+blTn+Rwd4skbiIdbQbEDO; __utmc=3375712; xman_t=rr/A0xwWzcNjpVptbDP061VCJ0dhjFwJPMn/JxOmi4eSjWlXq98nD8HBxnqOtR8ZIUClJqCqI39uwIkL6/R0WYQtBiqtFb8R0KGmzoiHDZ4R6dMhSZeEC5Am8y6iywMSG5My1MlUAhuWQI6/EPBlSYOWa8V/3IiNJnCOUd/Wm4DWQt6YHxS12kJbrUZxu2M7HeOquFa8Ga+yB/P0DT0Z9EhRum3S3uBC2+rFkh50z+91raLJiWJ0PV9NqHup3sPpAstiWlmem8QfBps0tFSx7tZn9WkllmyNJsTUYWO0cuxr0gpjWPU72Bb6fsroRovgRZ6xeqDah+WT94rnU2jrRybsL+7JDXmYPYC0GOTHjsSsloHSyGTvoD+FNyS3jGQPoP8KL7NXi+Dq+FrAqOETg3OH2oJp/h7nH5CWcsdojLHTngkABhNnB0ky/YRS8dV0s0oukEDPt+iXVEjQBBsIjAmtVX2fYx8KGRiRNiff/4rehQ4GDZzk2kdfJHItnUSk694SnpAgB6PrkNpGvu8adLjy8W6GuXk2XzujhsCSNkQ+3/uNpEbqoAimkCW+6KjJujJCPYIGevineVzSjMih7eDWpP/5TbgWtyhWKv3F5QbKZzibUq6w/YnerorvCHNPcssWgl0lswk=; __utmb=3375712.4.10.1375606693; xman_us_t=x_lid=my1023002521&sign=y&x_user=RmoP5to3fHwR+VNOC9lIAD7BpyTVa0YBflCR3S4eFIU=&need_popup=y
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 93
productId=523523529&standard=&quantity=7&country=MY&company=CPAM&cartfrom=main_store&skuAttr=
I've tried to implement the raw header(above) with the following:
<?php
// create a new cURL resource
$fields = array(
'productId' => 523523529,
'standard' => '',
'quantity' => 8,
'country' => 'MY',
'company' => 'CPAM',
'cartfrom' => 'main_store',
'skuAttr' => ''
);
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string, '&');
echo $fields_string . "<br/>";
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Host: www.aliexpress.com",
"Content-Type: application/x-www-form-urlencoded",
"Content-length: ". "93",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip, deflate",
"X-Requested-With: XMLHttpRequest")); //proceeding with the login.
curl_setopt($ch, CURLOPT_URL, urlencode("http://www.aliexpress.com/cross-domain/shoppingcart/index.html"));
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
//The encoded url below is referring to the login form for aliexpress.com
curl_setopt($ch, CURLOPT_REFERER, "http%3A%2F%2Fwww.aliexpress.com%2Fstore%2Fproduct%2FDual-sim-I9300-S3-MTK6589-quad-core-android-phone-1G-RAM-4G-ROM-4-7-inch%2F901666_1035163509.html%3FpromotionId%3D210526801");//This tells the server where were you directed from.
curl_setopt($ch,CURLOPT_POST, count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
//curl_setopt($ch, CURLOPT_COOKIESESSION, true);//indicates that this is a new session, i assume this forces the server to assign a new session?
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);//follows the redirection that is supplied by the server
curl_setopt($ch, CURLOPT_HEADER, true);
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);//THIS IS VERY IMPORTANT! This one of the most common option that is used because this simply means that
//the response from the server is returned as a string rather than output directly.
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);//This means to keep sending the login information(username and password) when there is a redirection
$str = curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
Thanks for your help!
Regards,
Dexter
try setting CURLOPT_POST to true
curl_setopt($ch, CURLOPT_POST, 1);
I think you should modify your code a bit:
remove the urlencode when setting the value of CURLOPT_URL
remove the Host and Content-length header from CURLOPT_HTTPHEADER
use http_build_query to build your $fields_string
For debugging purpose, I set the CURLOPT_RETURNTRANSFER to be true and var dump the respond.
My working code
$fields = array(
'productId' => 523523529,
'standard' => '',
'quantity' => 8,
'country' => 'MY',
'company' => 'CPAM',
'cartfrom' => 'main_store',
'skuAttr' => ''
);
$fields_string = http_build_query($fields);
echo $fields_string . "<br/>";
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/x-www-form-urlencoded",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip, deflate",
"X-Requested-With: XMLHttpRequest")); //proceeding with the login.
curl_setopt($ch, CURLOPT_URL, "http://www.aliexpress.com/cross-domain/shoppingcart/index.html");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
//The encoded url below is referring to the login form for aliexpress.com
curl_setopt($ch, CURLOPT_REFERER, "http://www.aliexpress.com/store/product/Dual-sim-I9300-S3-MTK6589-quad-core-android-phone-1G-RAM-4G-ROM-4-7-inch/901666_1035163509.html?promotionId=210526801");//This tells the server where were you directed from.
curl_setopt($ch,CURLOPT_POST, count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);//THIS IS VERY IMPORTANT! This one of the most common option that is used because this simply means that
//the response from the server is returned as a string rather than output directly.
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);//This means to keep sending the login information(username and password) when there is a redirection
$str = curl_exec($ch);
var_dump($str);
var_dump(curl_error($ch));
// close cURL resource, and free up system resources
curl_close($ch);
And the respond
HTTP/1.1 200 OK
Date: Sun, 04 Aug 2013 11:51:31 GMT
Server: Apache
P3P: CP="CAO PSA OUR"
Content-Language: en-US
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-XSS-protection: 1;mode=block
Content-Length: 56
Content-Type: plain/text;charset=utf-8
Set-Cookie: ali_apache_id=1.54.42.221.1375617091161.869918.6; path=/; domain=.aliexpress.com; expires=Wed, 30-Nov-2084 01:01:01 GMT
Set-Cookie: JSESSIONID=6EB1295945C27F8A2F788587D4C0E0A7; Path=/
Set-Cookie: ali_apache_track=; Domain=.aliexpress.com; Expires=Fri, 22-Aug-2081 15:05:38 GMT; Path=/
Set-Cookie: ali_apache_tracktmp=; Domain=.aliexpress.com; Path=/
Set-Cookie: acs_usuc_t=acs_rt=8fdfad47f53b46d489d0a905a5a9fb7c; Domain=.aliexpress.com; Path=/
Set-Cookie: xman_t=ZwO1ZDjGpaou2015+mejeWnS90vHjsN3YIDxbrXYOz/mbbJeIZM3q7Pw6ZGTygK2; Domain=.aliexpress.com; Path=/; HttpOnly
Set-Cookie: acs_t=2nqPb5i+QB7aDai5FXRM12xDJghxP4qjmcwPjwaXQ4SI6eV7eGpxjRGNjukEXuEW; Domain=.aliexpress.com; Path=/; HttpOnly
Set-Cookie: xman_f=MC/MUpjkYCKP+PRcAK43k9eQrTR+PE1rldMoChEUHVVlAUcYwh10BKJ0lxWlsPe4p+pYIPC/Vy4wIHJK8fiy4koUaF68CAolRC6UH7q0nmU5HcqWzgyjnA==; Domain=.aliexpress.com; Expires=Fri, 22-Aug-2081 15:05:38 GMT; Path=/; HttpOnly

C2DM 411. That’s an error. POST requests require a Content-length header

Am working on a push notification that makes user of C2DM. I have been able to obtain the users registration id, authentication token from Google. But when i try to send the message i receive this error from google "411. That’s an error. POST requests require a Content-length header".
function send($deviceRegistrationId, $msgType, $messageText) {
$f = fopen('request.txt', 'w');
$reg_id = $deviceRegistrationId; // Registration ID
$device_id = "1";
$data = array(
'registration_id' => trim($reg_id),
'collapse_key' => 'ck_'.trim($device_id),
'data.arg' => trim($messageText)
);
$dataStr = http_build_query($data);
$headers = array(
'Authorization: GoogleLogin auth='.$_SESSION['google'],
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: '.strlen($dataStr)
);
// Prepare the cURL request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://android.apis.google.com/c2dm/send");
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $dataStr);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_STDERR, $f);
// Send the request and echo the response body
$response = curl_exec($ch);
fclose($f);
echo "Reponse is ".$response;
}
Also below is the result i get when i output values of curl to a file
About to connect() to android.apis.google.com port 443 (#0)
Trying 74.125.39.139... * connected
Connected to android.apis.google.com (74.125.39.139) port 443 (#0)
successfully set certificate verify locations:
CAfile: none
CApath: /etc/ssl/certs
SSL connection using ECDHE-RSA-RC4-SHA
Server certificate:
subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.com
start date: 2011-11-10 07:48:51 GMT
expire date: 2012-11-10 07:58:51 GMT
subjectAltName: android.apis.google.com matched
issuer: C=US; O=Google Inc; CN=Google Internet Authority
SSL certificate verify ok.
POST /c2dm/send HTTP/1.1
Host: android.apis.google.com
Accept: /
Authorization: GoogleLogin auth=DQAAAMIAAADlJmeJmrTjmzdAbt1HiMvvVj_vdSduVXrkEFk_D19OG0o->FIn1ZzJ25d3MZfDTK2QErF_jEFAndPgC3RoGif6V-gs9w3-FA7VaEWd62qNPnscsqi1j6R0b0J5vtOwGItNmuXm5n1MZrOZ4sd3yx_D95rtzriymmyhilzLWNAyNjPO6FsmX-4Ty_3OwPaw02qe_oHeSvTNt7s6SW-_kT-T1hdJuywCoSf5p2esSzk9sUj9YDwtEXPneDIaB1z2Qy6NcMBjYY8X185GctBttXrjd
Content-Type: application/x-www-form-urlencoded
Content-Length: 207
HTTP 1.0, assume close after body
HTTP/1.0 411 Length Required
Content-Type: text/html; charset=UTF-8
Content-Length: 11791
Date: Sun, 27 Nov 2011 12:16:06 GMT
Server: GFE/2.0
Closing connection #0
how do i resolve this?
function send ($deviceRegistrationId, $msgType, $messageText) {
$reg_id = $deviceRegistrationId; // Registration ID
$device_id = "1";
// Build request body
$data = array (
'registration_id' => trim($reg_id),
'collapse_key' => 'ck_'.trim($device_id),
'data.arg' => trim($messageText)
);
$dataStr = trim(http_build_query($data));
// Headers for the request
$headers = array(
'Authorization: GoogleLogin auth='.trim($_SESSION['google']),
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: '.trim(strlen($dataStr)),
'Connection: close'
);
// Prepare the cURL request
$ch = curl_init();
curl_setopt_array($ch, array(
CURLOPT_URL => "https://android.apis.google.com/c2dm/send",
CURLOPT_HTTPHEADER => $headers,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $dataStr,
CURLOPT_RETURNTRANSFER => true
));
// For debugging
//$f = fopen('request.txt', 'w');
//curl_setopt($ch, CURLOPT_VERBOSE, true);
//curl_setopt($ch, CURLOPT_HEADER, true);
//curl_setopt($ch, CURLOPT_STDERR, $f);
// Send the request and echo the response body
$response = curl_exec($ch);
//fclose($f);
echo "Reponse is ".$response;
}
Some thing related to this is here http://codershelpingcoders.com.Have a look

Categories