Can't make Curl POST request from web app - php

A server I am working on appears to be denying outbound HTTP requests. The reason I think this is because I've tried both Guzzle and curl requests to the API.
The API lives on the same domain as the web server (this is temporary at clients request). I can make requests to the API server via Postman (Chrome plugin), but when I run that same request on the server, it doesn't return anything.
Here are the headers from the 'Postman' request:
POST /api2/user/session HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 49
Cache-Control: no-cache
Origin: chrome-extension://fdmmgilgnpjigdojojpjoooidkmcomcm
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: PHPSESSID=d9ad79c4c0822fc5c86f4d8799307f1b; _ga=GA1.2.1674422587.1425409444
Post data:
token=a559d5bba5a9e9517d5c3ed7aeb62db6&user=30972
This works. It returns the data. But when I call the same endpoint from within my web app, I get nothing.
$data = urlencode("token=a559d5bba5a9e9517d5c3ed7aeb62db6&user=30972");
$ch = curl_init('http://example.com/api2/user/session');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: ' . strlen($data))
);
$result = curl_exec($ch);
What I don't understand is I can run the following, and it returns the content:
print file_get_contents("http://www.google.com");
When I var_dump the $_POST fields on the endpoint user/session it returns the array of postdata using Postman but $_POST fields are blank when sending via the web app. Even before it makes any request to the database, the post fields should be set right?
Via SSH this also works:
curl -F token=a559d5bba5a9e9517d5c3ed7aeb62db6 -F user=30972 http://example.com/api2/user/session
As suggested in comments I've tried:
var_dump(function_exists('curl_version'));
// bool(true)
I can't figure out what's going on.
Edit: This works ... but I don't want to use sockets. Must be a curl issue.
$fp = fsockopen('example.com', 80);
$vars = array(
'token' => 'a559d5bba5a9e9517d5c3ed7aeb62db6',
'user' => '30972'
);
$content = http_build_query($vars);
fwrite($fp, "POST /api2/user/session HTTP/1.1\r\n");
fwrite($fp, "Host: example.com\r\n");
fwrite($fp, "Content-Type: application/x-www-form-urlencoded\r\n");
fwrite($fp, "Content-Length: ".strlen($content)."\r\n");
fwrite($fp, "Connection: close\r\n");
fwrite($fp, "\r\n");
fwrite($fp, $content);
header('Content-type: text/plain');
while (!feof($fp)) {
echo fgets($fp, 1024);
}
Edit:
curl_error() also returns no error.

To better understand the differences between the PHP code and cURL, I created a RequestBin instance and tried both on it. They yielded drastically different results:
It seemed like the POST data from the PHP script yielded an incorrect result for what was sent. This can be fixed by using a built-in PHP function http_build_query.
It will yield a more apt result:

This can be caused by a session lock... If you use curl to access the same server, the same session is used. While the script is running, the session is locked by default, this means that the current request has to finish before another is handled for the same session. This would explain a timeout of the request in curl, as your first request is not completed and another is made...
Using session_write_close() before the curl_exec will unlock the session and correct the problem.

It turns out I needed to use http_build_query.
$vars = array(
'token' => 'a559d5bba5a9e9517d5c3ed7aeb62db6',
'user' => '30972'
);
$content = http_build_query($vars);
$ch = curl_init('http://example.com/api2/user/session');
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/x-www-form-urlencoded',
'Content-Length: ' . strlen($content))
);
$result = curl_exec($ch);

Related

PHP file_get_contents returns with a 400 Error

My problem is pretty straightforward, but I cannot for the life of me figure out what is wrong. I've done something similar with another API, but this just hates me.
Basically, I'm trying to get information from https://owapi.net/api/v3/u/Xvs-1176/blob and use the JSON result to get basic information on the user. But whenever I try to use file_get_contents, it just returns
Warning: file_get_contents(https://owapi.net/api/v3/u/Xvs-1176/blob): failed to open stream: HTTP request failed! HTTP/1.1 400 BAD REQUEST in Z:\DevProjects\Client Work\Overwatch Boost\dashboard.php on line
So I don't know what's wrong, exactly. My code can be seen here:
$apiBaseURL = "https://owapi.net/api/v3/u";
$apiUserInfo = $gUsername;
$apiFullURL = $apiBaseURL.'/'.$apiUserInfo.'/blob';
$apiGetFile = file_get_contents($apiFullURL);
Any help would be largely appreciated. Thank you!
You need to set user agent for file_get_contents like this, and you can check it with this code. Refer to this for set user agent for file_get_contents.
<?php
$options = array('http' => array('user_agent' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:53.0) Gecko/20100101 Firefox/53.0'));
$context = stream_context_create($options);
$response = file_get_contents('https://owapi.net/api/v3/u/Xvs-1176/blob', false, $context);
print_r($response);
That's what page is sending: "Hi! To prevent abuse of this service, it is required that you customize your user agent".
You can customize it using curl like that:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://owapi.net/api/v3/u/Xvs-1176/blob");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$output = curl_exec($ch);
$output = json_decode($output);
if(curl_getinfo($ch, CURLINFO_HTTP_CODE) !== 200) {
var_dump($output);
}
curl_close($ch);
If you do curl -v https://owapi.net/api/v3/u/Xvs-1176/blob you will get a response and you will see what headers cURL includes by default. Namely:
> Host: owapi.net
> User-Agent: curl/7.47.0
> Accept: */*
So then the question is, which one does owapi care about? Well, you can stop cURL from sending the default headers like so:
curl -H "Accept:" -H "User-Agent:" -H "Host:" https://owapi.net/api/v3/u/Xvs-1176/blob
... and you will indeed get a 400 response. Experimentally, here's what you get back if you leave off the "Host" or "User-Agent" headers:
{"_request": {"api_ver": 3, "route": "/api/v3/u/Xvs-1176/blob"}, "error": 400, "msg": "Hi! To prevent abuse of this service, it is required that you customize your user agent."}
You actually don't need the "Accept" header, as it turns out. See the PHP docs on how to send headers along with file_get_contents.

Curl JSON request in PHP

I am trying to send the following cURL request in PHP:
$ curl -H 'Content-Type: application/json' -d '{"username":"a", "password":"b","msisdn":"447000000001","webhook":"http://example.com/"}' https://ms.4url.eu/lookup
Which should return:
198 bytes text/html; charset=UTF-8
{
"id": "ea26d0b2-b839-46b9-9138-50cc791bab47",
"msisdn": "447825001771",
"status": "Success",
"networkCode": "23471",
"countryName": "UK",
"countryCode": "GBR",
"network": "O2",
"networkType": "GSM",
"ported": "No"
}
I have tried to implement the code to send a request using cURL like so:
<?php
$data = array('{"username":"a", "password":"b", "msisdn":"447123121234", "webhook":"http://1f89e4a8.ngrok.io"}');
$data_string = json_encode($data);
$ch = curl_init('http://ms.4url.eu/lookup');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($data_string))
);
$result = curl_exec($ch);
?>
Using this method nothing seems to be happening I cannot see the JSON data being sent to ms.4url.eu/lookup, and when I try to echo $result I get no data echoed out?
Any help is much appreciated.
A Successful curl request is showing:
POST /testDir/getPost.php HTTP/1.1
host: 1f89e4a8.ngrok.io
accept: application/json
content-type: application/json
content-length: 198
Connection: close
X-Forwarded-For: 5.44.233.221
{"id":"ea26d0b2-b839-46b9-9138-50cc791bab47","msisdn":"447123121234","status":"Success","networkCode":"23471","countryName":"UK","countryCode":"GBR","network":"O2","networkType":"GSM","ported":"No"}
The post request from my PHP code is showing:
GET /testDir/curlPost.php HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Accept-Language: en-GB
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393
Accept-Encoding: gzip, deflate
Host: 1f89e4a8.ngrok.io
X-Forwarded-For: 92.11.143.199
Overall I would like to send the curl request from sendRequest.php and receive the Post for the webhook to getPost.php possibly using:
$entityBody = file_get_contents('php://input');
if ($entityBody != null){
echo $entityBody;
}
at the Minute I am using the getPost.php to send the HTTP 200 OK so ms.4url.eu stops sending requests 302 Found.
I think it is how you are building the json string...
You start by defining $data as an array of strings, and then json_encode it. But it is already in json format anyway (from a quick eyeball check).
The json_encode (and _decode) are meant to work with an associative array for your data.
Or just send the data string you are building, just check that it is in correct json format first.
<?php
// build an associative array
$data["username"]="a";
$data["password"]="b";
$data["msisdn"]="447123121234";
$data["webhook"]="http://1f89e4a8.ngrok.io";
// turn it into json formatted string
$json_data=json_encode($data);
print_r($data);
print($json_data);
?>
This gives you something like
Array
(
[username] => a
[password] => b
[msisdn] => 447123121234
[webhook] => http://1f89e4a8.ngrok.io
)
{"username":"a","password":"b","msisdn":"447123121234","webhook":"http:\/\/1f89e4a8.ngrok.io"}
Try to get like raw post data like below
<?php
$fp = fopen('php://input', 'r');
$rawData = stream_get_contents($fp);
echo "<pre>";
print_r($rawData);
echo "</pre>";
If you send only the json string via curl you have to use, on the destination page, php://input to retrieve the data, because there is not key => value and the variables $_POST and $_REQUEST don't intersect the request.
And, of course, check wich data are you sending in post. It seems incorrect to json_encode an array with an element "string"..
If you want to retrieve the request from the $_POST or $_REQUEST variable it's better if you put your json data into a key using the http_build_query function like following:
$data_string = json_encode($data);
$ch = curl_init('http://ms.4url.eu/lookup');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query(array('data' => $data_string)));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);

Why is my POST with cURL not returning JSON correctly?

In PHP, I'm trying to retrieve the url for a specific page in DocuSign that constantly refreshes. The POST to retrieve this url is in the form:
POST http://demo.docusign.net/restapi/{apiVersion}/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
This should return a json file in the form:
{
"url": "example.example.com"
}
However, I am extremely new to using PHP and POST methods and don't believe I'm doing this correctly. The API explorer for this method in particular is here. I am using cURL methods to make this request. Here is my code ($recipient,$account_id,$access_token are found accurately within another file):
$url = "http://demo.docusign.net/restapi/v2/accounts/$account_id
/envelopes/$envelope_id/views/recipient";
$body = array("returnUrl" => "http://www.docusign.com/devcenter",
"authenticationMethod" => "None",
"email" => "$recipient",
"userName" => "$recipient");
$body_string = json_encode($body);
$header = array(
'Accept: application/json',
'Content-Type: application/json',
'Content-Length: '.strlen($body_string),
);
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_POST, 1);
curl_setopt($curl, CURLOPT_POSTFIELDS, $body_string);
$json_response = curl_exec($curl);
$response = json_decode($json_response, true);
var_dump($response);
I am able to get the correct return on the API explorer, but not when making the request with PHP. I believe this is due to the fact that I am not incorporating the $header or $body correctly, but at this point I am just not sure.
ADDED: This is the raw output for the request when correctly running the method on the API Explorer:
Accept: application/json
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,fa;q=0.6,sv;q=0.4
Cache-Control: no-cache
Origin: https://apiexplorer.docusign.com
Referer: https://apiexplorer.docusign.com/
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Authorization: Bearer fGehcK7fkRvFguyu/7NGh01UUFs=
Content-Length:
Content-Type: application/json
This is the JSON request being formed in my code:
{
"returnUrl":"http:\/\/www.docusign.com\/devcenter",
"authenticationMethod":"Password",
"email":"example#example.com",
"userName":"example#example.com",
"clientUserId":"4c6228f4-fcfe-47f9-bee1-c9d5e6ab6a41",
"userId":"example#example.com"
}
You are not hitting a valid DocuSign URL in your cURL code. Right now you are sending requests to:
http://demo.docusign.net/apiVersion/v2/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
Instead of "apiVersion" it should be "restApi" like this:
http://demo.docusign.net/restapi/v2/accounts/{accountId}/envelopes/{envelopeId}/views/recipient
We can't send post fields, because we want to send JSON, not pretend to be a form (the merits of an API which accepts POST requests with data in form-format is an interesting debate). Instead, we create the correct JSON data, set that as the body of the POST request, and also set the headers correctly so that the server that receives this request will understand what we sent:
$data = array("name" => "Hagrid", "age" => "36");
$data_string = json_encode($data);
$ch = curl_init('http://api.local/rest/users');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($data_string))
);
$result = curl_exec($ch);
All these settings are pretty well explained on the curl_setopt() page, but basically the idea is to set the request to be a POST request, set the json-encoded data to be the body, and then set the correct headers to describe that post body. The CURLOPT_RETURNTRANSFER is purely so that the response from the remote server gets placed in $result rather than echoed. If you're sending JSON data with PHP, I hope this might help!
I know this question was asked more than 3 years ago, but this may help someone who finds this question because they are having the same problem. I do not see a cURL option that will decode the response in your code. I have found that I need to use the cURL option CURLOPT_ENCODING like this: curl_setopt($ch,CURLOPT_ENCODING,""); According to the PHP manual online, it says, 'CURLOPT_ENCODING - The contents of the "Accept-Encoding: " header. This enables decoding of the response. Supported encodings are "identity", "deflate", and "gzip". If an empty string, "", is set, a header containing all supported encoding types is sent.' You can find this option at https://www.php.net/manual/en/function.curl-setopt.php. I hope this helps save someone from having a headache.

How to avoid "HTTP/1.1 999 Request denied" response from LinkedIn?

I'm making request to LinkedIn page and receiving "HTTP/1.1 999 Request denied" response.
I use AWS/EC-2 and get this response.
On localhost everything works fine.
This is sample of my code to get html-code of the page.
<?php
error_reporting(E_ALL);
$url= 'https://www.linkedin.com/pulse/5-essential-strategies-digital-michelle';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$response = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
var_dump($response);
var_dump($info);
I don't need whole page content, just meta-tags (title, og-tags).
Note that the error 999 don't exist in W3C Hypertext Transfer Protocol - HTTP/1.1, probably this error is customized (sounds like a joke)
LinkedIn don't allow direct access, the probable reason of them blocking any "url" from others webservers access should be to:
Prevent unauthorized copying of information
Prevent invasions
Prevent abuse of requests.
Force use API
Some IP addresses of servers are blocked, as the "IP" from "domestic ISP" are not blocked and that when you access the LinkedIn with web-browser you use the IP of your internet provider.
The only way to access the data is to use their APIs. See:
Accessing LinkedIn public pages using Python
Heroku requests return 999
Note: The search engines like Google and Bing probably have their IPs in a "whitelist".
<?php
header("Content-Type: text/plain");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.linkedin.com/company/technistone-a-s-");
$header = array();
$header[] = "Host: www.linkedin.com";
$header[] = "User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0";
$header[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$header[] = "Accept-Language: en-US,en;q=0.5";
$header[] = "Accept-Encoding: gzip, deflate, br";
$header[] = "Connection: keep-alive";
$header[] = "Upgrade-Insecure-Requests: 1";
curl_setopt($ch,CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_HTTPHEADER , $header);
$my_var = curl_exec($ch);
echo $my_var;
LinkedIn is not supporting the default encoding 'identity' , so if you set the header
'Accept-Encoding': 'gzip, deflate'
you should get the response , but you would have to decompress it.
I ran into this while doing local web development and using the LinkedIn badge feature (profile.js). I was only getting the 999 Request denied in Chrome, so I just cleared my browser cache and localStorage and it started to work again.
UPDATE - Clearing cache was just a coincidence and the issue came back. LinkedIn is having issues with their badge functionality.
I submitted a help thread to their forums.
https://www.linkedin.com/help/linkedin/forum/question/714971

Get http-statuscode without body using cURL?

I want to parse a lot of URLs to only get their status codes.
So what I did is:
$handle = curl_init($url -> loc);
curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
curl_setopt($handle, CURLOPT_HEADER , true); // we want headers
curl_setopt($handle, CURLOPT_NOBODY , true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$response = curl_exec($handle);
$httpCode = curl_getinfo($handle, CURLINFO_HTTP_CODE);
curl_close($handle);
But as soon as the "nobody"-option is set to true, the returned status codes are incorrect (google.com returns 302, other sites return 303).
Setting this option to false is not possible because of the performance loss.
Any ideas?
The default HTTP request method for curl is GET. If you want only the response headers, you can use the HTTP method HEAD.
curl_setopt($handle, CURLOPT_CUSTOMREQUEST, 'HEAD');
According to #Dai's answer, the NOBODY is already using the HEAD method. So the above method will not work.
Another option would be to use fsockopen to open a connection, write the headers using fwrite. Read the response using fgets until the first occurrence of \r\n\r\n to get the complete header. Since you need only the status code, you just need to read the first 13 characters.
<?php
$fp = fsockopen("www.google.com", 80, $errno, $errstr, 30);
if ($fp) {
$out = "GET / HTTP/1.1\r\n";
$out .= "Host: www.google.com\r\n";
$out .= "Accept-Encoding: gzip, deflate, sdch\r\n";
$out .= "Accept-Language: en-GB,en-US;q=0.8,en;q=0.6\r\n";
$out .= "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36\r\n";
$out .= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8\r\n";
$out .= "Connection: Close\r\n\r\n";
fwrite($fp, $out);
$tmp = explode(' ', fgets($fp, 13));
echo $tmp[1];
fclose($fp);
}
cURL's nobody option has it use the HEAD HTTP verb, I'd wager the majority of non-static web applications I the wild don't handle this verb correctly, hence the problems you're seeing with different results. I suggest making a normal GET request and discarding the response.
i suggest get_headers() instead:
<?php
$url = 'http://www.example.com';
print_r(get_headers($url));
print_r(get_headers($url, 1));
?>

Categories