I am trying to upload a file to a Rest API (Tableau Server Rest API) endpoint using PHP cURL.
The file upload procedure exists of three steps:
Initiate file upload (request upload token)
Append file upload (upload file data)
Publish resource (save the file)
I was having issues with the server giving me a 500 status code on the second step. After contacting the support we figured out that the problem is most likely that the curl request seems to using the -data flag instead of the --data-binary flag which means some kind of encoding happens on the request body which should not be happening. This causes the server to respond with a 500 status code instead of an actual error message...
I would like to know how i can make a cURL request with the --data-binary flag in PHP.
Relevant sections of my current code:
// more settings
$curl_opts[CURLOPT_CUSTOMREQUEST] = $method;
$curl_opts[CURLOPT_RETURNTRANSFER] = true;
$curl_opts[CURLOPT_HTTPHEADER] = $headers;
$curl_opts[CURLOPT_POSTFIELDS] = $body;
//more settings
curl_setopt_array( $this->ch, $curl_opts );
$responseBody = curl_exec( $this->ch );
$method is "PUT", $headers contains an array with Content-Type: multipart/mixed; boundary=boundary-string and the $body is constructed like this:
$boundary = md5(date('r', time()));
$body = "--$boundary\n";
$body .= "Content-Disposition: name='request_payload'\nContent-Type: text/xml\n\n";
$body .= "\n--$boundary\n";
$body .= "Content-Disposition: name='tableau_file'; filename='$fileName'\nContent-Type: application/octet-stream\n\n";
$body .= file_get_contents( $path );
$body .= "\n--$boundary--";
Where the $boundary is the same as the boundary-string in the content-type header.
i am aware this is a kinda/very messy way to construct my body, i am planning to use Mustache as soon as i can actually upload my files :S
(i would like to mention that this is my first post here, please be gentle...)
CURLOPT_POSTFIELDS can accept an array of key=value field pairs. DON'T build your own mime body.
This is all you should have, really:
$data = array(
'tableau_file' => '#/path/to/file';
^---tell curl this field is a file
etc..
);
curl_setopt($this->ch, CURLOPT_POSTFIELDS, $data);
I ended up trying to implement a "--data-binary" using the exec function in PHP. Before i could move to this i had to cleanup some of my request body building code. This cleanup involved moving from string building ($body .= "this" . $that . "\n";) to a template engine (Mustache). Somehow this change fixed the problem.
Related
hello all
i am receiving the http post request from html form to my action.php file and then this php file is writing these values into a text file.
now i want the php file to send the data it receives to a remote http server for further processing (i am using a simple python http server)
here is my php code :
<?php
$data1 = $_REQUEST['key1'];
$data2 = $_REQUEST['key2'];
$data3 = $_REQUEST['key3'];
$data4 = $_REQUEST['key4'];
$data5 = $_REQUEST['key5'];
$fp = fopen('datafile.txt', 'w+');
fwrite($fp, implode("\n", [$data1, $data2, $data3, $data4, $data5]));
fclose($fp);
// example data
$data = array(
'key1'=> $data1,
'key2'=> $data2,
'key3'=> $data3,
'key4'=> $data4,
'key5'=> $data5
);
// build post body
$body = http_build_query($data); // foo=bar&baz=boom
// options, headers and body for the request
$opts = array(
'http'=>array(
'method'=>"POST",
'header'=>"Accept-language: en\r\n",
'data' => $body
)
);
// create request context
$context = stream_context_create($opts);
// do request
$response = file_get_contents('http://2.22.212.12:42221', false, $context)
?>
but when i submit the form , only the datafile.txt file is generated and no post request is sent to the remote python server
what am i doing wrong ?
I would highly recommend using Guzzle, but the docs for stream_context_create use fopen() instead of file_get_contents(), so that might one factor.
Another is the header. The comments on the doc page set the header this way for POST requests:
'header'=> "Content-type: application/x-www-form-urlencoded\r\n"
. "Content-Length: " . strlen($body) . "\r\n",
Take a look at the answers here too for more examples.
TL;DR: why does reuploding the uploaded data not work?
I' trying to upload the data a user uploaded to my file to another server. This means, I want to post my post data.
I want to upload some data to upload.php, which then should get posted to test.php (which simply outputs the raw post data). And due to memory saving, I wanted it to work without generating the post whole message as string. Thus I also don't want to use curl.
test.php
<?php
echo 'foo', file_get_contents('php://input'), 'bar';
upload.php
<?php
//new line variable
$NL = "\r\n";
//open the posted data as a resource
$client_upload = fopen('php://input', 'r');
//open the connection to the other server
$server_upload = fsockopen('ssl://example.com', 443);
//write the http headers to the socket
fwrite($server_upload, 'POST /test.php HTTP/1.1' . $NL);
fwrite($server_upload, 'Host: example.com' . $NL);
fwrite($server_upload, 'Connection: close' . $NL);
//header/body divider
fwrite($server_upload, $NL);
//loop until client upload reached the end
while (!feof($client_upload)) {
fwrite($server_upload, fread($client_upload, 1024));
}
//close the client upload resource - not needed anymore
fclose($client_upload);
//intitalize response variable
$response = '';
//loop until server upload reached the end
while (!feof($server_upload)) {
$response .= fgets($server_upload, 1024);
}
//close the server upload resource - not needed anymore
fclose($server_upload);
//output the response
echo $response;
When I post { "test": true } (from Fiddler) to test.php file, it outputs foo{ "test": true }bar.
Now when I try to do the same for upload.php, I just get foobar (and the http headers from test.php) but not the uploaded content.
Finally I managed to fix this error. Apparently the other server (and mine) depend on the http header Content-Length.
As you can see in my answer, I was not sending (neither calculating) this header. So when I finally now calculated and sent the Content-Length header everything worked. This are the missing lines:
$content_length = fstat($client_upload)['size'];
fwrite($server_upload, 'Content-Length: ' . $content_length . $NL);
The reupload (of the uploaded data to my server) didn't work, because the other server just read the body as long as it was specified in the Content-Length header. Because I was not sending this header, it didn't work.
Currently, in a PHP code base I work on, several timeline items and a bundle cover are inserted in 4 calls like this:
insert_timeline_item($mirror_service, $new_timeline_item_1, null, null);
insert_timeline_item($mirror_service, $new_timeline_item_2, null, null);
insert_timeline_item($mirror_service, $new_timeline_item_3, null, null);
insert_timeline_item($mirror_service, $new_timeline_item_bundle_cover, null, null);
I'm aware of the Java and Python ways to send these all in a single batch HTTP call to the Mirror API. How do I do that in PHP?
Right now the cards arrive on the Glass relatively slowly and a user will often try to scroll and see there is nothing to scroll before the other results arrive, for example. Anything that could help the results all arrive at once would help a lot. We already mitigate as much as possible by only making a notification sound on the last card, but it isn't enough for a good user experience.
A batch request to insert timeline items is not guaranteed to arrive all at once or in any particular order. The documentation on batch operations has a footnote that points this out (see the note at the end of the section titled "Response to a batch request" at https://developers.google.com/glass/batch). The only way to ensure that your cards arrive in order is to instead wait for each return response on insert, than fire the last card to create the bundle with your notification chime.
EDIT: There is a convenience method (Google_Http_Batch()) that you can use when you setup your client by setting the client to setUseBatch(true). See: https://developers.google.com/api-client-library/php/guide/batch
In terms of just the batch request in PHP, to my knowledge there is not an existing convenience method. You have to build your own request, setup your context via stream_context_create() and then send it via fopen() or file_get_contents().
I haven't tested the following code, but this is the basic concept:
// Our batch url endpoint
$endpoint = "/batch";
// Boundary for our multiple requests
$boundary = "===============SOMETHINGTHATDOESNOTMATCHCONTENT==\n";
//
// Build a series of timeline cards to insert
// probably spin this off into it's own method for simplicty sake
//
$timeline_items .= "--" . $boundary;
$timeline_items .= "Content-Type: application/http\n";
$timeline_items .= "Content-Transfer-Encoding: binary\n";
$timeline_items .= "POST /mirror/v1/timeline HTTP/1.1\n";
$timeline_items .= "Content-Type: application/json\n";
// You'd need the specific ouath2 bearer token for your user here
//
// Note, if you were simply sending a single batch to always one user,
// you could reasonably move this header to the outer /batch params
// as it will flow down to all child requests as per the documentation
//
// see "Format of a batch request" section at https://developers.google.com/glass/batch
//
$timeline_items .= "authorization: Bearer " . $user_bearer_token . "\n";
$timeline_items .= "accept: application/json\n";
$timeline_items .= "content-length: " . strlen($timeline_card_json) . "\n\n";
$timeline_items .= $timeline_card_json . "\n";
$timeline_items .= "--" . $boundary;
//
// Add some other timeline items into your $timeline_items batch
//
// Setup our params for our context
$params = array(
'http' => array(
'method' => 'POST',
'header' => 'Content-Type: multipart/mixed; boundary="' . $boundary . '"',
'accept-encoding' => 'gzip, deflate',
'content' => $timeline_items
)
);
// Create context
$context = stream_context_create($params);
// Fire off request
$batch_result = file_get_contents($endpoint, false, $context);
I am trying to convert this code snippet from PHP to Python (programming newbie) and am finding difficulty in doing so:
The PHP that I am trying to convert is as follows:
$fp = fsockopen($whmcsurl, 80, $errno, $errstr, 5);
if ($fp) {
$querystring = "";
foreach ($postfields AS $k=>$v) {
$querystring .= "$k=".urlencode($v)."&";
}
$header="POST ".$whmcsurl."modules/servers/licensing/verify.php HTTP/1.0\r\n";
$header.="Host: ".$whmcsurl."\r\n";
$header.="Content-type: application/x-www-form-urlencoded\r\n";
$header.="Content-length: ".#strlen($querystring)."\r\n";
$header.="Connection: close\r\n\r\n";
$header.=$querystring;
$data="";
#stream_set_timeout($fp, 20);
#fputs($fp, $header);
$status = #socket_get_status($fp);
while (!#feof($fp)&&$status) {
$data .= #fgets($fp, 1024);
$status = #socket_get_status($fp);
}
#fclose ($fp);
}
It corresponding Python code that I wrote is as follows:
fp = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
fp.connect(("my ip", 80))
if (fp):
querystring = ""
#print postfields
for key in postfields:
querystring = querystring+key+"="+urllib.quote(str(postfields[key]))+"&"
header = "POST "+whmcsurl+"modules/servers/licensing/verify.php HTTP/1.0\r\n"
header+="Content-type: application/x-www-form-urlencoded\r\n"
header+="Content-length: "+str(len(querystring))+"\r\n"
header+="Connection: close\r\n\r\n"
#header+=querystring
data=""
request = urllib2.Request(whmcsurl,querystring,header)
response = urllib2.urlopen(request)
data = response.read()
Here, I am faced with the following error:
request = urllib2.Request(whmcsurl,querystring,header)
File "/usr/lib64/python2.6/urllib2.py", line 200, in __init__
for key, value in headers.items():
AttributeError: 'str' object has no attribute 'items'
So I am guessing that Python is expecting a dictionary for the header. But the PHP sends it as a string.
May I know how to solve this issue?
Thanks in advance
You are overcomplicating things, by quite some distance. Python takes care of most of this for you. There is no need to open a socket yourself, nor do you need to build headers and the HTTP opening line.
Use the urllib.request and urllib.parse modules to do the work for you:
from urllib.parse import urlopen
from urllib.request import urlopen
params = urlencode(postfields)
url = whmcsurl + 'modules/servers/licensing/verify.php'
response = urlopen(url, params)
data = response.read()
urlopen() takes a second parameter, the data to be sent in a POST request; the library takes care of calculating the length of the body, and sets the appropriate headers. Most of all, under the hood it uses another library, httplib, to take care of the socket connection and producing valid headers and a HTTP request line.
The POST body is encoded using urllib.parse.urlencode(), which also takes care of proper quoting for you.
You may also want to look into the external requests library, which provides an easier-to-use API still:
import requests
response = requests.post(whmcsurl + 'modules/servers/licensing/verify.php', params=params)
data = response.content # or response.text for decoded content, or response.json(), etc.
your headers should look like this
headers = { "Content-type" : "application/x-www-form-urlencoded" };
I am echoing json_encoded data from one php script to another (the request is made by fsockopen/GET).
When having encoded an array with 40 elements, there is no problem. When doing exactly the same thing with 41, some numbers and \r\n is added to the beginning of the json string.
This is the beginning of the string just before I echo it:
{"transactions":[{"transaction_id":"03U191739F337671L",
This is how I send the data:
header('Content-Type: text/plain; charset=utf-8');
error_log(json_encode($transaction_list));
echo json_encode($transaction_list);
As soon as I have received the data in the requesting script I print it again to error_log:
27fc\r\n{"transactions":[{"transaction_id":"03U191739F337671L",
The "27fc\r\n" is not there if I retrieve less data.
This is how I handle the response:
$response="";
while (!feof($fp)) {
$response .= fgets($fp, 128);
}
//Seperate header and content
$separator_position = strpos($response,"\r\n\r\n");
$header_text = substr($response,0,$separator_position);
$body = substr($response,$separator_position+4);
error_log($body);
fclose($fp);
I have tried playing around with the time out of the fsockopen request, that doesn't matter. The same thing with max_execution_time and max_input_time in php.ini, doesn't matter. I was thinking that the content in some way may have been cut due to time out...
The 41st array is having no different format of the content than the preceding ones.
What is happening and how can I fix it?
I am using Linux, Apache (httpd) and PHP.
UPDATE
The data seems to be chunked. In the response, following header is included: "Transfer-Encoding: chunked".
Based on #Salmans idea of using file_get_contents, this is the working solution. This uses POST to send the data (GET didn't seem to be working, I think one has to append that query string to the URL oneself):
$postdata = http_build_query(
array('customer_id' => $customer_id)
);
$opts = array('http' =>
array(
'method' => 'POST',
'header' => 'Content-type: application/x-www-form-urlencoded',
'content' => $postdata
)
);
$context = stream_context_create($opts);
$content = file_get_contents($my_url, false, $context);
return $content;