Elasticsearch 7.13 bulk update PHP CURL - php

He or she who answers this question will have a very special place in my heart. I'm trying to update many records at once using Curl with PHP on elasticsearch 7.13. It seems that no matter what I do, I get the error "The bulk request must be terminated by a newline". I have tried many, many renditions of this code. Is there perhaps a way to overload what PHP CURL is doing to force it to use binary data? I believe it is stripping the newline characters, but if I change to plain text submission it won't be accepted by elasticsearch.
public function bulkUpdate(){
$query_json_store = '';
foreach($this->records['hits']['hits'] as $person){
$entry1 = [
"update"=>[
"_id"=> $person['_id'],
"type"=> "doc",
"_index"=> "human_data",
]
];
$entry2 = [
"doc"=>[
"scanned"=> true
]
];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'localhost:9200/human_data/doc/_bulk');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_POSTFIELDS, '\n'.json_encode($params)).'\n';
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$headers = array();
$headers[] = 'Content-Type: application/x-ndjson';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close($ch);
\Log::info($result);
}
}

Related

X-Cluster download failed response when running curl_exec PHP

I have this code implemented inside Laravel.
$ch = curl_init();
$link = "https://link";
$token = "";
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
"Accept: application/json",
"Authorization: Bearer {$token}",
"Content-Type: application/json"
]);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, function ($curl, $header) use (&$responseHeaders) {
$len = strlen($header);
$header = explode(':', $header, 2);
if (count($header) < 2)
return $len;
$responseHeaders[strtolower(trim($header[0]))][] = trim($header[1]);
return $len;
});
$response = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
It takes time to download https://link and then I got a $response value of
{
"Message": "X-Cluster download failed",
"Exception": null
}
From the return message I have no Idea what is
X-Cluster download failed
means.
I take to research but couldn't find a direct answer.
Any Idea?
Edit: https://link is valid link that represented tha original one it's a sensitive and private, the reason why I pasted like that.

Passing JSON to PHP CURL [duplicate]

I have been working on building an Rest API for the hell of it and I have been testing it out as I go along by using curl from the command line which is very easy for CRUD
I can successfully make these call from the command line
curl -u username:pass -X GET http://api.mysite.com/pet/1
curl -d '{"dog":"tall"}' -u username:pass -X GET http://api.mysite.com/pet
curl -d '{"dog":"short"}' -u username:pass -X POST http://api.mysite.com/pet
curl -d '{"dog":"tall"}' -u username:pass -X PUT http://api.mysite.com/pet/1
The above calls are easy to make from the command line and work fine with my api, but now I want to use PHP to create the curl. As you can see, I pass data as a json string. I have read around and I think I can probably do the POST and include the POST fields, but I have not been able to find out how to pass http body data with GET. Everything I see says you must attached it to the url, but it doesn't look that way on the command line form. Any way, I would love it if someone could write the correct way to do these four operations in PHP here on one page. I would like to see the simplest way to do it with curl and php. I think I need to pass everything through the http body because my php api catching everything with php://input
PUT
$data = array('username'=>'dog','password'=>'tall');
$data_json = json_encode($data);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json','Content-Length: ' . strlen($data_json)));
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
POST
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
GET
See #Dan H answer
DELETE
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($ch, CURLOPT_POSTFIELDS,$data_json);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
You can use this small library: https://github.com/ledfusion/php-rest-curl
Making a call is as simple as:
// GET
$result = RestCurl::get($URL, array('id' => 12345678));
// POST
$result = RestCurl::post($URL, array('name' => 'John'));
// PUT
$result = RestCurl::put($URL, array('$set' => array('lastName' => "Smith")));
// DELETE
$result = RestCurl::delete($URL);
And for the $result variable:
$result['status'] is the HTTP response code
$result['data'] an array with the JSON response parsed
$result['header'] a string with the response headers
Hope it helps
For myself, I just encode it in the url and use $_GET on the destination page. Here's a line as an example.
$ch = curl_init();
$this->json->p->method = "whatever";
curl_setopt($ch, CURLOPT_URL, "http://" . $_SERVER['SERVER_NAME'] . $this->json->path . '?json=' . urlencode(json_encode($this->json->p)));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
curl_close($ch);
EDIT: Adding the destination snippet... (EDIT 2 added more above at OPs request)
<?php
if(!isset($_GET['json']))
die("FAILURE");
$json = json_decode($_GET['json']);
$method = $json->method;
...
?>
I was Working with Elastic SQL plugin.
Query is done with GET method using cURL as below:
curl -XGET http://localhost:9200/_sql/_explain -H 'Content-Type: application/json' \
-d 'SELECT city.keyword as city FROM routes group by city.keyword order by city'
I exposed a custom port at public server, doing a reverse proxy with Basic Auth set.
This code, works fine plus Basic Auth Header:
$host = 'http://myhost.com:9200';
$uri = "/_sql/_explain";
$auth = "john:doe";
$data = "SELECT city.keyword as city FROM routes group by city.keyword order by city";
function restCurl($host, $uri, $data = null, $auth = null, $method = 'DELETE'){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $host.$uri);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
if ($method == 'POST')
curl_setopt($ch, CURLOPT_POST, 1);
if ($auth)
curl_setopt($ch, CURLOPT_USERPWD, $auth);
if (strlen($data) > 0)
curl_setopt($ch, CURLOPT_POSTFIELDS,$data);
$resp = curl_exec($ch);
if(!$resp){
$resp = (json_encode(array(array("error" => curl_error($ch), "code" => curl_errno($ch)))));
}
curl_close($ch);
return $resp;
}
$resp = restCurl($host, $uri); //DELETE
$resp = restCurl($host, $uri, $data, $auth, 'GET'); //GET
$resp = restCurl($host, $uri, $data, $auth, 'POST'); //POST
$resp = restCurl($host, $uri, $data, $auth, 'PUT'); //PUT
set one more property curl_setopt($ch, CURLOPT_SSL_VERIFYPEER , false);

ServiceM8 Post JSON via Curl PHP and API

Trying to send a post request to the ServiceM8 Api however when i attempt the request i get back no errors and nothing adding to the ServiceM8 api.
Here is what servicem8 docs suggest:
curl -u email:password
-H "Content-Type: application/json"
-H "Accept: application/json"
-d '{"status": "Quote", "job_address":"1 Infinite Loop, Cupertino, California 95014, United States","company_id":"Apple","description":"Client has requested quote for service delivery","contact_first":"John","contact_last":"Smith"}'
-X POST https://api.servicem8.com/api_1.0/job.json
and here is what i have:
$data = array(
"username" => "**************8",
"password" => "****************",
"job_address" => "1 Infinite Loop, Cupertino, California 95014, United States"
);
$url_send ="https://api.servicem8.com/api_1.0/job.json";
$str_data = json_encode($data);
function sendPostData($url, $post){
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS,$post);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
return $result;
-- UPDATE TO SHOW PREVIOUS ATTEMPTS TO RESOLVE BUT HAD NO LUCK..
<?php
$data = array(
"username" => "*******************",
"password" => "**********",
"job_address" => "1 Infinite Loop, Cupertino, California 95014, United States"
);
$url_send ="https://api.servicem8.com/api_1.0/job.json";
$str_data = json_encode($data);
function sendPostData($url, $post){
$headers= array('Accept: application/json','Content-Type: application/json');
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POSTFIELDS,$post);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
curl_close($ch); // Seems like good practice
return $result;
}
echo " " . sendPostData($url_send, $str_data);
?>
adding the headers as i have in that example still does nothing and does not create the record in servicem8 or show an error.
Hopefully someone can help me make the correct Curl request using the PHP libary.
Thanks
First issue is it looks like you are not setting authentication details correctly. To use HTTP basic auth in CURL:
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
Second issue is that job_status is a mandatory field when creating Jobs, so you need to make sure to include that as part of your create request.
Assuming that you receive a HTTP 200 response, then you the UUID of the record you just created is returned in the x-record-uuid header (docs). See this answer for an example of how to get headers from a HTTP response in CURL.
Here's your example code modified to include the above advice:
$data = array(
"job_address" => "1 Infinite Loop, Cupertino, California 95014, United States",
"status" => "Work Order"
);
$url_send = "https://api.servicem8.com/api_1.0/job.json";
$str_data = json_encode($data);
function sendPostData($url, $post, $username, $password) {
$ch = curl_init($url);
if ($username && $password) {
curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
}
$headers = array('Accept: application/json','Content-Type: application/json');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 1); // Return HTTP headers as part of $result
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$result = curl_exec($ch);
// $result is the HTTP headers followed by \r\n\r\n followed by the HTTP body
$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
$header = substr($result, 0, $header_size);
$body = substr($result, $header_size);
$strRecordUUID = "";
$arrHeaders = explode("\r\n", $header);
foreach ($arrHeaders as $strThisHeader) {
list($name, $value) = explode(':', $strThisHeader);
if (strtolower($name) == "x-record-uuid") {
$strRecordUUID = trim($value);
break;
}
}
echo "The UUID of the created record is $strRecordUUID<br />";
return $body;
}
echo "the response from the server was <pre>" . sendPostData($url_send, $str_data, $username, $password) . "</pre>";

PHP Curl Call when executed outputs an HTML content where as JSON was expected

When i do a curl call using php which is as shown below
<?php
$username = "XXXX";
$password = "XXXX";
$url = "https://domainname/method";
$ch = curl_init();
$fullAddress="202 220 GEORGE ST";
$payload = json_encode( array( "payload"=>[array("fullAddress"=>$fullAddress)],
"sourceOfTruth"=>'AUPAF'));
//var_dump($payload);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type:application/json','Accept: application/json'));
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_POST, true );
curl_setopt($ch, CURLOPT_POSTFIELDS, $payload );
# Return response instead of printing.
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
# Send request.
$result = curl_exec($ch);
$err = curl_error($ch);
curl_close($ch);
echo '<pre>', htmlentities($result), '</pre>';
$json = json_decode($result, true);
?>
it outputs an weird response saying that
<html><head><title>Apache Tomcat/7.0.52 (Ubuntu) - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:whi
I'm actually stuck up with it for several hours so any suggestions would be helpfull
I've post a comment, but I recon that it's not very well readable.
Here is my suggestion, to use for nested arrays:
json_encode( [ 'payload' => [
['fullAddress' => $fullAddress]
],
'sourceOfTruth' => 'AUPAF'
] );
In your example your arrays where further nested, maybe the server does not expact this.
A few other tips:
Make sure the URL to the API is correct.
The headers send to the server are as expected.
the credentials you entered, are they right? might this be a permission issue?
Shouldn't the password be base64_encode'd?
Also try just printing all content, and accept text/html for a bit. Just to see what the error message might be.

Symfony 1.4.2 no content in PHP cURL PUT request

Using PHP cURL and Symfony 1.4.2
I'm trying to do a PUT request including data (JSON) to modify an object in my REST web services, but can't catch the data on the server side.
It seems that the content is attached successfully when checking at my logs:
PUT to http://localhost:8080/apiapp_test.php/v1/reports/498 with post body content=%7B%22report%22%3A%7B%22title%22%3A%22The+title+has+been+updated%22%7D%7D
I attached the data like this:
$curl_opts = array(
CURLOPT_HTTPHEADER => $headers,
CURLOPT_POSTFIELDS => http_build_query(array('content' => $post_data)),
);
And wanted to get the data using something like this
$payload = $request->getPostParameter('content');
It is not working and I've tried many ways to get this data in my actions file.
I've tried the following solutions:
parse_str(file_get_contents("php://input"), $post_vars);
$payload = $post_vars['content'];
// or
$data = $request->getContent(); // $request => sfWebRequest
$payload = $data['content'];
// or
$payload = $request->getPostParameter('content');
// then I'd like to do that
$json_array = json_decode($payload, true);
I just don't know how to get this data in my actions and it's frustrating, I've read many topics here about it but none is working for me.
Additional informations:
I have these setup for my cURL request:
curl_setopt($curl_request, CURLOPT_CUSTOMREQUEST, $http_method);
if ($http_method === sfRequest::PUT) {
curl_setopt($curl_request, CURLOPT_PUT, true);
$content_length = array_key_exists(CURLOPT_POSTFIELDS, $curl_options) ? strlen($curl_options[CURLOPT_POSTFIELDS]) : 0;
$curl_options[CURLOPT_HTTPHEADER][] = 'Content-Length: ' . $content_length;
}
curl_setopt($curl_request, CURLOPT_URL, $url);
curl_setopt($curl_request, CURLOPT_CONNECTTIMEOUT, 4);
curl_setopt($curl_request, CURLOPT_TIMEOUT, 4);
curl_setopt($curl_request, CURLOPT_DNS_CACHE_TIMEOUT, 0);
curl_setopt($curl_request, CURLOPT_NOSIGNAL, true);
curl_setopt($curl_request, CURLOPT_RETURNTRANSFER, true);
In sfWebRequest.php, I've seen this:
case 'PUT':
$this->setMethod(self::PUT);
if ('application/x-www-form-urlencoded' === $this->getContentType())
{
parse_str($this->getContent(), $postParameters);
}
break;
So I tried to set the header's Content-Type to it but it doesn't do anything.
If you have any idea, please help!
According to an other question/answer, I've tested this solution and I got the correct result:
$body = 'the RAW data string I want to send';
/** use a max of 256KB of RAM before going to disk */
$fp = fopen('php://temp/maxmemory:256000', 'w');
if (!$fp) {
die('could not open temp memory data');
}
fwrite($fp, $body);
fseek($fp, 0);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PUT, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_INFILE, $fp); // file pointer
curl_setopt($ch, CURLOPT_INFILESIZE, strlen($body));
$output = curl_exec($ch);
echo $output;
die();
And on the other side, you can retrieve the content using:
$content = $request->getContent();
If you var_dump it, you will retrieve:
the RAW data string I want to send

Categories