Elasticsearch Bulk data insertion - JsonParseException[Unexpected character - Using PHP - php

While indexing the bulk data using PHP curl method I am getting the exception as "error":"JsonParseException[Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')\n at [Source: [B#14abb68; line: 1, column: 18]]","status":500}
Kindly find below code which I am using for the same and do let me know what could be wrong over here.
<?php
$ch = curl_init();
$method = "POST";
$url = "http://192.168.1.204/myindex/test/_bulk";
$qry = '
{"index":{"_index": "myindex","_type":"test"}}
{
"product_id": 1,
"title": "mobile"
}
{"index":{"_index": "myindex","_type":"test"}}
{
"product_id": 2,
"title": "laptop",
}
';
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PORT, 9200);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, strtoupper($method));
curl_setopt($ch, CURLOPT_POSTFIELDS, $qry);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
?>

You are not using the correct format (and there is an additional comma at the second element).
It should be:
action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n
Extracted from the documentation of elasticsearch - bulk
So, you should add the data using the following format:
{"index":{"_index": "myindex","_type":"test"}}
{"product_id": 1,"title": "mobile"}
{"index":{"_index": "myindex","_type":"test"}}
{"product_id": 2,"title": "laptop"}

Related

PHP cURL with CURLOPT_POSTFIELDS in json format

I'm not quite familiar with cURL and been struggling on this issue though.
I'm sending a post request by using cURL and the data field is in json format.
My coding:
$ch = curl_init($url);
$authorization = "Authorization: Bearer ".$token;
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json' , $authorization ));
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = "test";
$data = <<<DATA
{
"shop_id": 1231232,
"message": "test"
}
DATA;
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
What I'm trying to achieve here is to replace the "message" value with php variable $response. What is the correct way to do that? Thank you!
This has nothing to do with cURL specifically, it's about including variables in a data structure or string.
My first piece of advice would be: don't generate JSON string literals in your code, it's fiddly and potentially error-prone. Build a suitable PHP data structure and then use json_encode() to reliably turn it into a valid JSON string.
As it happens, following this approach also makes your current requirement very easy to achieve using basic PHP:
$data = [
"shop_id" => 1231232,
"message" => $response
];
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data));
P.S. In a different scenario (e.g. not trying to generate JSON) where using heredoc would be more appropriate, the solution is also simple, you just replace the hard-coded value with the variable:
$data = <<<DATA
{
"shop_id": 1231232,
"message": $response
}
DATA;

Sending array using curl

I want to send data from server 1 to server 2, first I select necessary data from the database, but how to send data with curl? I understand that I cannot send $result parameter just like in my code, but how should I do this?
My Code server 1:
public function setDivisions(){
$result = $this->conn->query("SELECT *FROM data_divisions");
$ch = curl_init('https://example.com/api.php?siteid='.$this->site_key.'');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $result);
curl_setopt($ch, CURLOPT_POST, 1);
$response = curl_exec($ch);
print_r($response);
}
Code on server 2:
$array = $_POST['result'];
//loop throw array and insert data into database
you can use it that way.
$ch = curl_init('https://upxxx.cod3fus1ontm.com/curl/json');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode((object)["records" => json_encode($result)]));
$response = curl_exec($ch);
var_dump($response);
on receipt, like this!
$json = file_get_contents("php://input");
$content = json_decode($json, true);
$records = json_decode($content['records'], true);
foreach($records as $record) {
echo $record['id'] . " - " . $record['text'] . "<br/>";
}
remember, that as you did to encode, you will have to do to decode
Come on, php://input returns all raw data after the request's HTTP headers, regardless of content type.
When we do this with file_get_contents (which reads the contents of a file and puts it into a variable of type string), we can read the content that was sent by curl.
Reviewing the code, you can optimize as follows, when sending to the server you placed, I suggested:
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode((object)["records" => json_encode($result)]));
you can replace it with:
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($result));
it's much simpler, but it's very important to put the query result inside a json_encode
finally, you can access the entire body of the message, through file_get_contents ("php://input") and put it inside a variable, which is nothing more than a JSON of the your query.
for you to understand how the parameters were passed, it is interesting to do the following:
$json = file_get_contents("php: // input");
var_dump($json); <- Here you see the thing as it is.
$records = json_decode($json, true); <- Here you generate an object with the content of json
var_dump($records);
With that, I think that solves the situation.
on server 1
$result = "result=".base64_encode($result)
curl_setopt($ch, CURLOPT_POSTFIELDS, $result);
...
on server 2
$array = base64_decode($_POST['result']);

Binance 400 response and duplicate values for a parameter detected

still trying to get my head around API's and connections.
I am trying to do a buy order to Binance but keep getting a 400 response. I know it's been an issue for some others but I just can't seem to suss out where I am going wrong. So hoping someone can help me out.
It's only issues with POST's, all the GET requests are working fine, and so to the signature and timestamp... or at least I am getting responses for my account so I assume so.
The first CURL is just for the serverTime, but the second CURL is for a buy/sell order.
This is the response I am getting now...
"Object ( [code] => -1102 [msg] => Mandatory parameter 'side' was not
sent, was empty/null, or malformed. )"
If I type the in the string manually it works just fine, but for one reason or another when I pass $qs it presents the above fault. I echoed $qs to the screen and copied that instead of passing $qs and it worked when I put in the new timestamp. I am stumped...
"symbol=TRXUSDC&side=SELL&type=LIMIT&timeInForce=GTC&quantity=63000.00000000&price=0.02550000&recvWindow=1000000&timestamp=1550922237310"
any advice?
$header = array('X-MBX-APIKEY:' . KEY);
$url = BINANCE . 'api/v1/time';
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($curl, CURLOPT_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_ENCODING, "");
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($curl, CURLOPT_URL,$url);
$response = curl_exec($curl);
if (FALSE === $response){
echo curl_error($curl), curl_errno($curl);
}
$serverTime = json_decode($response);
curl_close($curl);
$url = BINANCE . "api/v3/order";
$signature = NULL;
$queryString = NULL;
$query = array(
"symbol" => "TRXUSDC",
"side" => "SELL",
"type" => "LIMIT",
"timeInForce" => "GTC",
"quantity" => number_format(63000.00000000,8,'.',''),
"price" => number_format(0.02550000,8,'.',''),
"recvWindow" => 1000000,
"timestamp" => $serverTime->serverTime);
$qs = htmlentities(http_build_query(array_unique($query)));
$query['signature'] = hash_hmac('SHA256', $qs, SECRET );
$curl = curl_init();
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($curl, CURLOPT_TIMEOUT, 60);
curl_setopt($curl, CURLOPT_ENCODING, "application/x-www-form-urlencoded");
curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl,CURLOPT_FAILONERROR,FALSE);
curl_setopt($curl, CURLOPT_VERBOSE, TRUE);
curl_setopt($curl, CURLOPT_POST, TRUE);
curl_setopt($curl, CURLOPT_POSTFIELDS,$qs);
curl_setopt($curl, CURLOPT_URL, $url);
$response = curl_exec($curl);
if (FALSE === $response){
echo curl_error($curl).':'.curl_errno($curl);
}
$obj = json_decode($response);
curl_close($curl);
I tried this code and var_dump $qs
symbol=BTCUSDT&side=BUY&...
so I think you must remove amp;
Based on Binance spot api docs:
It is recommended to use a small recvWindow of 5000 or less! The max cannot go beyond 60,000!
so edit your query params with:
"symbol=TRXUSDC&side=SELL&type=LIMIT&timeInForce=GTC&quantity=63000.00000000&price=0.02550000&recvWindow=60000&timestamp=1550922237310"
Also you can check your other query params with exchange info that you can get with https://api.binance.com/api/v3/exchangeInfo that we have for your symbol:
{
"symbol": "TRXUSDC",
"status": "TRADING",
"baseAsset": "TRX",
"baseAssetPrecision": 8,
"quoteAsset": "USDC",
"quotePrecision": 8,
"quoteAssetPrecision": 8,
"baseCommissionPrecision": 8,
"quoteCommissionPrecision": 8,
"orderTypes": [
"LIMIT",
"LIMIT_MAKER",
"MARKET",
"STOP_LOSS_LIMIT",
"TAKE_PROFIT_LIMIT"
],
"icebergAllowed": true,
"ocoAllowed": true,
"quoteOrderQtyMarketAllowed": true,
"isSpotTradingAllowed": true,
"isMarginTradingAllowed": false,
"filters": [
{
"filterType": "PRICE_FILTER",
"minPrice": "0.00001000",
"maxPrice": "1000.00000000",
"tickSize": "0.00001000"
},
{
"filterType": "PERCENT_PRICE",
"multiplierUp": "5",
"multiplierDown": "0.2",
"avgPriceMins": 5
},
{
"filterType": "LOT_SIZE",
"minQty": "0.10000000",
"maxQty": "9000000.00000000",
"stepSize": "0.10000000"
},
{
"filterType": "MIN_NOTIONAL",
"minNotional": "10.00000000",
"applyToMarket": true,
"avgPriceMins": 5
},
{
"filterType": "ICEBERG_PARTS",
"limit": 10
},
{
"filterType": "MARKET_LOT_SIZE",
"minQty": "0.00000000",
"maxQty": "659177.02430556",
"stepSize": "0.00000000"
},
{
"filterType": "MAX_NUM_ORDERS",
"maxNumOrders": 200
},
{
"filterType": "MAX_NUM_ALGO_ORDERS",
"maxNumAlgoOrders": 5
}
],
"permissions": [
"SPOT"
]
}
for other possible reasons check spot api docs in the link https://github.com/binance/binance-spot-api-docs/blob/master/rest-api.md
I tested the code and saw a few glitches in it :
$qs = htmlentities(http_build_query(array_unique($query)));
gave me an error ({"code":-1102,"msg":"Mandatory parameter 'side' was not sent, was empty/null, or malformed."}), so i replaced it with:
$qs = http_build_query(array_unique($query));
After that you created your signature, but never added it to you request. So do the following :
$signature = hash_hmac('sha256', $qs, $apiSecret);
$qs .= "&signature=" . $signature;
Also update your 'recvWindow' to a max of 60000 (I suggest 50000 or less (see the answer of #Tofiq Samali)) and you are ready to go!

json responses and assigning a name prior to decode

Good Afternoon.
I am successfully uploading property data to the Rightmove test site using json to create $data (see below) and curl to transfer it. This is the curl script which works perfectly well and results in the expected success message form Rightmove..see below....
$url = 'abc,com/def.php';
$ch = curl_init();
$cert = "xyz.pem";
if (!stat($cert))
{
die('No cert');
}
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSLCERT, $cert);
curl_setopt($ch, CURLOPT_SSLCERTPASSWD, 'hidden');
curl_setopt($ch, CURLOPT_SSLVERSION, 5);
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
if($result = curl_exec($ch) === false)
{
die('Curl error: ' . curl_error($ch));
}
curl_close($ch);
However we need to be able to read the json success message (response) we are getting back (see example below) into a php script for further scripting our end. We know we can use decode to do that. So that isnt the question. Read on....
Here is an example of the json string success message we are getting back from Rightmove - all exactly as expected.
{
"message": "Your property has successfully been uploaded to Rightmove",
"success": true,
"errors": null,
"warnings": [],
"request_timestamp": "02-08-2016 13:07:50",
"response_timestamp": "02-08-2016 13:07:50",
"request_id": "231301-160802-130750-10990",
"property": {
"agent_ref": "agent950",
"rightmove_id": 123456789,
"change_type": "UPDATE",
"rightmove_url": "http://www.adftest.rightmove.com/property-to-rent/property-123456789.html"
}
}
What we need to test now is how to incorporate the above json response data into our php scripts starting with the json decode snippet below...
$var = json_decode($xxx, true);//outputs and associative array
print_r($var);//prints the array
Once we have the $var object back we can then use the data inside it in our scripts.
The Question
.....literally all we need to know is what is $xxx (ie what is the name of the json string success message response so we can add it to the decode snippet above and be on our way. Or how do we set $xxx. Rightmove couldnt help.
I suspect we are maybe missing the obvious here and I am going to cringe when you tell me the answer.
Thanks in anticipation

Edit json response from curl response

<?php
$json_url = "http://openexchangerates.org/api/latest.json?app_id=xxxxx&callback=angular.callbacks._0";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $json_url);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
$response = curl_exec($curl);
$jsonString = json_encode($response, true);
$data=json_decode($jsonString);
echo '<pre>',print_r($data),'</pre>';
$status = curl_getinfo($curl);
curl_close($curl);
The output is:
angular.callbacks._0({
"disclaimer": "xx",
"license": "xx",
"timestamp": 1368136869,
"base": "USD",
"rates": {
"AED": 3.672819,
"AFN": 53.209,
"ALL": 107.953875,
"AOA": 96.358934,
"ARS": 5.214887,
....
"XOF": 501.659003,
"XPF": 91.114876,
"ZMK": 5227.108333,
"ZMW": 5.314783,
"ZWL": 322.387247
}
})
But i need to edit this output to this one (only with three rates (AED/AFN/AOA)). So, basically edit the json response in the section of rates. How can i do that?
angular.callbacks._0({
"disclaimer": "xx",
"license": "xx",
"timestamp": 1368136869,
"base": "USD",
"rates": {
"AED": 3.672819,
"AFN": 53.209,
"AOA": 107.953875,
}
})
The $response is not in correct json format:
You must remove the unnecessary parts from it:
$jsonString= substr($response, 21, -1);
You can do:
<?php
$json_url = "http://openexchangerates.org/api/latest.json?app_id=5bf388eb6f7e40209b9418e6be44f04b&callback=angular.callbacks._0";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $json_url);
curl_setopt($curl, CURLOPT_HEADER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
$response = curl_exec($curl);
$jsonString= substr($response, 21, -1);
$data=json_decode($jsonString, true);
// Get the list of rates
$rates = $data['rates'];
$new_rates = array();
$new_rates['AED'] = $rates['AED'];
$new_rates['AFN'] = $rates['AFN'];
$new_rates['AOA'] = $rates['AOA'];
$data['rates'] = $new_rates;
echo 'angular.callbacks._0('.json_encode($data).')';
$status = curl_getinfo($curl);
curl_close($curl);
It's returning a response for jsonp. You would need that on the client side, but not the server. You might try:
http://openexchangerates.org/api/latest.json?app_id=xxxxx
(i.e. omit the callback). Otherwise take a look at their API to see if you can format the request differently to receive pure JSON back without the callback.
If you absolutely can't, then you can just strip off the bad parts:
preg_match('/{.*}/', $response, $matches);
$json = json_decode($matches[0]);
Once you have the valid JSON, it's a simple matter of unsetting:
unset($json->rates->XOF);
//etc.
You could also just write the content you need to some other object if it's easier.
You may want to take a look at the API to see if you can get only specific rates back.
Read the JSON string into a PHP mapping array with json_decode, pick only the elements you need, compose a new array with them and finally reserialize it with json_encode.
Note: this method is more robust than a regex/string based one, as it makes sense of what it's being processed, thus allowing for mutation in the structure of the JSON, as long as the elements you need are at their places.
For selecting elements, use:
$keep = array("AED", "AFN", "AOA");
$data["rates"] = array_intersect_key($data["rates"], array_flip($keep));

Categories