So here is the deal : I have been working on a huge system (PHP) for a couple years, and now, I decided to give up part of heavy jobs for golang scripts.
So far, I replicated a few php scripts to a go version. Then, I am able to benchmark which option is better ( ok, I know go is faster, but I need curl or sockets to comunication, so, I have to check if it is still worth ) .
One of the scripts just generate a random code, check if this new code is already in use ( on mysql db ), if not, record the new code and return it, if is already in use, just recursive call the function again until find an exclusive code. pretty simple one.
I already had this code generator in php, so, wrote new one in go to be called as http/post with json params.
Using linux terminal, I call it as
curl -H [headers] -d [jsondata] [host]
and I get back a pretty simple json
{"locator" : "XXXXXX"}
After, I wrote a simple php script to call the scripts and check how long each took to complete, something like :
<?php
public function indexAction()
{
$phpTime = $endPHP = $startPHP =
$goTime = $endGO = $startGO = 0;
// my standard function already in use
ob_start();
$startPHP = microtime(true);
$MyCodeLocator = $this->container->get('MyCodeLocator')->Generate(22, 5);
$endPHP = microtime(true);
ob_end_clean();
sleep(1);
// Lets call using curl
$data = array("comon" => "22", "lenght" => "5");
$data_string = json_encode($data);
ob_start();
$startGO = microtime(true);
$ch = curl_init('http://localhost:8888/dev/fibootkt/MyCodeGenerator');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POSTFIELDS, $data_string);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Content-Length: ' . strlen($data_string))
);
$result = curl_exec($ch);
curl_close($ch);
$endGO = microtime(true);
ob_end_clean();
$result_rr = json_decode($result);
// tst just echo the variable in a nice way, second parameter means no kill execution
tst($result, 1); // HERE IS MY PROBLEM, please read below
tst($result_rr, 1); // IT SHOW NULL
sleep(1);
// just to have params for comparision, always try by shell script
ob_start();
$startShell = microtime(true);;
$exec = "curl -H \"Content-Type: application/json\" -d '{\"comon\":\"22\"}' http://localhost:8888/dev/fibootkt/MyCodeGenerator";
$output = shell_exec($exec);
$endShell = microtime(true);
ob_end_clean();
tst(json_decode($output),1); // here show a stdclass with correct value
tst($output,1); // here shows a json typed string ready to be converted
// and here it is just for show the execution time ...
$phpTime = $endPHP - $startPHP;
$goTime = $endGO - $startGO ;
$shellTime = $endShell - $startShell;
tst("php " . $phpTime, 1);
tst("curl ". $goTime, 1);
tst("shell " . $shellTime, 1);
And I get the results from GO :
By Shell Script :
{"locator" : "DPEWX22"}
So, this one is pretty and easy decode to a stdobj.
But, using curl, the operation is faster ! So, I want to use it.
But, the curl request responds something like :
{"Value":"string","Type":{},"Offset":26,"Struct":"CodeLocatorParams","Field":"lenght"}
{"locator":"DPEWX22"}
And when I try to decode it, I get a null as result !!!
CodeLocatorParams the struct type I use in go to get the post params, as show below
so, here is my question : Why GO is returning this ? how to avoid it.
I have another similar go script which take no params and responds a similar json ( but in this case, a qrcode image path ) and it works fine !
My go function:
type CodeLocatorParams struct {
Comon string `json:"comon"`
Lenght int `json:"lenght"`
}
func Generate(w http.ResponseWriter, r *http.Request) {
data, err := ioutil.ReadAll(io.LimitReader(r.Body, 1048576))
if err != nil {
fmt.Println(err)
panic(err)
}
if err := r.Body.Close(); err != nil {
panic(err)
}
// retrieve post data and set it to params which is CodeLocatorParams type
var params CodeLocatorParams
if err := json.Unmarshal(data, ¶ms ); err != nil {
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
w.WriteHeader(422) // unprocessable entity
if err := json.NewEncoder(w).Encode(err); err != nil {
fmt.Println(err)
panic(err)
}
}
var result struct{
Locator string `json:"locator"`
}
// here actually comes the func that generates random code and return it as string, but for now, just set it static
result.Locator = "DPEWX22"
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
json.NewEncoder(w).Encode(result)
}
There is an error parsing the incoming JSON. This error is written to the response as {"Value":"string","Type":{},"Offset":26,"Struct":"CodeLocatorParams","Field":"lenght"}. The handler continues to execute and writes the normal response {"locator":"DPEWX22"}.
Here's how what to fix:
After writing error response, return from the handler.
The input JSON has lenght as a string value. Either change the struct field from int to string or modify the input to pass an integer.
Related
I'm new to JSON Code. I want to learn about the update function. Currently, I successfully can update data to the database. Below is the code.
<?php
require_once "../config/configPDO.php";
$photo_after = 'kk haha';
$report_id = 1;
$url = "http://172.20.0.45/TGWebService/TGWebService.asmx/ot_maintainReport?taskname=&reportStatus=&photoBefore=&photoAfter=". urlencode($photo_after) . "&reportID=$report_id";
$data = file_get_contents($url);
$json = json_decode($data);
$query = $json->otReportList;
if($query){
echo "Data Save!";
}else{
echo "Error!! Not Saved";
}
?>
the problem is, if the value of $photo_after is base64 string, which is too large string, it will give the error:
1) PHP Warning: file_get_contents.....
2) PHP Notice: Trying to get property 'otReportList' of non-object in C:
BUT
when I change the code to this,
<?php
require_once "../config/configPDO.php";
$photo_after = 'mama kk';
$report_id = 1;
$sql = "UPDATE ot_report SET photo_after ='$photo_after', time_photo_after = GETDATE(), ot_end = '20:30:00' WHERE report_id = '$report_id'";
$query = $conn->prepare($sql);
$query->execute();
if($query){
echo "Data Save!";
}else{
echo "Error!! Not Saved";
}
?>
The data will updated including when the value of $photo_after is in base 64 string.
Can I know what is the problem? Any solution to allow the base64 string update thru json link?
Thanks
// ...
// It's likely that the following line failed
$data = file_get_contents($url);
// ...
If the length of $url is more than 2048 bytes, that could cause file_get_contents($url) to fail. See What is the maximum length of a URL in different browsers?.
Consequent to such failure, you end up with a value of $json which is not an object. Ultimately, the property otReportList would not exist in $json hence the error: ...trying to get property 'otReportList' of non-object in C....
To surmount the URL length limitation, it would be best to embed the value of $photo_after in the request body. As requests made with GET method should not have a body, using POST method would be appropriate.
Below is a conceptual adjustment of your code to send the data with a POST method:
<?php
require_once "../config/configPDO.php";
# You must adapt backend behind this URL to be able to service the
# POST request
$url = "http://172.20.0.45/TGWebService/TGWebService.asmx/ot_maintainReport";
$report_id = 1;
$photo_after = 'very-long-base64-encoding-of-an-image';
$request_content = <<<CONTENT
{
"taskname": $taskname,
"report_id": $report_id,
"photoBefore": $photoBefore,
"photo_after": $photo_after,
"reportStatus": $reportStatus
}
CONTENT;
$request_content_length = strlen($request_content);
# Depending on your server configuration, you may need to set
# $request_headers as an associative array instead of a string.
$request_headers = <<<HEADERS
Content-type: application/json
Content-Length: $request_content_length
HEADERS;
$request_options = array(
'http' => array(
'method' => "POST",
'header' => $request_headers,
'content' => $request_content
)
);
$request_context = stream_context_create($request_options);
$data = file_get_contents($url, false, $request_context);
# The request may fail for whatever reason, you should handle that case.
if (!$data) {
throw new Exception('Request failed, data is invalid');
}
$json = json_decode($data);
$query = $json->otReportList;
if ($query) {
echo "Data Save!";
} else {
echo "Error!! Not Saved";
}
?>
sending a long GET URL is not a good practice. You need to use POST method with cURL. And your webservice should receive the data using post method.
Here's example sending post using PHP:
//
// A very simple PHP example that sends a HTTP POST to a remote site
//
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://www.example.com/tester.phtml");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,
"postvar1=value1&postvar2=value2&postvar3=value3");
// In real life you should use something like:
// curl_setopt($ch, CURLOPT_POSTFIELDS,
// http_build_query(array('postvar1' => 'value1')));
// Receive server response ...
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$server_output = curl_exec($ch);
curl_close ($ch);
// Further processing ...
if ($server_output == "OK") { ... } else { ... }
Sample code from: PHP + curl, HTTP POST sample code?
And all output from the webservice will put in the curl_exec() method and from there you can decode the replied json string.
The issue:
I'm working with PHP, cURL and a public API to fetch json strings.
These json strings are formatted like this (simplified, average size is around 50-60 kB):
{
"data": {},
"previous": "url",
"next": "url"
}
What am trying to do is fetch all the json strings starting from the first one by checking for the "next" attribute. So I have a while loop and as long as there's a "next" attribute, I fetch the next URL.
The problem is sometimes, randomly the loop stops before the end and I cannot figure out why after many tests.
I say randomly because sometimes the loop goes through to the end and no problem occurs. Sometimes it crashes after N loops.
And so far I couldn't extract any information to help me debug it.
I'm using PHP 7.3.0 and launching my code from the CLI.
What I tried so far:
Check the headers:
No headers are returned. Nothing at all.
Use curl_errno() and curl_error():
I tried the following code right after executing the request (curl_exec($ch)) but it never triggers.
if(curl_errno($ch)) {
echo 'curl error ' . curl_error($ch) . PHP_EOL;
echo 'response received from curl error :' . PHP_EOL;
var_dump($response); // the json string I should get from the server.
}
Check if the response came back null:
if(is_null($response))
or if my json string has an error:
if(!json_last_error() == JSON_ERROR_NONE)
Though I think it's useless because it will never be valid if the cURL response is null or empty. When this code triggers, the json error code is 3 (JSON_ERROR_CTRL_CHAR)
The problematic code:
function apiCall($url) {
...
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
}
$inc = 0;
$url = 'https://api.example.com/' . $id;
$jsonString = apiCall($url);
if(!is_null($jsonString)) {
file_put_contents('pathToDirectory/' . $id + $inc, $jsonString);
$nextUrl = getNextUrl($jsonString);
while ($nextUrl) {
$jsonString = apiCall($url . '?page=' . $nextUrl);
if(!is_null($jsonString)) {
$inc++;
file_put_contents('pathToDirectory/' . $id + $inc, $jsonString);
$nextUrl = getNextUrl($jsonString);
}
}
}
What I'm expecting my code to do:
Not stop randomly, or at least give me a clear error code.
The problem is that your API could be returning an empty response, a malformed JSON, or even a status code different of 200 and you would stop execution imediately.
Since you do not control the API responses, you know that it can fail randomly, and you do not have access to the API server logs (because you don't, do you?); you need to build some kind of resilience in your consumer.
Something very simple (you'd need to tune it up) could be
function apiCall( $url, $attempts = 3 ) {
// ..., including setting "$headers"
$ch = curl_init();
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_HTTPHEADER, $headers );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
for ( $i = 0; $i < $attempts; $i ++ ) {
$response = curl_exec( $ch );
$curl_info = curl_getinfo( $ch );
if ( curl_errno( $ch ) ) {
// log your error & try again
continue;
}
// I'm only accepting 200 as a status code. Check with your API if there could be other posssible "good" responses
if ( $curl_info['http_code'] != 200 ) {
// log your error & try again
continue;
}
// everything seems fine, but the response is empty? not good.
if ( empty( $response ) ) {
// log your error & and try again
continue;
}
return $response;
}
return null;
}
This would allow you to do something like (pulled from your code):
do {
$jsonString = apiCall($url . '?page=' . $nextUrl);
$nextUrl = false;
if(!is_null($jsonString)) {
$inc++;
file_put_contents('pathToDirectory/' . $id + $inc, $jsonString);
$nextUrl = getNextUrl($jsonString);
}
}
while ($nextUrl);
I'm not checking if the return from the API is non-empty, not a connection error, a status different from '200' and yet an invalid JSON.
You may want to check for these things as well, depending on how brittle the API you are consuming is.
I am trying to implement Vault of Satoshi's API in Google App Engine Go. Their reference API is in PHP:
<?php
$serverURL = 'https://api.vaultofsatoshi.com';
$apiKey = 'ENTER_YOUR_API_KEY_HERE';
$apiSecret = 'ENTER_YOUR_API_SECRET_HERE';
function usecTime() {
list($usec, $sec) = explode(' ', microtime());
$usec = substr($usec, 2, 6);
return intval($sec.$usec);
}
$url = 'https://api.vaultofsatoshi.com';
$endpoint = '/info/currency';
$url = $serverURL . $endpoint;
$parameters= array();
$parameters['nonce'] = usecTime();
$data = http_build_query($parameters);
$httpHeaders = array(
'Api-Key: ' . $apiKey,
'Api-Sign:' . base64_encode(hash_hmac('sha512', $endpoint . chr(0) . $data, $apiSecret)),
);
// Initialize the PHP curl agent
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, "something specific to me");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $httpHeaders);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
?>
My Go code looks like this:
func GenerateSignatureFromValues(secretKey string, endpoint string, values url.Values) string {
query:=[]byte(values.Encode())
toEncode:=[]byte(endpoint)
toEncode = append(toEncode, 0x00)
toEncode = append(toEncode, query...)
key:=[]byte(secretKey)
hmacHash:=hmac.New(sha512.New, key)
hmacHash.Write(toEncode)
answer := hmacHash.Sum(nil)
return base64.StdEncoding.EncodeToString(([]byte(strings.ToLower(hex.EncodeToString(answer)))))
}
func Call(c appengine.Context) map[string]interface{} {
serverURL:="https://api.vaultofsatoshi.com"
apiKey:="ENTER_YOUR_API_KEY_HERE"
apiSecret:="ENTER_YOUR_API_SECRET_HERE"
endpoint:="/info/order_detail"
tr := urlfetch.Transport{Context: c}
values := url.Values{}
values.Set("nonce", strconv.FormatInt(time.Now().UnixNano()/1000, 10))
signature:=GenerateSignatureFromValues(apiSecret, endpoint, values)
req, _:=http.NewRequest("POST", serverURL+endpoint, nil)
req.Form=values
req.Header.Set("Api-Key", apiKey)
req.Header.Set("Api-Sign", signature)
resp, err:=tr.RoundTrip(req)
if err != nil {
c.Errorf("API post error: %s", err)
return nil
}
defer resp.Body.Close()
body, _:= ioutil.ReadAll(resp.Body)
result := make(map[string]interface{})
json.Unmarshal(body, &result)
return result
}
Both of those pieces of code generate the same signature for the same input. However, when I run the PHP code (with the proper Key and Secret), the server responds with a proper response, but while I run the Go code, the server responds with "Invalid signature". This error indicates that the HTTP request generated by Go must be somehow malformed - either HTTP Header's values are wrong (if the header values are completely missing a different error appears), or the way the POST fields are encoded is wrong for some reason.
Can anyone help me find some reason why those two pieces of code generate different HTTP requests and how can I make Go generate requests like the PHP code?
See the documentation for Request.Form:
// Form contains the parsed form data, including both the URL
// field's query parameters and the POST or PUT form data.
// This field is only available after ParseForm is called.
// The HTTP client ignores Form and uses Body instead.
Form url.Values
Specifically "HTTP client ignores Form and uses Body instead."
With this line:
req, _:= http.NewRequest("POST", serverURL+endpoint, nil)
You should use this instead of nil:
bytes.NewBufferString(values.Encode())
Also keep in mind that the order of map is not guaranteed. url.Values is map[string][]string. So you should be using Encode() once and use the same result in the body and signature. There is a chance that by using Encode() twice the order could be different. This is an important difference between Go and PHP.
You should also make a habit of handling error instead of ignoring it.
I wanna run a GET curl in php A to get data from php B.
This is an example in php A (I got from here http://support.qualityunit.com/061754-How-to-make-REST-calls-in-PHP)
//next example will recieve all messages for specific conversation
$service_url = 'http://localhost/test/getFrom.php?id=1';
$curl = curl_init($service_url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$curl_response = curl_exec($curl);
if ($curl_response === false) {
$info = curl_getinfo($curl);
curl_close($curl);
die('error occured during curl exec. Additioanl info: ' . var_export($info));
}
curl_close($curl);
$decoded = json_decode($curl_response);
if (isset($decoded->response->status) && $decoded->response->status == 'ERROR') {
die('error occured: ' . $decoded->response->errormessage);
}
echo 'response ok!';
var_export($decoded->response);
And I tried this example as well (Trying to use curl to do a GET, value being sent is allows null)
In php B.
It will get the ID, run some script and will generate an ARRAY.
I want to get this ARRAY from B to A.
B will run only when A request GET from B.
The problem is I don't how the ARRAY can pass from B to A.
Please give some advice THANK YOU.
The code you provided is expecting back a JSON encoded array. The easiest way to do this is simply JSON encode your array in PHP B and echo it to the page.
Then CURL will be able to read the contents of PHP B, decode and process as required.
// PHP B
<?php
// Check for $_GET params
// Get ID
$id = $_GET['id'];
// Do processing, query etc
....
// Format and display array as JSON
echo(json_encode($result_array));
die();
?>
Also note:
if (isset($decoded->response->status) && $decoded->response->status == 'ERROR') {
die('error occured: ' . $decoded->response->errormessage);
}
The code is expecting the array to be formatted in a particular way. So either match your array in PHP B to the same format, or update the code to your needs.
API integration description
The API needs a form to be posted to the API URL with some input fields and a customer token. The API processes and then posts response to a callback.php file on my server. I can access the posted vals using $_POST in that file. That's all about the existing method and it works fine.
Requirement
To hide the customer token value from being seen from client side. So I started with sending server side post request.
Problem
I tried with many options but the callback is not happening -
1) CURL method
$ch = curl_init(API_URL);
$encoded = '';
$_postArray['customer_token'] = API_CUSTOMER_TOKEN;
foreach($_postArray as $name => $value)
{
$encoded .= urlencode($name).'='.urlencode($value).'&';
}
// chop off last ampersand
$encoded = substr($encoded, 0, strlen($encoded)-1);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $encoded);
$resp = curl_exec($ch);
curl_close($ch);
echo $resp;
$resp echoes 1 if the line curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); is removed but the callback does not happen. I am setting a session variable in the callback script to verify.Is it needed that the API be synchronous in order to use curl method, so that curl_exec returns the response?
2) without CURL as given in Posting parameters to a url using the POST method without using a form
But the callback is not happening.
I tried with the following code too, but looks like my pecl is not installed properly because the HttpRequest() is not defined.
$req = new HttpRequest($apiUrl, HttpRequest::METH_POST);
$req->addQueryData($params);
try
{
$r->send();
if ($r->getResponseCode() == 200)
{
echo "success";
// success!
}
else
{
echo "failure";
// got to the API, the API returned perhaps a RESTful response code like 404
}
}
catch (HttpException $ex)
{
// couldn't get to the API (probably)
}
Please help me out! I just need to easily send a server side post request and get the response in the callback file.
Try to debug your request using the curl_get_info() function:
$header = curl_getinfo($ch);
print_r($header);
Your request might be OK but it my result in an error 404.
EDIT: If you want to perform a post request, add this to your code:
curl_setopt($ch, CURLOPT_POST, true);
EDIT: Something else I mentioned at your code: You used a '1' at the 'CURLOPT_RETURNTRANSFER' but is should be 'true':
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
At least this is how I usually do it, and you never know if the function will also understand a '1' as 'true';
EDIT: The real problem: I copy-pasted your source and used it on one of my pages getting this error:
Warning: urlencode() expects parameter 1 to be string, array given in C:\xampp\htdocs\phptests\test.php on line 8
The error is in this line:
foreach($_postArray as $name => $value)
$_postArray is an array with one value holding the other values and you need either another foreach or you simple use this:
foreach($_postArray['customer_token'] as $name => $value)
As discussed in the previous question, the callback is an entirely separate thing from your request. The callback also will not have your session variables, because the remote API is acting as the client to the callback script and has its own session.
You should really show some API documentation here. Maybe we're misunderstanding each other but as far as I can see, what you are trying to do (get the callback value in the initial CURL request) is futile, and doesn't become any less futile by asking twice.