I am trying to save the subject of incoming emails to a database. The subject is not always encoded using the same encoding, so I made this code to convert it back to utf-8 when it's not.
private function convertSubjectEncoding($subject)
{
$encoding = mb_detect_encoding($subject);
if($encoding != 'UTF-8') {
return iconv_mime_decode($subject, 0, "UTF-8");
}
return $subject;
}
For the first message, Encoding is UTF-8 and the subject is Accès SQL. When it is saved to database, it becomes "Acce`s SQL" which is wrong and should be "Accès SQL".
For the second message, the subject is Ascii and the original subject is "=?utf-8?Q?Acc=C3=A8s_?=SQL". When converting, and also when saving it is 'Accès SQL' which is good.
Why is that a string that was originally formatted as ut8 and did not get any encoding change suddenly becomes a different string when saved?
I am using Laravel 6.
Here is the full relevant code:
const SUBJECT_REPLY_FORWARD_REGEX = "/([\[\(] *)?\b(RE|FWD?) *([-:;)\]][ :;\])-]*|$)|\]+ *$/im";
private function createFetchedMail($message)
{
$toList = $message->getTo();
$fetchedMail = FetchedMail::create([
'OriginalSubject' => $this->convertSubjectEncoding($message->getSubject()),
'Subject' => $this->cropSubject($this->convertSubjectEncoding($message->getSubject())),
]);
/**
* Removes subject reply and forwarding indacator (Re:, FWD:, etc.) and trims the result
*/
private function cropSubject($subject)
{
return trim(preg_replace(static::SUBJECT_REPLY_FORWARD_REGEX, '', $subject));
}
private function convertSubjectEncoding($subject)
{
$encoding = mb_detect_encoding($subject);
if($encoding != 'UTF-8') {
return iconv_mime_decode($subject, 0, "UTF-8");
}
return $subject;
}
I have tried to save directly without calling convertSubjectEncoding() and cropSubject(), I get the same erroneous string saved in database.
Related
I have a web app (built using ExtJS) where I let users update their basic info. I don't have problems with most of the update routine. However, when I tried updating a user with an ñ in their name, the PHP changed this to the Unicode u00f1.
For example, I pass the name Añana, which PHP shows is Au00f1ana, where "u00f1" replaced "ñ".
I already tried setting the charset to utf-8, htmlspecialchars, mb_convert_encoding, utf8-decode, and html_entity_decode, but none worked.
What I did to get around this problem is to use strpos and substr_replace to just replace the utf code with the original character.
if(strpos($my_string, 'u00f1') !== FALSE){
$start_index = strpos($my_string, "u00f1");
$last_name = substr_replace($my_string, "ñ", $start_index, 5);
}
elseif(strpos($my_string, 'u00F1') !== FALSE){
$start_index = strpos($my_string, "u00F1");
$last_name = substr_replace($my_string, "Ñ", $start_index, 5);
}
For more context, this is the store I use:
Ext.define('AppName.store.MyStore', {
extend: 'Ext.data.Store',
requires: [
'AppName.model.model_for_store',
'Ext.data.proxy.Ajax',
'Ext.data.reader.Json',
'Ext.data.writer.Json'
],
constructor: function(cfg) {
var me = this;
cfg = cfg || {};
me.callParent([Ext.apply({
remoteFilter: true,
remoteSort: true,
storeId: 'MyStore',
batchUpdateMode: 'complete',
model: 'AppName.model.model_for_store',
proxy: {
type: 'ajax',
batchActions: false,
api: {
create: 'folder/folder/create.php',
read: 'folder/folder/read.php',
update: 'folder/folder/update.php',
destroy: 'folder/folder/delete.php'
},
url: '',
reader: {
type: 'json',
keepRawData: true,
messageProperty: 'message',
rootProperty: 'data'
},
writer: {
type: 'json',
writeAllFields: true,
encode: true,
rootProperty: 'data'
}
}
}, cfg)]);
}
});
And this is the start of the PHP file that gets triggered for update:
<?php
require_once('../db_init.php');
require_once '../lib/response.php';
require_once '../lib/request.php';
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);
ini_set('default_charset', 'utf-8');
header("content-type: text/html; charset=UTF-8");
error_reporting(E_ALL);
define('CHARSET', 'ISO-8859-1');
define('REPLACE_FLAGS', ENT_COMPAT | ENT_XHTML);
class my_object{
//..... variables
}
$request = new Request(array());
if(isset($request->params)){
$array_r=$request->params;
$inputData->last_name=($array_r->last_name);
$inputData->first_name=($array_r->first_name);
$inputData->middle_name=($array_r->middle_name);
}
else{
//echo "Update of User failed";
$res = new Response();
$res->success = false;
$res->message = "Update User Failed!";
$res->data = array();
print_r($res->to_json());
}
?>
And as a reference, here's my Request.php file
<?php
class Request {
public $method, $params;
public function __construct($params) {
// $this->restful = (isset($params["restful"])) ? $params["restful"] : false;
$this->method = $_SERVER["REQUEST_METHOD"];
$this->parseRequest();
}
protected function parseRequest() {
// grab JSON data if there...
$this->params = (isset($_REQUEST['data'])) ? json_decode(stripslashes($_REQUEST['data'])) : null;
if (isset($_REQUEST['data'])) {
$this->params = json_decode(stripslashes($_REQUEST['data']));
} else {
$raw = '';
$httpContent = fopen('php://input', 'r');
//print_r($httpContent);
while ($kb = fread($httpContent, 1024)) {
$raw .= $kb;
}
$params = json_decode(stripslashes($raw));
if ($params) {
$this->params = $params->data;
}
}
}
}
?>
Although I've checked ExtJS's documentation, and for the encode property in the JSONWriter, it says:
Configure `true` to send record data (all record fields if writeAllFields is true) as a JSON Encoded HTTP Parameter named by the rootProperty configuration.
So my model data is being sent as a JSON Encoded HTTP Parameter, and the PHP is able to parse it because the rootProperty is data, which matches line 19 in my request.php file:
$this->params = (isset($_REQUEST['data'])) ? json_decode(stripslashes($_REQUEST['data'])) : null;
I think the json_decode(stripslashes(...)) is the one causing ñ` to be converted to the utf-8 equivalent, and the stripslashes removed the leading backslash we usually see with the utf-8.
I have a feeling there's a better way to write this that won't convert the ñ to utf-8.
Does anyone have a better solution to this one?
Solved it. The problem was the stripslash in the request.php file.
When ExtJS sends the data through the Store, I use encode, which puts it as an encoded JSON parameter. Normally it wouldn't be a problem when the request.php file gets it and runs json_decode, given that there were no escaped special characters. The problem is when I hit the ñ, it was converted to \u00f1, and the leading slash was trimmed. json_decode presumably did not recognize it again since there was no leading slash.
Major credits to IMSoP in the comments for the lead.
base on TRX documents and some search in GitHub I tried to generate wallet offline and I can't use API for some reasons.
based on Trx documents I should do these steps :
Generate a key pair and extract the public key (a 64-byte byte array representing its x,y coordinates).
Hash the public key using sha3-256 function and extract the last 20 bytes of the result.
Add 0x41 to the beginning of the byte array. The length of the initial address should be 21 bytes.
Hash the address twice using sha256 function and take the first 4 bytes as verification code.
Add the verification code to the end of the initial address and get an address in base58check format through base58 encoding.
An encoded Mainnet address begins with T and is 34 bytes in length.
Please note: the sha3 protocol adopted is KECCAK-256.
I find mattvb91/tron-trx-php in GitHub and this repository there is a wallet generator method /src/Wallet.php but the generated key validation return an Error Exception and validation get failed.
I try to recode mattvb91/tron-trx-php Wallet generator method and create my wallet generator
class walletGenerator
{
private $addressPrefix = "41";
private $addressPrefixByte = 0x41;
private $addressSize = 34;
public function __construct()
{
}
public function generateWallet()
{
$key = new Key();
$odd = false;
while (!$odd)
{
$keyPair = $key->GenerateKeypair();
if(strlen($keyPair['public_key']) % 2 == 0) $odd = true;
}
$privateKeyHex = $keyPair['private_key_hex'];
$pubKeyHex = $keyPair['public_key'];
$pubKeyBin = hex2bin($pubKeyHex);
$addressHex = $this->getAddressHex($pubKeyBin);
$addressBin = hex2bin($addressHex);
$addressBase58 = $this->getBase58CheckAddress($addressBin);
$validAddress = $this->validateAddress($addressBase58);
}
private function getAddressHex($pubKeyBin)
{
if (strlen($pubKeyBin) == 65) {
$pubKeyBin = substr($pubKeyBin, 1);
}
$hash = Keccak::hash($pubKeyBin , 256);
$hash = substr($hash, 24);
return $this->addressPrefix . $hash;
}
private function getBase58CheckAddress($addressBin){
$hash0 = Hash::SHA256($addressBin);
$hash1 = Hash::SHA256($hash0);
$checksum = substr($hash1, 0, 4);
$checksum = $addressBin . $checksum;
$result = Base58::encode(Crypto::bin2bc($checksum));
return $result;
}
private function validateAddress($walletAddress){
if(strlen($walletAddress) !== $this->addressSize) return false;
$address = Base58Check::decode($walletAddress , false , 0 , false);
$utf8 = hex2bin($address);
if(strlen($utf8) !== 25) return false;
if(strpos($utf8 , $this->addressPrefixByte) !== 0) return false; // strpos(): Non-string needles will be interpreted as strings in the future. Use an explicit chr() call to preserve the current behavior
$checkSum = substr($utf8, 21);
$address = substr($utf8, 0, 21);
$hash0 = Hash::SHA256($address);
$hash1 = Hash::SHA256($hash0);
$checkSum1 = substr($hash1, 0, 4);
if ($checkSum === $checkSum1) {
return true;
}
}
Here is my problems :
validateAddress method has an Error Exception
when I add Private Key in Trx Wallets manually the detected wallet address is different from the generated Address.
if(strpos($utf8 , $this->addressPrefixByte) !== 0) return false; // strpos(): Non-string needles will be interpreted as strings in the future. Use an explicit chr() call to preserve the current behavior
additional information
I used ionux/phactor to generating keyPair and iexbase/tron-api Support classes for Hash and ...
I can debug and solved the problem and share the solutions with you
nowhere is the solutions
this line of code has an Error Exception
if(strpos($utf8 , $this->addressPrefixByte) !== 0) return false; // strpos(): Non-string needles will be interpreted as strings in the future. Use an explicit chr() call to preserve the current behavior
this problem because of PHP 7.3 bugs and fix with PHP 7.2 ( Exception: stripos(): Non-string needles will be interpreted as strings in the future. Use an explicit chr() call to preserve the current behavior #770 )
and addresses differences
in the Key Pair array, I remove 0x from first of Private Key Hex and I can access to the wallet in Tron Link Extention don't remember to have a transition for wallet activation
<a href="<?php echo base_url().'daily_report/index/'.$this->encrypt->encode($this->session->userdata('employee_id')) ?>">
i have encrypted the above url using the codeigniter encrypt
i set the encryption key in codeigniter config file
$config['encryption_key'] = 'gIoueTFDwGzbL2Bje9Bx5B0rlsD0gKDV';
and i called in the autoload
$autoload['libraries'] = array('session','form_validation','encrypt','encryption','database');
when the ulr(href) load into the url it look like this
http://localhost/hrms/daily_report/index/FVjGcz4qQztqAk0jaomJiAFBZ/vKVSBug1iGPQeKQCZ/K7+WUE4E/M9u1EjWh3uKTKeIhExjGKK1dJ2awL0+zQ==
but the url is not decoded, and i;m not getting the employee_id it shows empty.
public function index($employee_id) {
$save_employee_id = $employee_id;
// decoding the encrypted employee id
$get_employee_id = $this->encrypt->decode($save_employee_id);
echo $employee_id; // answer: FVjGcz4qQztqAk0jaomJiAFBZ
echo "<br>";
echo $get_employee_id; // is display the null
echo "<br>";
exit();
// get the employee daily report
$data['get_ind_report'] = $this->daily_report_model->get_ind_report($get_employee_id);
// daily report page
$data['header'] = "Daily Report";
$data['sub_header'] = "All";
$data['main_content'] = "daily_report/list";
$this->load->view('employeelayout/main',$data);
}
complete url(3) is
FVjGcz4qQztqAk0jaomJiAFBZ/vKVSBug1iGPQeKQCZ/K7+WUE4E/M9u1EjWh3uKTKeIhExjGKK1dJ2awL0+zQ==
it shows only
FVjGcz4qQztqAk0jaomJiAFBZ
i tried to change in the
$config['permitted_uri_chars'] = 'a-zA-Z 0-9~%.:_\-#=+';
by / in the permitted uri chars
but it throwing error
So, i need to encryption the $id in the url using the codeigniter encrypt class and decrypt in the server side to get the actual $id, So that i fetch data from the DB. any help would be appreciated
You have to extend encryption class and avoid the / to get it working. Place this class in your application/libraries folder. and name it as MY_Encrypt.php.
class MY_Encrypt extends CI_Encrypt
{
/**
* Encodes a string.
*
* #param string $string The string to encrypt.
* #param string $key[optional] The key to encrypt with.
* #param bool $url_safe[optional] Specifies whether or not the
* returned string should be url-safe.
* #return string
*/
function encode($string, $key="", $url_safe=TRUE)
{
$ret = parent::encode($string, $key);
if ($url_safe)
{
$ret = strtr(
$ret,
array(
'+' => '.',
'=' => '-',
'/' => '~'
)
);
}
return $ret;
}
/**
* Decodes the given string.
*
* #access public
* #param string $string The encrypted string to decrypt.
* #param string $key[optional] The key to use for decryption.
* #return string
*/
function decode($string, $key="")
{
$string = strtr(
$string,
array(
'.' => '+',
'-' => '=',
'~' => '/'
)
);
return parent::decode($string, $key);
}
}
FVjGcz4qQztqAk0jaomJiAFBZ/vKVSBug1iGPQeKQCZ/K7+WUE4E/M9u1EjWh3uKTKeIhExjGKK1dJ2awL0+zQ==
Shows
FVjGcz4qQztqAk0jaomJiAFBZ
If you look at your url closely, you could see that after the result which has been shown there is a '/' . Now any string after that will be treated as another segment. Hence it could not decode.
The encrypt library in this case would not work.
Either you stop passing that through the URL or use another different technique base_encode().
Hope that helps
This is happening as the character "/" is part of html uri delimiter. Instead you can work around it by avoiding that character in html url by rawurlencoding your encrytion output string before attaching it to url.
\edit:
I tried rawurlencode, but wasn't able to get the proper output.
Finally succeeded by using this code.
Define two functions:
function hex2str( $hex ) {
return pack('H*', $hex);
}
function str2hex( $str ) {
return array_shift( unpack('H*', $str) );
}
Then use call str2hex and pass it the encrypted user id to convert encrypted string into hexcode.
Reverse the process to get the correct string so that you can decrypt it.
I was able to properly encode and decode:
"FVjGcz4qQztqAk0jaomJiAFBZ/vKVSBug1iGPQeKQCZ/K7+WUE4E/M9u1EjWh3uKTKeIhExjGKK1dJ2awL0+zQ=="
to:
"46566a47637a3471517a7471416b306a616f6d4a694146425a2f764b56534275673169475051654b51435a2f4b372b57554534452f4d397531456a576833754b544b65496845786a474b4b31644a3261774c302b7a513d3d"
The url would become rather long though.
I'm writing an app that deals with sales data. The data is outputted to a private webpage, which my app can access and read. However, my app does not always receive all of the data that it's supposed to. (If I open the webpage in a desktop browser, the data is always complete) I suspect that this may be due to a race condition in which the app tries to read the data on the webpage before the webpage has loaded completely. Here is the code that reads the data from the webpage:
try {
URL url = new URL(myurl);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setReadTimeout(120000 /* milliseconds, or 2 minutes */);
conn.setConnectTimeout(120000 /* milliseconds, or 2 minutes */);
conn.setRequestMethod("GET");
conn.setDoInput(true);
// Starts the query
conn.connect();
int response = conn.getResponseCode();
Log.d(DEBUG_TAG, "The response is: " + response);
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
is = conn.getInputStream();
// Convert the InputStream into a string
String contentAsString = readIt(is, len);
return contentAsString;
// Makes sure that the InputStream is closed after the app is
// finished using it.
} finally {
if (is != null) {
is.close();
}
}
As you can see, I added a Thread.sleep() to give the page time to load, and this has helped, but has not solved the issue completely.
What can I do to make sure that the app waits until the webpage has loaded completely before trying to read data from it?
Edit: Here is my readit function:
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
Reader reader = null;
reader = new InputStreamReader(stream, "UTF-8");
char[] buffer = new char[len];
reader.read(buffer);
return new String(buffer);
}
Edit 2: I have edited readIt to loop until a certain character sequence is found in the buffer. This works, but if the data doesn't load quickly enough, then the app will crash because Android thinks that there's an infinite loop. Here is the edited code:
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
boolean xyz = false;
Reader reader = null;
char[] buffer = null;
while (xyz == false){
reader = new InputStreamReader(stream, "UTF-8");
buffer = new char[len];
reader.read(buffer);
String test = new String(buffer);
System.out.println(test);
if (test.contains("###")){
xyz = true;
}
}
return new String(buffer);
}
Well, here's what I found that worked. The issue was that the reader would read from the stream even if the stream wasn't fully loaded. It would also read 1000000 characters from the stream, even if there weren't that many characters to be read. This caused the reader to read whatever was available, and then fill in the rest of the 1000000 characters with UTF-8 unknown symbols (�). The code that you see here loops on the stream, each time reading whatever has loaded since the last loop, and discarding the � characters. It does require that whatever you are reading not contain any non-UTF-8 characters, because it splits the string on the first � that it finds, but if you are reading in UTF-8, you should make sure of that anyway. It also requires that you place an uncommon sequence of characters at the end of your data to mark the end.
// This converts the data stream into a String that can be manipulated.
public String readIt(InputStream stream, int len) throws IOException, UnsupportedEncodingException {
boolean isDone = false; // Boolean to keep track of whether the data is loaded or not.
Reader reader = new InputStreamReader(stream, "UTF-8"); // Input stream reader
char[] buffer = null; // Character array to hold the data from the stream
String readDump = null; // String to hold data converted from the character array, which will most likely contain junk characters.
String splitDump [] = null; // String array that holds the valid data and junk from readDump after they've been separated.
String dataHolder = ""; // Output string that holds the valid data
// While the final characters have not been read (###)
while (isDone == false){
buffer = new char[len];
reader.read(buffer); // Read data from the stream
readDump = new String(buffer); // Turn it into a string
splitDump = readDump.split("[\\x00\\x08\\x0B\\x0C\\x0E-\\x1F]", 2); //Split the string into valid data and junk.
dataHolder += splitDump[0]; // Add the valid data to the output string
System.out.println(dataHolder); //Debug
if (dataHolder.contains("###")){
// If the output string has the final characters in it, then we are done.
isDone = true;
}
}
return dataHolder;
}
I'm trying to pipe my incoming mails to a PHP script so I can store them in a database and other things. I'm using the class MIME E-mail message parser (registration required) although I don't think that's important.
I have a problem with email subjects. It works fine when the title is in English but if the subject uses non-latin Characters I get something like
=?UTF-8?B?2KLYstmF2KfbjNi0?=
for a title like
یک دو سه
I decode the subject like this:
$subject = str_replace('=?UTF-8?B?' , '' , $subject);
$subject = str_replace('?=' , '' , $subject);
$subject = base64_decode($subject);
It works fine with short subjects with like 10-15 characters but with a longer title I get half of the original title with something like ��� at the end.
If the title is even longer, like 30 characters, I get nothing. Am I doing this right?
You can use the mb_decode_mimeheader() function to decode your string.
Despite the fact that this is almost a year old - I found this and am facing a similar problem.
I'm unsure why you're getting odd characters, but perhaps you are trying to display them somewhere your charset is unsupported.
Here's some code I wrote which should handle everything except the charset conversion, which is a large problem that many libraries handle much better. (PHP's MB library, for instance)
class mail {
/**
* If you change one of these, please check the other for fixes as well
*
* #const Pattern to match RFC 2047 charset encodings in mail headers
*/
const rfc2047header = '/=\?([^ ?]+)\?([BQbq])\?([^ ?]+)\?=/';
const rfc2047header_spaces = '/(=\?[^ ?]+\?[BQbq]\?[^ ?]+\?=)\s+(=\?[^ ?]+\?[BQbq]\?[^ ?]+\?=)/';
/**
* http://www.rfc-archive.org/getrfc.php?rfc=2047
*
* =?<charset>?<encoding>?<data>?=
*
* #param string $header
*/
public static function is_encoded_header($header) {
// e.g. =?utf-8?q?Re=3a=20Support=3a=204D09EE9A=20=2d=20Re=3a=20Support=3a=204D078032=20=2d=20Wordpress=20Plugin?=
// e.g. =?utf-8?q?Wordpress=20Plugin?=
return preg_match(self::rfc2047header, $header) !== 0;
}
public static function header_charsets($header) {
$matches = null;
if (!preg_match_all(self::rfc2047header, $header, $matches, PREG_PATTERN_ORDER)) {
return array();
}
return array_map('strtoupper', $matches[1]);
}
public static function decode_header($header) {
$matches = null;
/* Repair instances where two encodings are together and separated by a space (strip the spaces) */
$header = preg_replace(self::rfc2047header_spaces, "$1$2", $header);
/* Now see if any encodings exist and match them */
if (!preg_match_all(self::rfc2047header, $header, $matches, PREG_SET_ORDER)) {
return $header;
}
foreach ($matches as $header_match) {
list($match, $charset, $encoding, $data) = $header_match;
$encoding = strtoupper($encoding);
switch ($encoding) {
case 'B':
$data = base64_decode($data);
break;
case 'Q':
$data = quoted_printable_decode(str_replace("_", " ", $data));
break;
default:
throw new Exception("preg_match_all is busted: didn't find B or Q in encoding $header");
}
// This part needs to handle every charset
switch (strtoupper($charset)) {
case "UTF-8":
break;
default:
/* Here's where you should handle other character sets! */
throw new Exception("Unknown charset in header - time to write some code.");
}
$header = str_replace($match, $data, $header);
}
return $header;
}
}
When run through a script and displayed in a browser using UTF-8, the result is:
آزمایش
You would run it like so:
$decoded = mail::decode_header("=?UTF-8?B?2KLYstmF2KfbjNi0?=");
Use php native function
<?php
mb_decode_mimeheader($text);
?>
This function can handle utf8 as well as iso-8859-1 string.
I have tested it.
Use php function:
<?php
imap_utf8($text);
?>
Just to add yet one more way to do this (or if you don't have the mbstring extension installed but do have iconv):
iconv_mime_decode($str, ICONV_MIME_DECODE_CONTINUE_ON_ERROR, 'UTF-8')
Would the imap-mime-header-decode function help here?
Found myself in a similar situation today.
http://www.php.net/manual/en/function.imap-mime-header-decode.php