I'm having a problem to insert a oriental character with bind variables in SQL Server.
i'm using MSSQL commands and PHP.
My PHP code is like this:
$sql = "
CREATE TABLE table_test
( id int
,nvarchar_latin nvarchar(255) collate sql_latin1_general_cp1_ci_as
);";
$stmt = mssql_query($sql);
$conn = mssql_connect("server","user","pass");
mssql_select_db('test')
$stmt = mssql_init('test..sp_chinese', $conn);
$id = 1;
$nvarchar_latin = '重建議';
mssql_bind($stmt, '#id' , $id , SQLINT1);
mssql_bind($stmt, #nvarchar_latin, $nvarchar_latin, SQLVARCHAR);
mssql_execute($stmt);
My procedure is like this:
ALTER PROCEDURE sp_chinese
#id int
,#nvarchar_latin nvarchar (255)
AS
BEGIN
INSERT INTO char_chines (id, nvarchar_latin)
VALUES (#id, #nvarchar_latin);
END
this work if I change the oriental characters for normal one.
if I run directly this insert, it work's fine:
INSERT INTO table_test (id, nvarchar_latin)
VALUES (1, '重建議');
So, cleary the problem is when I send the variable from PHP to SQL Server.
Anyone have a clue how to make this works? some casting or something?
Thanks!
A solution that uses just the PHP (or even JavaScript) is to convert the character to its HEX value and store that. I don't know if you want to go this route but and I don't have time to show you the code but here is the full theory:
A non-English character is detected, like so: 重
Convert to HEX value (Look here for starters. But a search for Javascript will help you find better ways to do this even in PHP): 14af
NOTE: That is not what 重 really is in HEX
Store in a way that you can convert back to its original value. For example how can you tell what this is: 0d3114af is it 0d - 31 - 14 - af OR is it 0d31 - 14af. You can use deliminators like | or a . but one way is to provide padding of 00 in front. An English character would be only 2 characters long like 31 or af non-English will be 4 like 14af. Knowing this you can just split every 4 characters and convert to their values.
Downside is you will need to change your Database to accommodate these changes.
[ UPDATE ] -----
Here is some JavaScript code to send you off in the right direction. This is completely possible to replicate in PHP. This does not search for characters though, its part of an encryption program so all it cares about is turning everything into HEX. English characters will be padded with 00 (This is my own code hence no link to source):
function toHex(data) {
var result = '';
// Loop through entire string of data character by character
for(var i=0;i<data.length;i++) {
// Convert UTF-16 Character to HEX, if it is a 2 chracter HEX add 00 padding in front
result += (data.charCodeAt(i) + 0x10000).toString(16).slice(1);
}
// Display the result for testing purposes
document.getElementById('two').value = result;
}
function fromHex(data) {
var result = '', block = '', pattern = /(00)/; // Pattern is the padding
for(var i=0;i<data.length;i = i+4) {
// Split into separate HEX blocks
block = data.substring(i,i+4);
// Remove 00 from a HEX block that was only 2 characters long
if(pattern.test(block)){
block = block.substring(2,4);
}
// HEX to UTF-16 Character
result += String.fromCharCode(parseInt(block,16));
}
// Display the result for testing purposes
document.getElementById('two').value = result;
}
Related
I'm working on Laravel (v5.7) app that converts uploaded CSV (with contacts) into array that is then passed as argument when job class is being dispatched.
Here is the example of CSV file (format that is supported):
123456,Richard,Smith
654321,John,Doe
Uploaded (CSV) file is handled like this:
$file_path = $request->file_name->store('contacts');
$file = storage_path('app/' . $file_path);
$contactsIterator = $this->getContacts($file);
$contacts = iterator_to_array($contactsIterator); // Array of contacts from uploaded CSV file
protected function getContacts($file)
{
$f = fopen($file, 'r');
while ($line = fgets($f))
{
$row = explode(",", $line);
yield [
'phone' => !empty($row[0]) ? trim($row[0]) : '',
'firstname' => !empty($row[1]) ? trim($row[1]) : '',
'lastname' => !empty($row[2]) ? trim($row[2]) : '',
];
}
}
Finally, $contacts array is passed to a job that is dispatched:
ImportContacts::dispatch($contacts);
This job class looks like this:
public function __construct($contacts)
{
Log::info('ImportContacts#__construct START');
$this->contacts = $contacts;
Log::info('ImportContacts#__construct END');
}
public function handle()
{
Log::info('ImportContacts#handle');
}
... and everything worked fine (no errors) until I've tried with this CSV:
123456,Richardÿ,Smith
654321,John,Doe
Please notice ÿ. So, when I try with this CSV - I get this error exception:
/code_smsto/vendor/laravel/framework/src/Illuminate/Queue/Queue.php | 91 | Unable to JSON encode payload. Error code: 5
... and my log file looks like this:
error local 2019-11-11 17:17:18 /code_smsto/vendor/laravel/framework/src/Illuminate/Queue/Queue.php | 91 | Unable to JSON encode payload. Error code: 5
info local 2019-11-11 17:17:18 ImportContacts#__construct END
info local 2019-11-11 17:17:18 ImportContacts#__construct START
As you can see - handle method was never executed. If I remove ÿ - no errors and handle is executed.
I've tried to solve this, but without success:
Apply utf8_encode:
protected function getContacts($file, $listId)
{
$f = fopen($file, 'r');
while ($line = fgets($f))
{
$row = explode(",", $line);
yield [
'phone' => !empty($row[0]) ? utf8_encode($row[0]) : '',
'firstname' => !empty($row[1]) ? utf8_encode($row[1]) : '',
'lastname' => !empty($row[2]) ? utf8_encode($row[2]) : '',
];
}
}
... and it works (no errors, no matter if there's that ÿ), but then Greek and Cyrillic letters are turned into question marks. For example, this: Εθνικής will become ???????.
I also tried with mb_convert_encoding($row[1], 'utf-8') - and it doesn't turn Greek or Cyrillic letter into question marks, but this ÿ character will become ?.
Move "handling" (converting to array) of uploaded CSV file into #handle method of a Job class worked, but then I was not able to store the data from that array into DB (MongoDB). Please see the update below.
DEBUGGING:
This is what I get from dd($contacts);:
So, it has that "b" where ÿ is. And, after some "googling" I found that this "b" means "binary string", that is, a non unicode string, on which functions operate at the byte level (What does the b in front of string literals do?).
What I understand is this: When dispatching Job class, Laravel tries to "JSON encode" it (passed arguments/data) but it fails because there are binary data (non-unicode strings).
Anyway, I was not able to find a solution (to be able to handle such CSV file with ÿ).
I am using:
Laravel 5.7
PHP 7.1.31-1+ubuntu16.04.1+deb.sury.org+1 (cli) (built: Aug 7 2019 10:22:48) ( NTS )
Redis powered queues
UPDATE
When I move "handling" (converting to array) of uploaded CSV file into #handle method of a Job class - I don't get this error (Unable to JSON encode payload. Error code: 5), but when I try to store that problematic binary data with ÿ (b"Richardÿ") into MongoDB - it fails. The weird thing is that I don't get any error-exception message in log file, so I put all in try-catch like this:
try {
// Insert data into MongoDB
} catch (Exception $e) {
Log::info($e->getFile());
Log::info($e->getLine());
Log::info($e->getMessage());
}
... and this is the result:
Anyway, I believe that it failed because of b"Richardÿ", and I guess that the solution is in encoding string, but as I've mentioned - I was not able to find a solution that works:
utf8_encode works (no errors, no matter if there's that ÿ), but then Greek and Cyrillic letters are turned into question marks. For example, this: Εθνικής will become ???????
mb_convert_encoding($row[1], 'utf-8') - it doesn't turn Greek or Cyrillic letter into question marks, but this ÿ character will become ?.
iconv('windows-1252', 'UTF-8', $row[1]) - works (no errors, no matter if there's that ÿ), but when there are Greek or Cyrillic letters - it fails (I get this error exception: iconv(): Detected an illegal character in input string)
You have several ways to deal with it but I'd recommend the following two. In both cases, the idea is that you store a UTF-8 string.
A simpler approach, figure out what encoding it is out of the (your) predefined list and convert it to UTF8.
$encoding = mb_detect_encoding($content, 'UTF-8, ISO-8859-1, WINDOWS-1252, WINDOWS-1251', true);
if ($encoding != 'UTF-8') {
$string = iconv($encoding, 'UTF-8//IGNORE', $row[1]);
}
The second approach is to use a third party library outlined in this answer
I have the following SQLite table
CREATE TABLE keywords
(
id INTEGER PRIMARY KEY,
lang INTEGER NOT NULL,
kwd TEXT NOT NULL,
count INTEGER NOT NULL DEFAULT 0,
locs TEXT NOT NULL DEFAULT '{}'
);
CREATE UNIQUE INDEX kwd ON keywords(lang,kwd);
Working in PHP I typically need to insert keywords in this table, or update the row count if the keyword already exists. Take an example
$langs = array(0,1,2,3,4,5);
$kwds = array('noel,canard,foie gras','','','','','');
I now these data run through the following code
$len = count($langs);
$klen = count($kwds);
$klen = ($klen < $len)?$klen:$len;
$sqlite = new SQLite3('/path/to/keywords.sqlite');
$iStmt = $sqlite->prepare("INSERT OR IGNORE INTO keywords (lang,kwd)
VALUES(:lang,:kwd)");
$sStmt = $sqlite->prepare("SELECT rowid FROM keywords WHERE lang = :lang
AND kwd = :kwd");
if (!$iStmt || !$sStmt) return;
for($i=0;$i < $klen;$i++)
{
$keywords = $kwds[$i];
if (0 === strlen($keywords)) continue;
$lang = intval($langs[$i]);
$keywords = explode(',',$keywords);
for($j=0;$j < count($keywords);$j++)
{
$keyword = $keywords[$j];
if (0 === strlen($keyword)) continue;
$iStmt->bindValue(':kwd',$keyword,SQLITE3_TEXT);
$iStmt->bindValue(':lang',$lang,SQLITE3_INTEGER);
$sStmt->bindValue(':lang',$lang,SQLITE3_INTEGER);
$sStmt->bindValue(':kwd',$keyword,SQLITE3_TEXT);
trigger_error($keyword);
$iStmt->execute();
$sqlite->exec("UPDATE keywords SET count = count + 1 WHERE lang =
'{$lang}' AND kwd = '{$keyword}';");
$rslt = $sStmt->execute();
trigger_error($sqlite->lastErrorMsg());
trigger_error(json_encode($rslt->fetchArray()));
}
}
which generates the following trigger_error output
Keyword: noel
Last error: not an error
SELECT Result: {"0":1,"id":1}
Keyword: canard
Last Error: not an error
SELECT Reult:false
Keyword:foiegras
Last Error: not an error
SELECT Result: false
From the SQLite command line I see that the three row entries are present and correct in the table with the id/rowid columns set to 1, 2 and 3 respectively. lastErrorMsg does not report an error and yet two of the three $rslt->fetchArray() statements are returning false as opposed to an array with rowid/id attributes. So what am I doing wrong here?
I investigated this a bit more and found the underlying case. In my original code the result from the first SQLite3::execute - $iStmt-execute() - was not being assigned to anything. I did not see any particular reason for fetching and interpreting that result. When I changed that line of code to read $rslt = $iStmt->execute() I got the expected result - the rowid/id of the three rows that get inserted was correctly reported.
It is as though internally the PHP SQLite3 extension buffers the result from SQLiteStatement::execute function calls. When I was skipping the assignment my next effort at running such a statement, $sStmt->execute() was in effect fetching the previous result. This is my interpretation without knowing the inner workings of the PHP SQLite3 extension. Perhaps someone who understands the extension better would like to comment.
Add $rslt = NONE; right after trigger_error(json_encode($rslt->fetchArray())); and the correct results appear.
FetchArray can only be called once and somehow php is not detecting that the variable has changed. I also played with changing bindValue to bindParam and moving that before the loop but that is unrelated to the main issue.
It is my opinion that my solution should not work unless there is a bug in php. I am too new at the language to feel confident in that opinion and would like help verifying it. Okay, not a bug, but a violation of the least surprise principle. The object still exists in memory so without finalizing it or resetting the variable, fetch array isn't triggering.
Using a mysqli prepared statement I would like to insert an array into a mysql database table.
Being aware that bind-param and arrays do not go together, we would like to write the query in php first, then process this as a prepared statement:
$tagQuery = "INSERT INTO word_tags(speaks) VALUES ";
// Count total array values
$icoderayInsideCount = count($icoderayInside);
foreach ($icoderayInside as $icoderayInsideKey=>$icoderayInsideValue)
{
// Last value
// Currrent Array Key Total / Last Array Value
if ($icoderayInsideKey == $icoderayInsideCount)
{
$tagQuery .= "('$icoderayInsideValue')";
}
// All other values
else
{
$tagQuery .= "('$icoderayInsideValue'), ";
}
}
// Send array (keywords) to database
if ($stmt2 = $link2->prepare($tagQuery))
{
if (!$stmt2->execute())
{
// #2 If it can prepare but can't execute, why?
echo "Error {$link2->errno} : {$link2->error} (Cant execute?) <br/>";
// Dump query to check the end result
var_dump($tagQuery);
exit();
}
$stmt2->close();
}
else
{
// #1 If it cant prepare, why?
echo "Error {$link2->errno} : {$link2->error} (Cant prepare?)";
exit();
}
When i run this via PHP / Server i get:
Error 1064 : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('DONGLES'),' at line 1 (Cant prepare?)
So, the statement prepares but is not being executed.
The value of the generated query, $tagQuery is:
INSERT INTO word_tags(speaks) VALUES ('PMP3670B_BK'), ('PRESTIGIO'), ('MULTIPAD'), ('7"'), ('800X480'), ('1GHZ'), ('ARM'), ('CORTEX'), ('A8'), ('CPU'), ('512MB'), ('DDR3'), ('DRAM'), ('ANDROID'), ('4.1'), ('JELLY'), ('BEAN'), ('MALI'), ('400'), ('MP'), ('GPU'), ('VIDEO'), ('4GB'), ('INTERNAL'), ('FLASH'), ('MEMORY'), ('SUPPORT'), ('32GB'), ('SDHC/SD'), ('USB/WI-FI/HEADSET'), ('PORT'), ('LITHIUM'), ('POLYMER'), ('BLACK'), ('HDMI'), ('OUTPUT'), ('UPTO'), ('1080'), ('HD'), ('USB2.0'), ('MINI'), ('HIGH'), ('SPEED'), ('FOR'), ('3G'), ('DONGLE'), ('OTG'), ('CABLE'), ('INCLUDED')
The fourth value from the end is('DONGLE') which is what the error message is complaining about.
When i run this exact same query through phpmyadmin there is no error involved.
What i assume is happening, is that there is some kind of length limit involved within creating a prepared statement... Or something to this effect.
Have scratched my brains for hours now to try to solve this and have not found any relating information.
If anyone could offer some assistance / advice / indication / input or otherwise as to what the conflict of problem may be within this, it would be GREATLY appreciated.
Thanks so much for the time and effort in readying through this!
EDIT:
#Mihai - Thanks for the thought.
It seems that the word dongle does have something string to it. In the original string, before being parsed to an array, it looks like this: DONGLE,
I run preg_replace to remove this comma from the string before parsing it to an array:
$icode = preg_replace('#,#', '', $icode);
Then into an array:
$icoderayInside = explode(" ", $icode);
Still cannot think of any reason this would cause a conflict as the output string, the query is as i have previously stated and includes no comma, or anything... Any would be greatly appreciated!
EDIT 2:
#ShadyAtef
Original input is stored in mysql as varchar, latin_general_ci:
PRESTIGIO MULTIPAD, 7" 800X480, 1GHZ ARM CORTEX A8 CPU, 512MB DDR3 DRAM, ANDROID 4.1 JELLY BEAN, MALI 400 MP GPU, VIDEO, 4GB INTERNAL FLASH MEMORY, SUPPORT 32GB SDHC/SD, USB/WI-FI/HEADSET PORT, LITHIUM POLYMER, BLACK, HDMI OUTPUT UPTO 1080 HD, USB2.0 MINI HIGH SPEED PORT FOR 3G DONGLE, OTG CABLE INCLUDED
Brought into php then processed to an array with additional requirements:
// Convert Var a + b to String
$icode = $itemCode . ' ' . $description;
// Clean of Unwanteds
$icode = preg_replace('#,#', '', $icode);
$icode = preg_replace('#\(#', '', $icode);
$icode = preg_replace('#\)#', '', $icode);
// Creates array from sting
$icoderayInside = explode(" ", $icode);
// Remove array duplicates
$icoderayInside = array_unique($icoderayInside);
Before being built into the query. Any assistance would be GREATLY appreciated!
EDIT 3:
#ShadyAtef
// Currrent Array Key Total / Last Array Value
if ($icoderayInsideKey == $icoderayInsideCount)
{
// dump here shows:
$icoderayInsideKey == 49
$icoderayInsideCount == 49
}
This was really tricky,but I got it : The tricky wrong part is that
if ($icoderayInsideKey == $icoderayInsideCount)
It should be
if ($icoderayInsideKey == ($icoderayInsideCount-1))
Because the last key in the array equals to the array (length -1) So you should change your if condition
I want to store the query string of the current URL as a shorter alphanumeric string with the ability to convert it back to the original query string.
For example:
inputType=timeline&count=50&hashtag=%23li&filterspecifiedhashtag=1&filterhashtagsend=1&filterscreennames=1&extracturl=1&deshortifyurl=1&filterurls=1
I want to be able to use the resultant alphanumeric string as a filename.
I want to avoid using MYSQL or storing values in a text file.
Is there a way to convert to an alphanumeric string as well as being able to convert it back to the original query string? I am not very knowledgeable about hashing but wondering whether some kind of "two way hashing" technique could work?
What you are seeking is not a hash - since hash is a one-way function in common case.
Here is a possible solution - use both base64 encryption and something like parameters map, so you will able to get shorter file name because you'll not store parameters names, only values:
class Holder
{
const NAME_PARAM_DELIMIER = '|';
public static function getParametersMap()
{
return [
0 => 'count',
1 => 'deshortifyurl',
2 => 'extracturl',
3 => 'filterhashtagsend',
4 => 'filterscreennames',
5 => 'filterspecifiedhashtag',
6 => 'filterurls',
7 => 'hashtag',
8 => 'inputType',
];
}
public static function getParamsByName($sName, $bReturnAsArray=true)
{
$rgParams = #array_combine(self::getParametersMap(), explode(self::NAME_PARAM_DELIMIER, base64_decode($sName)));
if(!is_array($rgParams))
{
return null;
}
return $bReturnAsArray?$rgParams:http_build_query($rgParams);
}
public static function getNameByParams($sQuery)
{
parse_str($sQuery, $rgParams);
ksort($rgParams);
return base64_encode(join(self::NAME_PARAM_DELIMIER, array_values($rgParams)));
}
}
$sQuery = 'inputType=timeline&count=50&hashtag=%23li&filterspecifiedhashtag=1&filterhashtagsend=1&filterscreennames=1&extracturl=1&deshortifyurl=1&filterurls=1';
$sName = Holder::getNameByParams($sQuery);
$rgData = Holder::getParamsByName($sName);
var_dump($sName); //NTB8MXwxfDF8MXwxfDF8I2xpfHRpbWVsaW5l
var_dump($rgData);
Also note that base64 will produce "=" symbols - I am not sure that it is allowed on all file systems (I'm using Reiser, so it's ok in my case)
You can use base64_encode() and base64_decode(), but if you want it as a filename, you'll likely hit filename length limit of filesystem (255 chars for ext3). If it's possible you hit this limit, you can use each X chars as directory name, and create full path.
I would like to generate the mysql-password hash from js.
I know the method with php functions,
$p = "example";
echo("$p<br>");
$p2= sha1($p,true);
echo("$p2<br>"); //ĂI')s~űvŠ-Ëo?
$result = sha1($p2);
echo("$result<br>"); //*57237bb49761f29ab9724ba084e811d70c12393d - this is the same as password("example") in mysql
and I'm trying to do this in javascript.
here is located the sha1 function:
http://pajhome.org.uk/crypt/md5/sha1.html
this is the hex2bin function I use to give the same result as sha1("",true);
function hex2bin(hex)
{
var bytes = [], str;
for(var i=0; i< hex.length-1; i+=2)
bytes.push(parseInt(hex.substr(i, 2), 16));
return String.fromCharCode.apply(String, bytes);
}
but at the last step it does not work. What can be the problem?
var p = "example";
console.log(p);
var p2 = hex2bin(hex_sha1(p));
console.log(p); //ÃI')s~ûv©-Ëo? - SEEMS OK
var result = hex_sha1(p2);
console.log(result); //9a5355dce26b1adfa0bdbe9f2b2a6e5ae58e5c9d WRONG
Use http://code.google.com/p/crypto-js/#SHA-1 with 2 encodings.
Here is an example:
var password = "TestPassword";
var result = ("*"+CryptoJS.SHA1(CryptoJS.SHA1(password))).toUpperCase();
console.log("result : " + result);
Result will be *0F437A73F4E4014091B7360F60CF81271FB73180. If you check it with mysql password() it will be the same:
mysql> select password("TestPassword") as result;
+-------------------------------------------+
| result |
+-------------------------------------------+
| *0F437A73F4E4014091B7360F60CF81271FB73180 |
+-------------------------------------------+
SOLUTION 2017
Download file 2.5.3-crypto-sha1.js
https://code.google.com/archive/p/crypto-js/downloads
var key = '*' + Crypto.SHA1(Crypto.util.hexToBytes(Crypto.SHA1('test'))).toUpperCase();
key = *94BDCEBE19083CE2A1F959FD02F964C7AF4CFC29
Test in MySQL
SELECT PASSWORD('test') /* function */
UNION
SELECT '*94BDCEBE19083CE2A1F959FD02F964C7AF4CFC29' /* Key generated by javascript */
UNION
select CONCAT('*', UPPER(SHA1(UNHEX(SHA1('test'))))); /* Logic for Password() function */
So, in the example above, you are invoking: hex_sha1(hex2bin(hex_sha1(string))). Are you sure that is correct? Why would you need hex_sha1 twice?
Few month ago, I needed SHA1 in my JS and found this: http://phpjs.org/functions/sha1:512
... and it worked pretty ok ;)
The solution (partly):
Well, as you suggested, the main problem was charset encoding. It seems that MySQL does not encode any data prior to hashing it. This does not represent problem when dealing it with hex data, however, MySQL's UNHEX(hex2bin in JS) returns some unreadable chars and if we encode the result using UTF8 that's where all breaks apart.
When I edited JS's SHA1 not to encode data to UTF8 at line (I only commented the line):
str = this.utf8_encode(str);
everything seemed ok. However, as soon as I supplied some UTF8 chars to input (such as Central European chars čćžšđ) it started failing again.
So, bottom line, if you disable UTF8 in JSs SHA1 function and do not supply UTF8 chars on input it works fine.
I'm sure there is a solution to this such as JS function that will decode UTF8 input so, JS could hash it properly.