I am building a web application which will run in electron with angular as a frontend framework and laravel as a backend framework. In the application it's possible to login with a smartcard (thanks to node-pcsclite), it reads the bytes on the smartcard and then I convert them.
The smartcard contains a code which is linked to the staff table in my MSSQL database. I can retrieve the code from the smartcard and I can log into the application when it uses mysql as database server.
Now when I'm trying to do the same but with mssql, I get an error which should be viewed in html mode instead of the error page itself.
(The code can be alphanumeric)
So it adds all these strange characters (probably non-existing characters), not that much of a problem right? At least, that's what I thought. So I tried to fix it by using this code inside my laravel controller:
preg_replace('/[^A-Za-z0-9\-]/', '', $string);
This didn't solve anything. Then I thought I might have a problem with the query, so I ran SQL Profiler, the problem is that (probably because of the special characters) the query is broken.
select top 1 * from [Staff] where [CodeInit] = '
go
So does anyone know how to really remove the strange characters?
If you need more information feel free to ask.
I had this problem and landed to this question when searching for a solution. I was unable to find any fix.
The string with non-printable characters retrieved from mdecrypt_generic() so I wanted a way to remove those characters. When I copy and paste the retrieved value from browser to Brackets text editor, it show these red dots.
I just pasted it to google and then it was encoded to %10. Nothing helped till now, so as a temporary solution I just used rtrim() to remove those dots.
Copy the dot in brackets and replace with "DOT_HERE".
rtrim(rtrim($pvp, "DOT_HERE"), "\0\4");
"\0\4" will remove only nulls and EOT but not that dot character(%10).
Further here is a screenshot with that red dot. You can use Brackets text editor to see this.
Note that $pvp is the decrypted text.
Related
I give up trying to solve this by myself, I need help! I have been working on a WordPress project that has a few features working with AJAX. After updating PHP to 5.6 (as the latest WordPress requested), many of my AJAX functions are broken because of a mysterious no-break line character %a0 appearing in its response and breaking the JSON structure.
The response is from json_encode().
JSON response I am getting:
{
"term_id":75,
"name":"iPhone
3G",
"slug":"iphone-3g"
},
Investigation:
After many hours of reading about this, I tried several solutions that worked for others, but they didn't work for me.
Turning off magic_quotes_gpc in php.ini
Escape the string with preg_replace on the server side for all different no-break/new-line special characters
Escape the string on client side with str.replace
I checked the database, and there is no %a0 for that entry, there is a space %20, which is correct. I also noticed that if I remove that space, this happens to the next item that has a space %20.
I should also mention that this example above with the iPhone 3G is not unique. After this item, a few items are cleared (even those with space %20) but then it happens again later on down the latter with other items, same situation.
So it appears that PHP is replacing %20 with %a0 every so often.
What should I do?
We're running into a weird edge case where we are trying to store a json blob in a table in our database, and that blob needs to be able to contain the \ character. So a user were to enter in \test it needs to come back as exactly that, but instead its coming back as a tab followed by "est"
As far as I can tell, whats happening is that when a user enters and submits "\test" it gets evaluated into "\ \test" (remove the space, cant put two backslashes in here and have it display right?) by the client and then entered into the table. I can verify that in the SQL that gets called against the table there are two backslashes. When I look at it in the table after this step its back to "\test". When the client loads it up again it gets evaluated into a tab followed by an "est".
We are under the impression that the second backslash is necessary so that the first backslash will get escaped and not evaluated but maybe that is what is causing issues? I sort of assume when the query runs one of the backslashes gets escaped anyway but I'm not really sure what to do about that. Is there something with out our database is handling backslashes that we need to be looking out for? Is there a way to handle this that we haven't considered?
It's a Postgres database if that's helpful. I'd say I'm beginner to intermediate on this sort of thing, I'm looking through documentation but if anyone can even point me in the right direction that would be very helpful.
Postgres version as far as i can tell through Amazon Aws is 9.3
EDIT
I think ive tracked this issue down to a line in our php backend that I don't really understand. I'm looking at the documentation for that now and will mark this as answered since I've verified that its not an issue with SQL.
Blockquote A backslash as - by default - no special meaning in SQL. This might be caused by whatever code is processing those values (and sending them to the database). See here for an online example: rextester.com/QLLYG57275 – a_horse_with_no_name
I'm accepting this as the answer as I've verified that the issue is with out backend code constructing the SQL, and not how the SQL is being handled on the database end.
I got a strange issue.
We're using transaction-IDs to our customers bank bookings in the following format: U123C123P123B123.
After a test booking, we copied the transaction-ID (correctly displayed) text from our online-banking output with "ctrl+c". This text, entered into our system again (by pasting strg+v) could not be processed, because:
When pasting, "<wbr>" turns into a minus sign. The minus sign is not displayed in the form field. But it will pass. MySQL can not process the request with that strange minus sign.
The source code of the bank output is as follows: "U123C123P123<wbr>B123".
Most Software (firefox, thunderbird, ...) shows the correct T-ID after pasting. Some Editors, like notepad++, Dreamweaver are showing the pasted T-ID including the minus sign. Like: "U123C123P123-B123"
How can I remove this dimwitted minus sign? I have already tried with str_replace and strip_tags.
Sorry about my english.
My best guess is that somehow the particular character ends up in your output before the B123.
Check the online banking system. Go to the page that shows the transaction id and save it on your hard drive. Open it with an editor that can show non printable characters (vi for instance in Linux) and check the area where the particular string appears. If the string appears with the <wbr> character then you can change the online banking system software so as it doesn't output that character there.
However (as I suspect) if you don't have access to that system in terms of modifying it and you are in need of fixing things in your end, you can attach a filter in the form field or filter the data yourself prior to the database call.
$field = strtoupper($_POST['transaction_id']);
$sanitized = str_replace('<WBR>', '', $field);
$sanitized = str_replace(chr(173), '', $sanitized);
173 is the soft hyphen ­ so that is most likely what is coming in. You can always do a var_dump on the posted data and check the actual ASCII code of what is coming in.
HTH
I'd really appreciate some help with this. I've wasted days on this problem and none of the suggestions I have found online seem to give me a fix.
I have a CSV file from a supplier. It appears to have been exported from an Microsoft system.
I'm using PHP to import the data into MySQL (both latest versions).
I have one particular record which contains a strange character that I can't get rid of. Manual editing to remove the character is possible, but I would prefer an automated solution as this will happen multiple times a day.
The character appears to be an interpretation of a “smart quote”. A hex editor tells me that the character codes are C2 and 92. In the hex editor it looks like a weird A followed by a smart quote. In other editors and Calc, Writer etc it just appears as a box. ﾒ
I'm using mb_detect_encoding to determine the encoding. All records in the CSV file are returned as ASCII, except the one with the strange character, which is returned as UTF-8.
I can insert the offending record into MySQL and it just appears in Workbench as a square.
MySQL tables are configured to utf-8 – utf8_unicode_ci and other unusual UTF characters (eg fractions) are ok.
I've tried lots of solutions to this...
How to detect malformed utf-8 string in PHP?
Remove non-utf8 characters from string
Removing invalid/incomplete multibyte characters
How to detect malformed utf-8 string in PHP?
How to replace Microsoft-encoded quotes in PHP
etc etc but none of them have worked for me.
All I really want to do is remove or replace the offending character, ideally with a search and replace for the hex values but none of the examples I have tried have worked.
Can anyone help me move forward with this one please?
EDIT:
Can't post answer as not enough reputation:
Thanks for your input. Much appreciated.
I'm just going to go with the hex search and replace:
$DodgyText = preg_replace("/\xEF\xBE\x92/", "" ,$DodgyText);
I know it's not the elegant solution, but I need a quick fix and this works for me.
Another solution is:
$contents = iconv('UTF-8', 'Windows-1251//IGNORE',$contents);
$contents = iconv('Windows-1251', 'UTF-8//IGNORE',$contents);
Where you can replace Windows-1251 to your local encoding.
At a quick glance, this looks like a UTF-8 file. (UTF-8 is identical with the first 128 characters in the ASCII table, hence everything is detected as ASCII except for the special character.)
It should work if your database connection is also UTF-8 encoded (which it may not be by default).
How to do that depends on your database library, let us know which one you're using if you need help setting the connection encoding.
updated code based on established findings
You can do search & replace on strings using hexadecimal notation:
str_replace("\xEF\xBE\x92", '', $value);
This would return the value with the special code removed
That said, if your database table is UTF-8, you shouldn't need that conversion; instead you could look at the connection (or session) character set (i.e. SET NAMES utf8;). Configuring this depends on what library you use to connect to your database.
To debug the value you could use bin2hex(); this usually helps in doing searches online.
if i trying to access this url http://localhost/common/news/33/+%E0%B0%95%E0%B1%87%E0%B0%B8.html , it shows an An Error Was Encountered, The URI you submitted has disallowed characters. I set $config['permitted_uri_chars'] = 'a-z 0-9~%.:??_=+-?' ; ..// WHat i do ?
Yeah, if you want to allow non-ASCII bytes you would have to add them to permitted_uri_chars. This feature operates on URL-decoded strings (normally, unless there is something unusual about the environment), so you have to put the verbatim bytes you want in the string and not merely % and the hex digits. (Yes, I said bytes: _filter_uri doesn't use Unicode regex, so you can't use a Unicode range.)
Trying to filter incoming values (instead of encoding outgoing ones) is a ludicrously basic error that it is depressing to find in a popular framework. You can turn this misguided feature off by setting permitted_uri_chars to an empty string, or maybe you would like a range of all bytes except for control codes ("\x20-\xFF"). Unfortunately the _filter_uri function still does crazy, crazy, broken things with some input, HTML-encoding some punctuation on the way in for some unknown bizarre reason. And you don't get to turn this off.
This, along with the broken “anti-XSS” mangler, makes me believe the CodeIgniter team have quite a poor understanding of how string escaping and security issues actually work. I would not trust anything they say on security ever.
What to do?
Stop using unicode characters in an URL - for the same reasons as you shouldn't name files on a filesystem with unicode characters.
But, if you really need it, I'll copy/paste some lines from the config:
Leave blank to allow all characters -- but only if you are insane.
I would NOT suggest trying to decode them or use any other tricks, instead I would suggest using urlencode() and urldecode() functions.
Since I don't have a copy of your code, I can't add examples, if you could provide me some, I can show you an example how to do it.
However, it's pretty straightforward to use, and it's built in PHP4 and PHP5.
I had a similar problem and wanted to share the solution. It was reset password, and I had to send the username and time, as the url will be active for an hour only. Codeigniter will not accept certain characters in url for security reasons and I did not want to change that. So here is what I did:
concat user name, '__' and time() in a var $str
encrypt $str using MCRYPT_BLOWFISH, this may contain '/', '+'
re-encrypt using str2hex (got it from here)
put the encoded string as the 3rd argument in the link sent by
email, like,
http://xyz.com/users/resetpassword/3123213213ABCDEF238746238469898
-you can see that the url contains only 0-9 and A-Z.
When link from email is clicked, get the 3rd uri segment, use
hex2str() to decrypt to blowfish encrypted string, and then apply
blowfish decrypt to get the original string.
split with '__' to get the user name and time
I know that its almost a year till this question was asked, but I am hoping that someone will find this solution helpful after coming here by google.