So I have been working on an API for a little side project and all the user input won't be directly used for SQL or displayed to the users so do I need to sanitize it? If I do what should I do exactly?
I am currently checking to make sure if they integers, strings, arrays, etc. but other than that is there anything else I need to do?
The question is always for what purpose? If you just take values you get from a user and you don't do anything with them (you just store and display them), then there's nothing to sanitize. If you let those values "actively do" something, you may want to sanitize them to avoid them doing something you don't like.
For instance, you accept HTML input from a user and want HTML formatted content, but you want to avoid XSS problems; in this case you will want to selectively remove HTML elements, i.e. you want to sanitize the input.
// some HTML is allowed, but not everything
echo remove_unwanted_html_elements($_POST['content']);
If OTOH you do not allow HTML input to be interpreted anyway, i.e. whatever the user posts is just displayed literally back to him without any of being interpreted as HTML, then you do not need to sanitize anything. You may just need to escape the content according to its target format.
// don't care what the user enters, just display it right back as is
echo htmlspecialchars($_POST['content']);
Sanitization is only relevant if you evaluate the value in some not entirely predictable way. Sanitization means to take a value and change it into something else, typically removing something from it. This must be very targeted and purposeful since it can be a very error prone operation; you don't just sanitize data somehow just because. The other alternative is simple validation, i.e. checking that a value conforms to expected norms and otherwise rejecting it outright.
Even taking a supposed number entered by the user and casting it to an int is a very simple form of sanitization; it's effective since it means you are guaranteed to get a harmless number, but that number may or may not have anything to do with the value the user submitted. Validation may be the better option here.
Related
I have a simple app programmed in PHP using CodeIgniter 4 framework and, as a web application, it has some HTML forms for user input.
I am doing two things:
In my Views, all variables from the database that come from user input are sanitized using CodeIgniter 4's esc() function.
In my Controllers, when reading HTTP POST data, I am using PHP filters:
$data = trim($this->request->getPost('field', FILTER_SANITIZE_SPECIAL_CHARS));
I am not sure if sanitizing both when reading data from POST and when printing/displaying to HTML is a good practice or if it should only be sanitized once.
In addition, FILTER_SANITIZE_SPECIAL_CHARS is not working as I need. I want my HTML form text input to prevent users from attacking with HTML but I want to keep some 'line breaks' my database has from the previous application.
FILTER_SANITIZE_SPECIAL_CHARS will NOT delete HTML tags, it will just store them in the database, not as HTML, but it is also changing my 'line breaks'. Is there a filter that doesn't remove HTML tags (only stores them with proper condification) but that respects \n 'line breaks'?
You don't need to sanitize User input data as explained in the question below:
How can I sanitize user input with PHP?
It's a common misconception that user input can be filtered. PHP even
has a (now deprecated) "feature", called
magic-quotes,
that builds on this idea. It's nonsense. Forget about filtering (or
cleaning, or whatever people call it).
In addition, you don't need to use FILTER_SANITIZE_SPECIAL_CHARS, htmlspecialchars(...), htmlentities(...), or esc(...) either for most use cases:
-Comment from OP (user1314836)
I definitely think that I don't need to sanitize user-input data
because I am not writing SQL directly but rather using CodeIgniter 4's
functions to create SQL safe queries. On the other hand, I do
definitely need to esc() that same information when showing to avoid
showing html where just text is expected.
The reason why you don't need the esc() method for most use cases is:
Most User form input in an application doesn't expect a User to submit/post HTML, CSS, or JavaScript that you plan on displaying/running later on.
If the expected User input is just plain text (username, age, birth date, etc), images, or files, use form validation instead to disallow unexpected data.
I.e: Available Rules and Creating Custom Rules
By using the Query Builder for your database queries and rejecting unexpected User input data using validation rules (alpha, alpha_numeric_punct, numeric, exact_length, min_length[8], valid_date, regex_match[/regex/], uploaded, etc), you can avoid most potential security holes i.e: SQL injections and XSS attacks.
Answer from steven7mwesigwa gets my vote, but here is how you should be thinking about it.
Rules Summary
You should always hold in memory the actual data that you want to process.
You should always convert the data on output into a format that the output can process.
Inputs:
You should strip from all untrusted inputs (user forms, databases that you didn't write to, XML feeds that you don't control etc)
any data that you are unable to process (e.g. if you are not able to handle multi-byte strings as you are not using the right functions, or your DB won't support it, or you can't handle UTF8/16 etc, strip those extra characters you can't handle).
any data that will never form part of the process or output (e.g. if you can only have an integer/bool than convert to int/bool; if you are only showing data on an HTML page, then you may as well trim spaces; if you want a date, strip anything that can't be formatted as a date [or reject*]).
This means that many "traditional" cleaning functions are not needed (e.g. Magic Quotes, strip_tags and so on): but you need to know you can handle the code. You should only strip_tags or escape or so on if you know it is pointless having that data in that field.
Note: For user input I prefer to hold the data as the user entered and reject the form allowing them to try again. e.g. If I'm expected a number and I get "hello" then I'll reload the form with "hello" and tell the user to try again. steven7mwesigwa has links to the validation functions in CI that make that happen.
Outputs:
Choose the correct conversion for the output: and don't get them muddled up.
htmlspecialchars (or family) for outputting to HTML or XML; although this is usually handled by any templating engine you use.
Escaping for DB input; although this should be left to the DB engine you use (e.g. parameterised queries, query builder etc).
urlencode for outputting a URL
as required for saving images, json, API responses etc
Why?
If you do out output conversion on input, then you can easily double-convert an input, or lose track of if you need to make it safe before output, or lose data the user wanted to enter. Mistakes happen but following clean rules will prevent it.
This also mean there is no need to reject special characters (those forms that reject quote marks are horrible user experience, for example, and anyone putting restrictions on what characters can go in a password field are only weakening security)
In your particular case:
Drop the FILTER_SANITIZE_SPECIAL_CHARS on input, hold the data as the user gave it to you
Output using template engine as you have it: this will display < > tags as the user entered then, but won't break your output.
You will essentially sanitize each and every output (that you appear to want to avoid), but that's safer than accidentally missing a sanitize on output and a better user experience than losing stuff they typed.
From my understanding,
FILTER_SANITIZE_SPECIAL_CHARS is used to sanitize the user input before you act on it or store it.
Whereas esc is used to escape HTML etc in the string so they don't interfere with normal html, css etc. It is used for viewing the data.
So, you need both, one for input and the other for output.
Following from codeigniter.com. Note, it uses the Laminas Escaper library.
esc($data[, $context = 'html'[, $encoding]])
Parameters
$data (string|array) – The information to be escaped.
$context (string) – The escaping context. Default is ‘html’.
$encoding (string) – The character encoding of the string.
Returns
The escaped data.
Return type
mixed
Escapes data for inclusion in web pages, to help prevent XSS attacks. This uses the Laminas Escaper library to handle the actual filtering of the data.
If $data is a string, then it simply escapes and returns it. If $data is an array, then it loops over it, escaping each ‘value’ of the key/value pairs.
Valid context values: html, js, css, url, attr, raw
From docs.laminas.dev
What laminas-Escaper is not
laminas-escaper is meant to be used only for escaping data for output, and as such should not be misused for filtering input data. For such tasks, use laminas-filter, HTMLPurifier or PHP's Filter functionality should be used.
Some of the functions they do are similar. Such as both may/will convert < to <. However, your stored data may not have come just from user input and it may have < in it. It is perfectly safe to store it this way
but it needs to be escaped for output otherwise the browser could get confused, thinking its html.
I think for this situation using esc is sufficient. FILTER_SANITIZE_SPECIAL_CHARS is a PHP sanitize filter that encode '"<>& and optionally strip or encode other special characters according to the flag. To do that you need to set the flag. It is third parameter in getPost() method. Here is an example
$this->request->getPost('field', FILTER_SANITIZE_SPECIAL_CHARS, FILTER_FLAG_ENCODE_HIGH)
This flag can be change according to your requirements. You can use any PHP filter with a flag. Please refer php documentation for more info.
I have a dynamic PHP web app which gets input params in the url (no surprise here). However, bingbot sometimes requests etremely long URLs from the site. E.g. > 10000 characters long urls. One of the inputs is an UTF name and bingbot somehow submits sketchy input names, thousands of characters long like this: \xc2\x83\xc3\x86... (goes on for thousands of characters).
Obviously, it gets a 404, because there is no such name in the database (and therefore no such page), but it occurred to me whether it's worth it to check the input length before querying the db (e.g. a name cannot be more than 100 characters long) and return a 404 instantly if it's too long. Is it standard practice? Or it's not worth the trouble, because the db handles it?
I'm thinking of not putting extra load on the db unnecessarily. Is this long input submitted as is by the db client interface (two calls: first a prepare for sanitizing the input and then the actual query) or the php db client knows the column size and truncates the input string before sending it down the wire?
Not only what you're asking is more than legit, but I'd say it's something that you should be doing as part of the input filtering/validation. If you expect your input to be always shorter than 100 characters, everything that's longer should be filtered.
Also, it appears that you're getting UTF-8 strings: if you're not expecting them, you could simply filter out all characters that are not part of the standard ASCII set (even reduced, filtering all control characters away. For example $string = filter_var($input, FILTER_SANITIZE_FULL_SPECIAL_CHARS, FILTER_FLAG_STRIP_LOW).
This is not just a matter of DB performance, but also security!
PS: I hardly doubt that bot is actually Bing. Seems like a bot trying to hack your website.
Addendum: some suggestions about input validation
As I wrote above in some comments (and as others have written too), you should always validate every input. No matter what is that or where it comes from: if it comes from outside, it has to be validated.
The general idea is to validate your input accordingly to what you're expecting. With $input any input variable (anything coming from $_GET, $_POST, $_COOKIE, from external API's and from many $_SERVER variables as well - plus anything more that could be altered by a user, use your judgement and in doubt be overly cautious).
If you're requesting an integer or float number, then it's easy: just cast the input to (int) or (float)
$filtered = (int)$input;
$filtered = (float)$input;
If you're requesting a string, then it's more complicated. You should think about what kind of string you are requesting, and filter it accordingly. For example:
If you're expecting a string like a hexadecimal id (like some databases use), then filter all characters outside the 0-9A-Fa-f range: $filtered = preg_replace('/[^0-9A-Fa-f]/', '', $input);
If you're expecting an alphanumeric ID, filter it, removing all characters that are not part of that ASCII range. You can use the code posted above: $string = filter_var($input, FILTER_SANITIZE_FULL_SPECIAL_CHARS, FILTER_FLAG_STRIP_LOW);. This one removes all control characters too.
If you're expecting your input to be Unicode UTF-8, validate it. For example, see this function: https://stackoverflow.com/a/1523574/192024
In addition to this:
Always encode HTML tags. FILTER_SANITIZE_FULL_SPECIAL_CHARS will do that as well on filter_var. If you don't do that, you risk XSS (Cross-Site Scripting) attacks.
If you want to remove control characters and encode HTML entities but without removing the newline chracters (\n and \r), then you can use: $filtered = preg_replace('/[\x00-\x09\x0B\x0C\x0E-\x1F\x7F]/u', '', htmlspecialchars($input, ENT_COMPAT, 'UTF-8'));
And much more. Use your judgement always.
PS: My approach to input filtering is to prefer sanitization. That is, remove everything "dangerous" and accept the sanitized input as if that was what the user wrote. Other persons will instead argue that input should only be accepted or refused.
Personally, I prefer the "sanitize and use" approach for web applications, as your users still may want to see something more than an error web page; on desktop/mobile apps I go with the "accept or refuse" method instead.
However, that's just a matter of personal preference, backed only by what my guts tell me about UX. You're free to follow the approach you prefer.
There should be some sort of validation done on any data before it is used in a query. If you have a limit on the length of the name, then you could use that as part of the validation when checking the input. If it's over the limit, it can't be in there and then handle it accordingly. Whether it's a 404 or a page that displays an error message.
The load will go down if you are bypassing queries because a name is too long. Depending on how you are querying the database, LIKE or MATCH AGAINST and how your indexes are set up, will determine just how much load will go down.
Currently I am upgrading a web application in which I will get most of the input from logged in users. The input will contains valid html, images, audio, video & upload facilities to user defined path. The application then formats it into nice ui and displays to end users. These privileged users can add / modify / delete the content using a web based interface.
As per the basic rule of thumb: I should escape my data before entering in DB, and not to receive data receive from user. To achieve that I have planned to follow following security measures. Which also includes my questions
I am using prepared statements to store all user inputs to DB. I hope this eliminates the DB injection threat.
Is this measure enough? or do i need to check for % and _ symbols as well for mysql LIKE queries?
The user input (lets call input A), where I am not expecting any HTML/css, I use strip_tags & htmlentities before inserting in DB.
Is this adequate measure ? Should I be using more
The user input (lets call input B), in which user can have html/css tags, I user htmlentities on text then insert in DB.
As far as I am aware I should not use htmlentities before inserting in the DB, but have to as previous programmer was using it. Are there any negative impacts for this?
After fetching from DB and Before displaying the input A / input B , I am not doing any pre processing assuming, the data added to DB should be clean.
Should i process / sanitize the data before displaying ? If yes then how ?
I want to html tags enters by user to be parsed by browser and not displayed to user. e.g. if user had entered <p style='color:red;'>hello</p><p class='noclass'>world</p>, I want user to see 2 words only and not actual text.
To achieve this how can I make sure that user doesn't add malicious script and at the same time the html tags are stored, fetched and parsed by browser correctly.
Please guide if the current approach is sufficient / not sufficient / less / incorrect.
I am neither a 100% newbie to php nor I m pro. I know the basics about php (or we can say over all web applications') security. So can someone can please guide me if I am making any mistake security wise OR should not be doing something OR should be doing something more or less.
I know the basics of security but I still get confused over
Which exact security measure to apply at which exact point ? (e.g. escape string BEFORE inserting to DB)
At every point what the functions available in php? (e.g. to escape strings use prepared statements)
Yes, prepared statements are great at preventing SQL injections problems. Yes, you will have to take care of % and _ in LIKE queries, a prepared statement cannot escape them since it has no way to know whether you want those values there or not.
through 5.: It's always a bad idea to escape data going into the database for a format it's destined for on output. Why? First of all, why are you so sure you're always going to use the data in an HTML context? Maybe you'll be using it in a different format in the future, and then you'll have garbage looking data. (This is more hypothetical in your case, as you're explicitly storing HTML.)
Secondly though, your output code will have to rely on your input code to correctly have escaped data in advance, possibly with a long time between input and output. Your output code can have no confidence whatsoever that the input code did the correct job for what the output code needs it to do. Therefore, escaping for output must happen at the time of output. No sooner, no later.
Thirdly (is that a word?), strip_tags is absolutely insufficient to accept some HTML but not other "insecure" HTML. You need a more complex library which has more complex whitelisting rules than what strip_tags can do. Supposedly the only library that does that is HTML Purifier. I'd run all user HTML through it.
To summarise:
Prepared statements.
HTML-escape data that is not supposed to contain literal HTML on output.
Run any data that is supposed to contain literal HTML through HTML Purifier. Whether you do this before or after inserting to the database is up to you, depending on whether you want to store the literal input the user sent you or whether you don't mind discarding that original data immediately and storing only sanitised data instead. But, the same caveat about having confidence in your output code applies too.
I am trying to figure out what is the best way to manage the data a user inputs concerning non desirable tags he might insert:
strip_tags() - the tags are removed and they are not inserted in the database
the tags are inserted in the database, but when reading that field and displaying it to the user we would use htmlspecialchars()
What's the better, and is there any disadvantage in any of these?
Regards
This depends on what your priority is:
if it's important to display special characters from user input (like on StackOverflow, for example), then you'll need to store this information in the database and sanitize it on display - in this case, you'll want to at least use htmlspecialchars() to display the output (if not something more sophisticated)
if you just want plain text comments, use strip_tags() before you stick it in the database - this way you'll reduce the amount of data that you need to store, and reduce processing time when displaying the data on the screen
the tags are inserted in the database, but when reading that field and displaying it to the user we would use htmlspecialchars()
This. You usually want people to be able to type less-than signs and ampersands and have them displayed as such on the page. htmlspecialchars on every text-to-HTML output step (whether that text came directly from user input, or from the database, or from somewhere else entirely) is the right way to achieve this. Messing about with the input is a not-at-all-appropriate tactic for dealing with an output-encoding issue.
Of course, you will need a different escape — or parameterisation — for putting text in an SQL string.
The measures taken to secure user input depends entirely on in what context the data is being used. For instance:
If you're inserting it into a SQL database, you should use parameterized statements. PHP's mysql_real_escape_string() works decently, as well.
If you're going to display it on an HTML page, then you need to strip or escape HTML tags.
In general, any time you're mixing user input with another form of mark-up or another language, that language's elements need to be escaped or stripped from the input before put into that context.
The last point above segues into the next point: Many feel that the original input should always be maintained. This makes a lot of sense when, later, you decide to use the data in a different way and, for instance, HTML tags aren't a big deal in the new context. Also, if your site is in some way compromised, you have a record of the exact input given.
Specifically related to HTML tags in user input intended for display on an HTML page: If there is any conceivable reason for a user to input HTML tags, then simply escape them. If not, strip them before display.
As I prepare to tackle the issue of input data filtering and sanitization, I'm curious whether there's a best (or most used) practice? Is it better to filter/sanitize the data (of HTML, JavaScript, etc.) before inserting the data into the database, or should it be done when the data is being prepared for display in HTML?
A few notes:
I'm doing this in PHP, but I suspect the answer to this is language agnostic. But if you have any recommendations specific to PHP, please share!
This is not an issue of escaping the data for database insertion. I already have PDO handling that quite well.
Thanks!
When it comes to displaying user submitted data, the generally accepted mantra is to "Filter input, escape output."
I would recommend against escaping things like html entities, etc, before going into the database, because you never know when HTML will not be your display medium. Also, different types of situations require different types of output escaping. For example, embedding a string in Javascript requires different escaping than in HTML. Doing this before may lull yourself into a false sense of security.
So, the basic rule of thumb is, sanitize before use and specifically for that use; not pre-emptively.
(Please note, I am not talking about escaping output for SQL, just for display. Please still do escape data bound for an SQL string).
i like to have/store the data in original form.
i only escape/filter the data depending on the location where i'm using it.
on a webpage - encode all html
on sql - kill quotes
on url - urlencoding
on printers - encode escape commands
on what ever - encode it for that job
There are at least two types of filtering/sanitization you should care about :
SQL
HTML
Obviously, the first one has to be taken care of before/when inserting the data to the database, to prevent SQL Injections.
But you already know that, as you said, so I won't talk about it more.
The second one, on the other hand, is a more interesting question :
if your users must be able to edit their data, it is interesting to return it to them the same way they entered it at first ; which means you have to store a "non-html-specialchars-escaped" version.
if you want to have some HTML displayed, you'll maybe use something like HTMLPurifier : very powerful... But might require a bit too much resources if you are running it on every data when it has to be displayed...
So :
If you want to display some HTML, using a heavy tool to validate/filter it, I'd say you need to store an already filtered/whatever version into the database, to not destroy the server, re-creating it each time the data is displayed
but you also need to store the "original" version (see what I said before)
In that case, I'd probably store both versions into database, even if it takes more place... Or at least use some good caching mecanism, to not-recreate the clean version over and over again.
If you don't want to display any HTML, you will use htmlspecialchars or an equivalent, which is probably not that much of a CPU-eater... So it probably doesn't matter much
you still need to store the "original" version
but escaping when you are outputing the data might be OK.
BTW, the first solution is also nice if users are using something like bbcode/markdown/wiki when inputting the data, and you are rendering it in HTML...
At least, as long as it's displayed more often than it's updated -- and especially if you don't use any cache to store the clean HTML version.
Sanitize it for the database before putting it in the database, if necessary (i.e. if you're not using a database interactivity layer that handles that for you). Sanitize it for display before display.
Storing things in a presently unnecessary quoted form just causes too many problems.
I always say escape things immediately before passing them to the place they need to be escaped. Your database doesn't care about HTML, so escaping HTML before storing in the database is unnecessary. If you ever want to output as something other than HTML, or change which tags are allowed/disallowed, you might have a bit of work ahead of you. Also, it's easier to remember to do the escaping right when it needs to be done, than at some much earlier stage in the process.
It's also worth noting that HTML-escaped strings can be much longer than the original input. If I put a Japanese username in a registration form, the original string might only be 4 Unicode characters, but HTML escaping may convert it to a long string of "〹𐤲䡈穩". Then my 4-character username is too long for your database field, and gets stored as two Japanese characters plus half an escape code, which also probably prevents me from logging in.
Beware that browsers tend to escape some things like non-English text in submitted forms themselves, and there will always be that smartass who uses a Japanese username everywhere. So you may want to actually unescape HTML before storing.
Mostly it depends on what you are planning to do with the input, as well as your development environment.
In most cases you want original input. This way you get the power to tweak your output to your heart's content without fear of losing the original. This also allows you to troubleshoot issues such as broken output. You can always see how your filters are buggy or customer's input is erroneous.
On the other hand some short semantic data could be filtered immediately. 1) You don't want messy phone numbers in database, so for such things it could be good to sanitize. 2) You don't want some other programmer to accidentally output data without escaping, and you work in multiprogrammer environment. However, for most cases raw data is better IMO.