I'm using cURL to pull selected data out of web pages, until recently I've been recording the output to plain text files but now that I have a better appreciation of MySQL/SQLite I've decided to make the full switch to relational databases, just makes managing data so much easier.
But I've run into a couple of issues I haven't been able to solve on my own:
1) EDIT: problem solved by simply using sqlite_escape_string individually.
However the other problem remains:
2) The content that gets inserted into the database appears to have it's encoding messed up e.g. ™ becomes â„¢, ' becomes ’ and so forth. This happens ONLY when it is being processed by SQLite, if I otherwise echo the exact same data out or save it to text file the encoding (UTF-8) is preserved and everything renders properly.
I've tried using utf8_encode(); prior to insertion but that just messes things up even more. I should note that manual editing after insertion works so the issue is clearly not with the DB itself but with the way the data is being inserted into it
Here's the current code:
$name = sqlite_escape_string($name);
$category = sqlite_escape_string($category);
$html = sqlite_escape_string($html);
$db = new SQLiteDatabase('DB.sqlite');
$query = 'INSERT INTO Table (Name, Category, Html)' . "VALUES ('$name', '$category', '$html')";
$db->queryExec($query);
Apparently others are having similar issues with this:
Link
Link
$insert = compact('name','category','html');
$insert = array_map('sqlite_escape_string',$insert);
$insert = '"'. implode('", ',$insert).'"';
Or one-liner:
$insert = '"'. implode('", ', array_map('sqlite_escape_string', compact('name','category','html'))).'"';
Related
It sounds strange to me. I have a simple PHP script that inserts data into MYSQL table.
Upon receiving the content from the client via AJAX the data is stored in a variable:
$content=$_POST['content'];
$sql="insert into contents values('$content')";
mysql_query($sql);
The problem is that if the content contains a '&' symbol,the sub-string before & is stored in MYSQL and the rest of the string is discarded. If I try directly in MYSQL then it stores complete string containg & symbol.why?
The problem is that mysql regocnizes '&' as AND. Check this out:
$content = mysql_real_escape_string($_POST['content']);
$sql = "insert into contents (column) values('$content')";
mysql_query($sql);
First off if this site is live take it down lol. This is classic sql injection vulnerability.
You need to be using mysqli now instead of mysql.
The way you use this is the same but it has this REALLY cool feature called 'real escape string'
What it does is parameterize the data before you pass it into the database
$content = $_POST['content'];
$connection = new mysqli('ipaddress','username','password','database');
$content = $connection->real_escape_string($content);
$sql="insert into contents values('$content')";
$connection->query($sql);
This is a much safer way of passing in data
My method $csv->getContent(); contains my column names and the 'data' is my individual rows to import to the database. The loop below works flawlessly when importing my various CSVs to the database. However, when I have a column in the CSV that has html content, many rows import fine but there are some that do not and I get the SQLSTATE[HY093]: "Invalid parameter number: number of bound variables does not match number of tokens" error.
From my research, I am seeing that doing things the way I have shown below is supposed to properly escape characters to avoid the need for addslashes but it does not seem to make a difference. The only time this does work is if I addslashes(addslashes($values)); but thats kind of ridiculous and it leaves my content in the database with \\\ before the applicable characters.
Am I missing a step here or am I just stuck. The way I see it, I should be able to "bulletproof" my content to get it into my db each and every time.
$this->db->exec("SET CHARACTER SET utf8");
$this->db->beginTransaction();
$content = $csv->getContent();
foreach($content['data'] as $key => $value) {
$sql = "INSERT INTO `".$destination_table."` (`";
$sql .= implode("`, `", $content['columns']);
$sql .= "`) VALUES (";
$sql .= implode(", ", array_fill(0, count($content['columns']), "?"));
$sql .= ")";
$statement = $this->db->prepare($sql);
$statement->execute($value);
}
$this->db->commit();
I do not prefer your code. What if there are thousands of data and you loop each data then insert to database. There is a mysql query to import csv file to database table.
source: how-to-import-csv-file-to-mysql-table
Ok, I have found an answer to my question. I built a small app for personal use where I can keep the content for auctions I list online. Some of the content is old and contains mal-formed html so when parsing the csv and importing into the database, mal-formed html can really screw things up. However, after more research and reading, the "bindParam" and then "execute" methods of PDO allowed to seemlessly import the content no matter how poorly formed the html is. Here is how I structured my test query and then just kept experimenting with messed up html to the database.
$sql = $objDb->prepare('UPDATE `autolist_html` SET `value` = :html WHERE entity = "'.$entity[1].'"');
$sql->bindValue(':html',$_POST[$entity[1].'_html'],PDO::PARAM_STR);
$sql->execute();
Am trying to write a new record using the jquery function.
$.post("insertuser.php",$("#rohanStart").serialize(),function(data){
alert(data);
});
This seems to work and i do get an alert with the echo'ed statmenet. Problem is the values are not getting written into the database. Is there something wrong in the Query Statement?
mysql_query("INSERT INTO ajax_demo1( FirstName,LastName,Unit,Group,photo)
VALUES (
'".$arr['FirstName']."',
'".$arr['LastName']."',
'".$arr['Unit']."',
'".$arr['Group']."',
'".$arr['photo']."'
)");
echo $arr['Group'];
First don't use jQuery or any frameworks, they rely on the proprietary Microsoft JScript innerHTML method which does not work correctly with the DOM thus adding huge amounts of ambiguity in scripting.
Secondly you're NOT correctly escaping data going in to the database, that is a serious security issue.
Thirdly your approach to database queries is not taking error handling in to account, you're just dumping queries directly in and hoping for the best.
You should ALWAYS number your queries and enclose them as I have below. Note that besides errors it is good to fail conditions up-front however with database structure you should execute if successful first and THEN fail to increase your indentation (by a single space, not this tab waste where you have five screens to horizontally scroll) so you can visualize where you are in your own code.
$query1 = "SELECT * FROM table_name WHERE something='value'";
$result1 = mysql_query($query1);
if ($result1)
{
$row1 = mysql_fetch_assoc($result1);
}
else {mysql_error_report($query1,mysql_error(),__FUNCTION__);}
If your main header includes (you DO have a main header being included for all requests except AJAX correct?) you should have a universal MySQL error handling function that you can use to log SQL errors.
The following is the universal database error handler. You should have administrative error logs for HTTP, JavaScript, PHP and SQL errors so you can review and correct issues that your visitors encounter instead of if they only inconvenience you.
function mysql_error_report($q,$e,$f)
{
if (isset($_SESSION['database']))
{
if (isset($_SESSION['id_member'])) {$id = $_SESSION['id_member'];} else {$id = 0;}
if (isset($_SESSION)) {$session = mysql_real_escape_string(session_id());} else {$session = 0;}
$ip = mysql_real_escape_string(getenv('REMOTE_ADDR'));
$query = mysql_real_escape_string($q);
$error = mysql_real_escape_string($e);
$function = mysql_real_escape_string($f);
if (isset($_SESSION['type'])) {$type = mysql_real_escape_string($_SESSION['type']);} else if (isset($_SESSION['cms_browser'])) {$type = 'Browser';} else {$type = 'Unknown';}
if (isset($_SERVER['REQUEST_URI'])) {$url = $_SERVER['REQUEST_URI'];} else {$url = '';}
if (isset($_SERVER['HTTP_USER_AGENT'])) {$ua = mysql_real_escape_string($_SERVER['HTTP_USER_AGENT']);} else {$ua = '';}
$query1 = "INSERT INTO log_errors_sql (id_session, type, id_user, date, ip, function, mysql_error, mysql_query, url, user_agent) VALUES ('$session', '$type', '$id', NOW(), INET_ATON('$ip'), '$function', '$error', '$query', '$url', '$ua')";
$result1 = mysql_query($query1);
if (!$result1) {mysql_error_report_mail($q,$e,$f,$ua);}
}
else {mysql_error_report_mail($q,$e,$f);}
}
By using that approach you'll strengthen your coding practices to be much stricter. You don't want ambiguity, you want to be a total tightass about your code because the less subjectivity there is in your coding the more your code will be able to handle.
Also your white-space is very loose.
This...
INSERT INTO ajax_demo1( FirstName,LastName,Unit,Group,photo)
Should be formatted like this...
INSERT INTO ajax_demo1(FirstName, LastName, Unit, Group, photo)
You might ask why keeping your white-space like that is important, if you haven't spent a ton of time with find and replace (look up "Advanced Find & Replace", it works on wine/Linux and blows the crap away out of the native Linux console command performance wise and it's dirt cheap, supports regex, etc) you'll find yourself making mass site-wide edits in the blink of an eye because even your white-space is uniformly the same strict approach.
Should you heed my advice use a AFR (Advanced Find and Replace) to search for (but not replace) all instances of "mysql_query" and correct the formatting of everything you've written. Mix in a little AJAX notifications and you can see the errors instantly while you're still in the browser without a single alt-tab. That's how I roll.
...and of course doing this will make your debugging much easier. This isn't a fish, this is fishing and I hope it helps.
I want to load data from several CSV documents in a database.
The problem is that there is a table for a file, thus the table name has to be stored in a variable progressively updated by the loop. The tables are created, but after that the script fails; Nothing happens, my tables and their fields always have 0 lines.
I think that my idea is feasible, because some topics describe some tips in this way, but nothing works in my case.
This is my code that fails:
$query = "INSERT INTO". $name_of_the_file." (id, content) values (". $arrayGenre[0].",'".trim($arrayGenre[1])."')";
where $name_of_the_file refers to the table created in the loop, referring ultimately to the file targeted by the loop.
Following your recommendations, I added a space and print the query by:
echo ("<br>". $query);
Here is an example for the first line. No offense for the xxx substitutions: these are nominative data.
INSERT INTO names.csv (id, url, content) values (1,'xxx_some_adressxxx,'agency: xxx')
I do not know enough PHP to say that it is the expected result, but that seems a good SQL request and the values are fine.
Here are the next lines of my script:
$mysqli->query($query);
and at the beginning of my script, the definition of $mysqli:
$mysqli = new mysqli($dbhost, $dbuser, $dbpass, $dbname);
New edit :
Well it was in fact SQL code which was badly written...
Try adding a space:
$query = "INSERT INTO ". $name_of_the_file ...
It's always worth outputting your SQL via echo to make sure it looks like what you expect it to look like.
Edit:
INSERT INTO names.csv (id, url, content) values (1,'xxx_some_adressxxx,'agency: xxx')
You need to add an apostrophe at the end of the second field:
INSERT INTO names.csv (id, url, content) values (1,'xxx_some_adressxxx','agency: xxx')
I'd also recommend that you look into things like PDO, to make your code more robust and more secure.
You can also try pasting your SQL directly into the database, to see if it gives you an error.
I'm having a hard time quoting SQL string properly, I want to know if there's a program or a website that can help me quote the string correctly. Where I can enter the SQL strings then the website will analyze and quote it for me. Or maybe a program.. I need it cause I'm having a trouble with them..
Sorry for not providing a detailed information about my question, I have this SQL string. I got it from a book, But I'm having a hard time quoting it properly.
$sql = "INSERT INTO books(title, author, isbn, publisher, year, summary) VALUES (" .$conn->quote($_POST['title']) .
', ' . $conn->quote($_POST['author']) .
', ' . $conn->quote($_POST['isbn']) .
', ' . $conn->quote($_POST['publisher']) .
', ' . $conn->quote($_POST['year']) .
', ' . $conn->quote($_POST['summary']) .
')';
You can see that there are many quotes happening there! I got it from a book, but when it comes a time where I need to devise my own SQL string with the same difficulty like that, maybe I'll split.
Do you have a recommendation where a website or a program can help me escape/quote the strings properly?
Can you be a bit more specific? When writing ADO.NET code, you generally use a SqlParameter object like so, which takes care of all of this stuff automatically:
var cmd = new SqlCommand("select * from foo where fooName = #fooName;", connection);
cmd.Parameters.AddWithValue("#fooName", "O'Reilly is a bloviator");
cmd.ExecuteReader();
Okay, it looks like you're using PHP. You should not manually quote, but rather use prepared statements. Below is one way to do it, using PDO. Other valid syntaxes are given at PDOStatement->execute:
$stmt = $db->prepare("INSERT INTO books(title, author, isbn, publisher, year, summary) VALUES(:title, :author, :isbn, :publisher, :year, :summary)");
$title="Hitchhiker's Guide";
$author="Douglas Adams";
$isbn="0345391802";
$publisher="Del Rey";
$year="1995";
$summary="Arthur Dent accidentally saves the world.";
$stmt->bindParam(":title", $title);
$stmt->bindParam(":author", $author);
$stmt->bindParam(":isbn", $isbn);
$stmt->bindParam(":publisher", $publisher);
$stmt->bindParam(":year", $year);
$stmt->bindParam(":summary", $summary);
$stmt->execute();
The thing that helps me most when writing sql is a good text editor that highlights the text. There are many discussions about which text editor to use. I prefer vi, but I'm positive emacs would do it as well, along with eclipse and just about everything that isn't plain old windows notepad.
Do you have a recommendation where a
website or a program can help me
escape/quote the strings properly?
You might not need to escape the strings in php if a setting called magic_quotes_gpc is on. You can find out if it is on or not by writing a php file with this code in it:
<?php
var_dump(get_magic_quotes_gpc());
?>
It will show a bool(true) or bool(false). If it is false then use a function called mysql_real_escape_string to escape. If it is true, you don't need to do anything, your input will be escaped automatically. Remember, don't do both, you dont want to escape twice. Alternatively you could use this function which will find out if you need to escape or not:
<?php
function clean($input) {
if (get_magic_quotes_gpc()) {
// magic quotes are on, no need to escape
return $input;
} else {
// magic quotes are off, we need to escape
return mysql_real_escape_string($input);
}
?>
Then just use it like this:
<?php
$result = mysql_query(sprintf('SELECT * FROM `table` WHERE user="%s"', clean($_POST['user'])));
?>
I'm not able to check the code I've submitted for typos but hopefully...