I need some suggestions and ideas.
Here's the scenario. The server receives a bunch of IDs from client via Ajax. Some of these IDs may already exist in database some may not. I need to save those that are not.
One way would be to set sql queries to select * whose ID is what I have. But this requires to a select statement for each id. Each time I receive something about 300 IDs which means 300 sql queries. This I think would slow the server. So what do you think is a better way to do this? Is there a way to extract the non-existing IDs with one SQL query?
P.S. The server is running on CakePHP.
I think what you need is SQL's IN keyword:
SELECT id FROM table WHERE id IN (?)
Where you would insert your IDs separated by comma, e.g.
$id_str = implode(',', $ids);
Make sure that $ids is an array of integers to prevent SQL injection
The outcome is a MySQL result containing all ids that exist. Build them into an array and use PHP's array_diff to get all IDs that do not exist. Full code:
$result = $connection->query('SELECT id FROM table WHERE id IN ('.
implode(',', $ids) . ')');
while($row = $result->fetch_row()) {
$existent[] = $row[0];
}
$not_existent = array_diff($ids, $existent);
If I understand you correctly, an insert ignore could do the trick
INSERT IGNORE INTO `table` (`id`,`col`,`col2`) VALUES ('id','val1','val2');
then any duplicate id's will be silently dropped, so long as id is unique or primary.
Also the keyword IN can be useful for finding rows with a value in a set. Eg
SELECT * FROM `table` WHERE `id` IN (2,4,6,7)
Related
Okay, basically I have a table that contains statements like:
incident.client_category = 1
incident.client_category = 8
incident.severity = 1
etc.
I would like to use the contents from this table to generate other tables that fulfill the conditions expressed in this one. So I would need to make it something like
SELECT * FROM incident WHERE incident.client_category = 1
But the last part of the where has to come from the first table. Right now what I'm trying to do is something like
SELECT * FROM incident WHERE (SELECT condition FROM condition WHERE id = 1)
id = 1 stands for the condition's id. Right now I only want to work with ONE condition for testing purposes. Is there a way to achieve this? Because if there isn't, I might have to just parse the first query's results through PHP into my incident query.
Table schemas:
Engineering Suggestion - Normalize the DB
Storing a WHERE clause, like id = 10, in a field in a MySQL table, is not a good idea. I recommend taking a look at MySQL Normalization. You shouldn't store id = 10 as a varchar, but rather, you should store something like OtherTableid. This allows you to use indices, to optimize your DB, and to get a ton of other features that you are deprived of by using fields as WHERE clauses.
But sometimes we need a solution asap, and we can't re-engineer everything! So let's take a look at making one...
Solution
Here is a solution that will work even on very old, v. 5.0 versions of MySQL. Set the variable using SET, prepare a statement using PREPARE, and execute it using EXECUTE. Let's set our query into a variable...
SET #query = CONCAT(
"SELECT * FROM incident WHERE ",
(SELECT condition FROM condition WHERE id = 1)
);
I know for a fact that this should work, because the following definitely works for me on my system (which doesn't require building any new tables or schema changes)...
SET #query = CONCAT("SELECT id FROM myTable WHERE id = ", (SELECT MAX(id) FROM myTable));
If I SELECT #query;, I get: SELECT id FROM myTable WHERE id = 1737901. Now, all we need to do is run this query!
PREPARE stmt1 FROM #query;
EXECUTE stmt1;
DEALLOCATE PREPARE stmt1;
Here we use a prepare to build the query, execute to execute it, and deallocate to be ready for the next prepared statement. On my own example above, which can be tested by anyone without DB schema changes, I got good, positive results: EXECUTE stmt1; gives me...
| id | 1737901 | .
here is one way to achieve your goal by using what is called dynamic sql, be ware that this works only select from condition table returns only one record.
declare #SQLSTRING varchar(4000)
, #condition VARCHAR(500) -- change the size to whatever condition column size is
SELECT #condition = condition
FROM
condition
WHERE
id = 1
SET #SQLSTRING= 'SELECT * FROM incident WHERE ' + #condition
exec sp_executesql(#SQLSTRING)
Since you have also tagged the question with PHP, I would suggest using that. Simply select the string from the condition table and use the result to build up a SQL query (as a string in PHP) including it. Then run the second query. Psudo-code (skipping over what library/framework you re using to call the db):
$query = "select condition from condition where id = :id";
$condition = callDbAndReturnString($query, $id);
$query = "select * from incident where " . $condition;
$result = callDb($query);
However, be very careful. Where and how are you populating the possible values in the condition table? Even how is your user choosing which one to use? You run the risk of opening yourself up to a secondary SQL injection attack if you allow the user to generate values and store them there. Since you are using the value from the condition table as a string, you cannot parametrise the query using it as you (hopefully!) normally would. Depending on the queries you run and the possible values there as conditions, there might also be risk even if you just let them pick from a pre-built list. I would seriously ask myself if this (saving parts of SQL queries as strings in another table) is the best approach. But, if you decide it is, this should work.
I am trying to make an inventory / invoice web application. The user enters information such as order ID, date, order total, and then each of the products bought along with their respective quantity. I'm using PDO for the sql queries.
I do not know in advance how many unique products are going to be in an invoice so I have an associative array that stores the products and their quantities (product name is used as the key) when the form is submitted.
On submit a prepared statement is built/executed.
Right now I have the order_id, date, and order_total query done.
$stmt = $connection->prepare("INSERT INTO table_1 (order_id, order_date, order_total) VALUES ('$orderid', '$date', '$total_cost')");
$stmt->execute();
That part is simple enough. The aim of the other query is the following.
$testStmt = $connection->prepare("INSERT INTO table_2 (keys from the assoc array are listed here) VALUES (values from the assoc arrays are listed here)");
$testStm->execute();
My array would end up looking like this once the user inputs some products:
$array
(
"product1" => quantity1
"product2" => quantity2
)
The idea I have had so far is to make a string for columns that need to be included in the sql query and then a string for the values for the sql query. Then iterate through the array and append the keys and values to the respective strings in such a way that I could use them in the sql query. I haven't gotten it to work and am worried that it could open myself up to sql injection (I am still quite unfamiliar with sql injection so I have been trying to read up on it).
$columns;
$values_input;
foreach($assoc_array as $product => $quant)
{
$columns .= "' " . $product . "', ";
$values_input .= "' " . $quant . "', ";
}
The idea being that $columns and $values_input string would end up containing all the appropriate column names and the quantities to be entered into those columns. Then I figured I could be able to use those strings as part of the SQL query. Something like this.
INSERT INTO $columns VALUES $values_input
I'd appreciate any help or insight. If I'm way off here or doing something in a retarded way feel free to shout about it, I'd rather fix a screw up than continue on with it if that's the case.
You are already using PDO, which is a good thing if you want to protect yourself from SQL injection. You are even trying to prepare your statement, but since you are not binding any parameters, one could argue if that is really what you are doing. Example 5 on the PHP docs page is in fact pretty close to what you want to do. Allow me to adapt it to your use case:
// create a placeholders string looking like "?, ?, ..., ?"
$placeholders = implode(',', array_fill(0, count($params), '?'));
// prepare the statement
$qry = $connection->prepare("INSERT INTO table_2 ($params) VALUES ($params)");
// bind the parameters to the statement. (We first need all columns, then all values)
$qry->execute(array_merge(array_keys($params), array_values($params)));
This should result in a query that looks exactly like your first example, but with a dynamic number of columns, or parameters. And since you are preparing your statement and binding the parameters on execution, PDO should handle all quoting and escaping to prevent SQL injection.
As a side note, your table structure seems a bit of to me. I don't think you normalized your data correctly, though it is a bit hard to tell with the table names you are using. I believe your structure should look something like this, and I fear it doesn't:
TABLE orders (id, date, total, client_id)
TABLE products (id, name, price, ...)
TABLE order_lines (id, order_id, product_id, quantity)
TABLE clients (...)
The exact structure obviously depends on your use case, but I believe this is about the simplest structure you can get away with if you want to build an order system that you can easily query and that can serve as a base for possible expansion in the future.
Since you are trying to make an inventory/invoice application, do you happen to have a product database? If you do, you may want to use the product id instead of the product names as key. As product names sound like there could be duplicates or can change. If product name changes, you will have problems querying.
Do you accept products not in db to be entered into the invoice? If so, it adds some complications.
On SQL injections, you should sanitize input before using it for queries. Read: What's the best method for sanitizing user input with PHP?
Most modern frameworks have many built in protections against SQL injections if you do not query manually. So consider using them.
Many of them use active record pattern see: http://www.phpactiverecord.org/projects/main/wiki/Basic_CRUD (So you don't have to deal with writing queries manually like you do.)
An example of active record in a framework: https://www.codeigniter.com/user_guide/database/query_builder.html
I am trying to display the data from 'table' if a key inputted by the user is found in the database. Currently I have it set up so that the database checks if the key exists, like so:
//Select all from table if a key entry that matches the user specified key exists
$sql = 'SELECT * FROM `table` WHERE EXISTS(SELECT * FROM `keys` WHERE `key` = :key)';
//Prepare the SQL query
$query = $db->prepare($sql);
//Substitute the :key placeholder for the $key variable specified by the user
$query->execute(array(':key' => $key));
//While fetched data from the query exists. While $r is true
while($r = $query->fetch(PDO::FETCH_ASSOC)) {
//Debug: Display the data
echo $r['data'] . '<br>';
}
These aren't the only SQL statements in the program that are required. Later, an INSERT query along with possibly another SELECT query need to be made.
Now, to my understanding, using WHERE EXISTS isn't always efficient. However, would it be more efficient to split the query into two separate statements and just have PHP check if any rows are returned when looking for a matching key?
I took a look at a similar question, however it compares multiple statements on a much larger scale, as opposed to a single statement vs a single condition.
#MarkBaker Join doesn't have to be faster than exists statement. Query optymalizer is able to rewrite the query live if it sees better way to accomplish query. Exists statement is more readable than join.
Fetching all the data and making filtering directly in PHP is always bad idea. What if your table grow up to milions of records? MySQL is going to find the best execute plan for you. It will automaticaly cache the query if it is going to improve performance.
In other words, your made everything correctly as far as we can see your code now. For futher analyse show us all of your queries.
I'm using PHP PDO to access a sql database. There's a table with just two columns (ID and VALUE). I want to read that table into an array such that
$array[ID]=VALUE
I know how I can do it manually with a for loop or while loop going through one by one… but I was wondering if there's any better way of doing it.
You can use PDO::FETCH_KEY_PAIR constant
$sql = "select id, username from users limit 10";
$data = $pdo->query($sql)->fetchAll(PDO::FETCH_KEY_PAIR);
I have an html page where I collect an array of values from checkboxes to insert in a database. The html page posts to a PHP page that collects the data and then stores in the database.
For each value, there are a few other fields I would like to include that are the same for all the values such as time entered. I can easily convert the captured array into a comma delimited list using implode. I use such a comma delimited list of ids to update and delete records. However, when I want to insert them, MYSQL does not seem to allow you to use a comma delimited list. My question is, what is the easiest way to insert records, one for each value in the c comma delimited list, without using a loop.
html page
<input type="checkbox" name="var[]" value=1>
<input type="checkbox" name="var{}" value=2>
PHP page
$vars = $_POST['var'];
This gives me an array that I can convert to a comma delimited list using implode.
To delete, I can go
$sql = "DELETE * from table WHERE id in '$vars'";
To update I can go
$sql = "UPDATE table, WHERE id in '$vars'";
But there does not seem to be an equivalent for Insert.
Following would work:
$sql = "INSERT into table (var, timeentered) values (1,now()) (2,now())";
However, that's not how I have my data. what I would like to do is something like
$sql = "INSERT into table (var,timeentered) values($vars), now()" but of course that doesn't work.
Do I have to convert my nice comma delimited list that works so well for update and delete into something that looks like (1,now) (2, now()) for inserting or is there an alternative?
Thanks for any suggestions.
Unfortunately you have to build whole query by yourself:
$sql ="insert into table (var, timeentered) values ";
$sql .= "(".implode(", now()), (", $vars).")";
You need to loop through your data set and create the multi-line insert query manually. There is no other alternative if you want to insert multiple rows with a single query. That is outside of using certain DB frameworks which might present a better interface for doing this. Of course at the end of the day, such a DB framework would in essence be building the multi-item insert query manually at some point.
The question might come done to one of how many items are you going to insert. If you are only going to be inserting a few records at a time, then you might want to consider just using prepared statements with individual inserts. However if you are going to be inserting hundreds of records at a time, that would probably not be a good idea.
In your mysql database you can set the default for the column "time_created" to be a TIMESTAMP default CURRENT_TIMESTAMP. This way you don't have to worry about it. Just use the regular insert and it will automatically set the "time_created" column.
For your other issue of multi-line inserts you can create an $update array and use a foreach loop to issue a sql insert command on every row of data.
Two options I can think of.
Build a dynamic insert query like you suggest. However do not call now() each time but just insert a single date ie
$time = gmdate();
$sql = "INSERT into table (var, timeentered) values (1,$time) (2,$time)";
Or use a prepared statement of the single insert below, turn off autocommit, start a transaction and execute the prepared statement in a for loop for the number of inserts needed, then commit the transaction.
$sql = "INSERT into table (var, timeentered) values (?,?)"
Mostly you will have to build your query using some type of looping structure. Convention and best practice aside if you just want to know how to make your array acceptable for a multiple insert statement then why not just do this:
$values = '('.implode('),(', $array).')';
or if already CSV then:
$values = '('.implode('),(', explode(',' $csv)).')';
then you can just use $values in your query using double quotes.