working on the PHP project related to web scraping and my aim is to store the data into the mysql database,i'm using unique key index on 3 indexes in 9 columns table and records are more than 5k.
should i check for unique data at program level like putting values in arrays and then comparing before inserting into database ?
is there any way so that i can speed up my database insertion ?
Never ever create a duplicate table this is a anti SQL pattern and it makes it more difficult to work with your data.
Maybe PDO and prepared statement will give you a little boost but dont expect wonders from it.
multible INSERT IGNORE may also give you a little boost but dont expect wonders from it.
You should generate a multiinsert query like so
INSERT INTO database.table (columns) VALUES (values),(values),(values)
Keep in mind to keep under the max packet size that mysql will have.
this way the index file have to be updated once.
You could create a duplicate of the table that you currently have except no indices on any field. Store the data in this table.
Then use events to move the data from the temp table into the main table. Once the data is moved to the main table then delete if from the temp table.
you can follow your updates with triger. You should do update table and you have to right trigger for this table.
use PDO, mysqli_* function, to increase insertion into database
You could use "INSERT IGNORE" in your query. That way the record will not be inserted if any unique constraints are violated.
Example:
INSERT IGNORE INTO table_name SET name = 'foo', value = 'bar', id = 12345;
Related
I have a table users that has an auto-incrementing id column. For every new user, I essentially need to insert three rows into three different tables, which are 1. user, 2. user_content, and 3. user_preferences. The rows inserted into user_content and user_preferences are referenced by their id's which correspond to each user's id (held in user)
How do I accomplish this?
Should I do the INSERT INTO user query first, obtaining that auto-incremented id with last_insert_id(), and then the other two INSERT INTO queries using the obtained user id? Or, is there a more concise way to do this?
(note: I am using MySQL and PHP, and if it makes a difference, I am using a bigint to store the id values in all three tables.)
Thank you!
The approach that you've described (insert into user first, take the result of last_insert_id(), and use that to insert to the other two tables) is perfectly reasonable; I see nothing wrong with it.
It might be technically possible to combine the three queries and use the LAST_INSERT_ID() MySQL function to insert values to the other two tables, but this would be significantly more complex without any corresponding benefits. Not really worth doing, in other words.
I see 3 options:
PHP side using some *_last_insert_id (as you describe)
Create a trigger
Use a stored procedure.
Just looking for some tips and pointers for a small project I am doing. I have some ideas but I am not sure if they are the best practice. I am using mysql and php.
I have a table called nomsing in the database.
It has a primary key called row id which is an integer.
Then I have about 8 other tables referencing this table.
That are called nomplu, accsing,accplu, datsing, datplu for instance.
Each has a column that references the primary key of nomsing.
Withing my php code I have all the information to insert into the tables except one thing , the row id primary key of the nomsing table. So that php generates a series of inserts like the following.
INSERT INTO nomsing(word,postress,gender) VALUES (''велосипед","8","mask").
INSERT INTO nomplu(word,postress,NOMSING?REFERENCE) VALUES (''велосипеды","2",#the reference to the id of the first insert#).
There are more inserts but this one gets the point across. The second insert should reference the auto generated id for the first insert. I was this to work as a transaction so all inserts should complete or none.
One idea I have is to not auto generate the id and generate it myself in php. That way would know the id given before the transaction but then I would have to check if the id was already in the db.
Another idea I have is to do the first insert and then query for the row id of that insert in php and then make the second insert. I mean both should work but they don't seem like an optimal solution. I am not too familiar with the database transactional features but what would be the best approach to do in this case. I don't like the idea of inserting then querying for the id and then running the rest of the queries. Just seems very inefficient or perhaps I am wrong.
Just insert a row in the master table. Then you can fetch the insert id ( lastInserId when on PDO) and use that to populate your other queries.
You could use the php version as given by JvdBerg , or Mysql's LAST_INSERT_ID. I usually use the former option.
See a similar SO question here.
You could add a new column to the nomsing table, called 'insert_order' (or similar) with a default value of 0, then instead of generating one SQL statement per insert create a bulk insert statement e.g.
INSERT INTO nomsing(word,postress,gender, insert_order)
VALUES (''велосипед","8","mask",1), (''abcd'',"9","hat",2).....
you generate the insert_order number with a counter in your loop starting at one. Then you can perform one SELECT on the table to get the ids e.g.
SELECT row_id
FROM nomsing
WHERE insert_order > 0;
now you have all the IDs you can now do a bulk insert for your following queries. At the end of your script just do an update to reset the insert_order column back to 0
UPDATE nomsing SET insert_order = 0 WHERE insert_order > 0;
It may seem messy to add an extra column to do this but it will add a significant speed increase over performing one query at a time.
Is it possible to UPDATE and then INSERT where row exists in mysql? I have this query,
$q = $dbc -> prepare("UPDATE accounts SET lifeforce = maxLifeforce, inHospital = 0 WHERE hospitalTime <= NOW() AND inHospital = 1");
$q -> execute();
How can I either get the primary key into an associative array to then do an insert for each item in the array, or do an UPDATE AND INSERT?
Or does it involve doing a SELECT to get all that match criteria, then UPDATE then INSERT using array from the select? This seems rather a long way to do it?
Basically I need to INSERT onto another table using the same primary keys that get updated.
Or does it involve doing a SELECT to get all that match criteria, then UPDATE then INSERT using array from the select?
Yes, sorry, that's the main way.
Another approach is to add a column called (say) last_updated, that you set whenever you update the column. You can then use that column in a query that drives your insert. That would have other advantages — I find last_updated columns to be useful for many things — but it's overkill if this is the only thing you'd ever use it for.
Edited to add: Another option, which just occurred to me, is to add a trigger to your accounts table, that will perform the insert you need. That's qualitatively different — it causes the insertion to be a property of accounts, rather than a matter of application logic — but maybe that's what you want? Even the most extreme partisans of the "put-all-constraints-in-the-database-so-application-logic-never-introduces-inconsistency" camp are usually cautious about triggers — they're really not a good way to implement application logic, because it hides that logic somewhere that no-one will think to look for it. But if the table you're inserting into is some sort of account_history table that will keep track of all changes to account, then it might be the way to go.
You can use a multiple table update as written in the manual: http://dev.mysql.com/doc/refman/5.0/en/update.html
If the second table needs an insert, you probably would have to do it manually.
You can use the mysqli_last_id function:
http://php.net/manual/en/mysqli.insert-id.php
Also, when running consecutive queries like that, I'd recommend using transactions:
http://www.techrepublic.com/article/implement-mysql-based-transactions-with-a-new-set-of-php-extensions/6085922
I have hundred of thousands of elements to insert into a database. I realized calling an insert statement per element is way too costly and I need to reduce the overhead.
I recon each insert can have multiple data elements specified such as
INSERT INTO example (Parent, DataNameID) VALUES (1,1), (1,2)
My issue is that since the "DataName" keeps repeating itself for each element I thought it would optimize space if I stored these string names in another table and reference it.
However that causes problems for my idea of the bulk insert which now requires a way to actually evaluate the ID from the name before calling the bulk insert.
Any recommendations?
Should I simply de-normalize and insert the data every time as plain string to the table?
Also what is the limit of the size of the string as the string query amounts to almost 1.2 MB?
I am using PHP with MySQL backend
You haven't given us a lot of info on the database structure or size, but this may be a case where absolute normalization isn't worth the hassle.
However if you want to keep it normalized and the strings are already in your other table (let's call it datanames), you can do something like
INSERT INTO example (Parent, DataNameID) VALUES
(1, (select id from datanames where name='Foo')),
(1, (select id from datanames where name='Bar'))
First you should insert the name in the table.
Than call LAST_INSERT_ID() to get the id.
Than you can do your normal inserts.
If your table is MYisam based you can use INSERT DELAYED to improve performance: http://dev.mysql.com/doc/refman/5.5/en/insert-delayed.html
You might want to read up on load data (local) infile. It works great, I use it all the time.
EDIT: the answer only addresses the sluggishness of individual inserts. As #bemace points out, it says nothing about string IDs.
I was wondering how can I select a value from a database that a user just entered it into and then add it to another mysql table all in the same script before the script is finished running.
You're probably looking for an insert ... select statement.
If you're talking about adding a value that a user just entered into a form, to something, and then putting that into the database, you should do the addition while in PHP. There's no point in going to the database after you've just inserted the value for this purpose.
If I'm misunderstanding something, please elaborate your question and let us know WHY you would want to figure out a just-inserted database value and do an operation on it, rather than trying to do it before you insert in the first place.
Also, if it's a fairly simple modification consider using an UPDATE statement, not a select --> insert.
Like nash said, you perform a select.
But to get the data from the row that the user just entered, you'll need:
mysql_insert_id()
Which grabs the last ID inserted (this is assuming you have an increment id column)
So assuming just entered his first and last name in a form, you'd insert his first and last name in the database(which i assume you know how since the title of this question is "SELECT a value from MySQL database"), you can get what he just entered by:
$last_id = mysql_insert_id();
If there are no rows on that table yet, then this will return 1. $last_id is now 1 (one).
To select:
SELECT * FROM users WHERE userID = "$last_id"
this will grab what the user just inserted....however, this seems pointless as you can use the variables from the form he just filled
enter code here
In the PHP MySQL module, you normally perform a mysql_select_db() to switch database.
You can insert your data into tables in different databases by switching between them with that function.
However, you can insert data into any table of any database (which the user has access to) by prefixing the database name to the table like so:
INSERT INTO test_db.test_table (`column1`,`column2`) VALUES ('abc',123);
You can use that also to insert data from one table into another using:
INSERT INTO `db1`.`myTable` (`column1`,`column2`) SELECT `column1`,`column2` FROM `db2`.`myTable` WHERE `id`= 5
The WHERE id part should obviously match the id of a row in db2.myTable
If you use doctrine you have the inserted data in the object representing the table and in addition you have primary key assigned for the record inside the object.
Con is doctrine is huge database abstraction layer, so if your application is not big doctrine is hammer for mosquito.
what is the structure of your database? The names of your tables, columns?
Some tutorial that you may want to look at: (grabbed from google)
http://www.phpf1.com/tutorial/php-mysql-tutorial.html
In theory you perform a select, take the data you need and perform an insert.