I need to add bulk data to my table 'wp_usermeta'. The data is the same but with just the user_id changing for every row. A PHP example would be:
for($i = 1; $i < 10000; $i++){
mysql_query(
"INSERT INTO wp_usermeta ('user_id', 'meta_key', 'meta_value')
VALUES ($i, "key", "value")");
}
So I'm adding one row for every user up to 10,000 users which is exactly the same except for the user_id which is taken from the iteration variable.
Is there any way to do this more efficiently in mysql?
This will do it in one SQL statement:
$sql="
INSERT INTO wp_usermeta ('user_id', 'meta_key', 'meta_value')
VALUES
";
for($i = 1; $i < 10000; $i++){
$sql.="($i, 'key', 'value'),";
}
$sql=rtrim($sql, ",");
Please don't use the deprecated mysql_ functions, use mysqli_ or PDO instead.
There's no way to avoid an INSERT statement for each row, however, if you do them all at once (in a single query), it will be faster than doing them separately:
INSERT INTO wp_usermeta (user_id, meta_key, meta_value)
VALUES (1, 'key', 'value'),
VALUES (2, 'key', 'value'),
...
VALUES (10000, 'key', 'value');
MySQL updates indexes after each query completes. Here, it's one query, so the indexes are only updated after all of the rows are inserted.
Even doing the inserts separately, but with a prepared statement and bound parameters, would be faster than your current method.
You could do it with this statement:
INSERT INTO wp_usermeta (user_id, meta_key, meta_value)
SELECT
nums.n, 'key', 'value'
FROM (
SELECT
a.N + b.N * 10 + c.N * 100 + +d.N * 1000 + 1 AS n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) c
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) d
ORDER BY n
) nums;
Explanation
The inner select builds the numbers from 1 to 10.000
Demo
Try this code
<?php
$query = "INSERT INTO wp_usermeta ('user_id', 'meta_key', 'meta_value') VALUES ";
$user_details = array();
for($i = 1; $i < 10000; $i++){
$user_details[] = "(".$i.",'key','value')";
}
$query .= implode(",", $user_details);
mysql_query($query);
?>
Related
Hello i have mysql table which named numbers like this;
ID
Number
1
3002
2
3004
2000
7545
When i need to insert a new data products table i have to find first number which is not on this table between(3000 to 35000). I mean i need to find first number between this numbers not on numbers table. How can i find it?
$statement = $con->prepare("SELECT * FROM numbers");
$statement->execute();
$result = $statement->fetchAll(PDO::FETCH_ASSOC);
foreach($result as $row)
{
$arr1[] = $row['numbers'];
}
$arr2 = range(3000,35000);
$missing_numbers = array_diff($arr2,$arr1);
print_r($missing_numbers);
i tried array_diff but i gives results with different keys. When i write $missing_numbers[0] i want to see first missing number.
If you just want the first missing number, it may be easier to read the rows in number order and check for the first one which isn't in the sequence you are after. So a query (with the start and end points as parameters) order by the number and a counter which is what it's expecting to get on each row...
$startPoint = 3000;
$endPoint = 35000;
$statement = $db->prepare("SELECT number
FROM numbers
WHERE number >= :start
and number <= :end
order by number");
$statement->execute([
'start' => $startPoint,
'end' => $endPoint
]);
$expected = $startPoint;
while ($row = $statement->fetch()) {
if ( $row['number'] != $expected ) {
echo "Missing=" . $expected;
break;
}
$expected++;
}
I think your solution will work only thing missing is reordering of the indexes which can be done by using the array_values() function. Update your last lines of code as -
$missing_numbers = array_values(array_diff($arr2,$arr1));
print_r($missing_numbers);
so on printing $missing_numbers[0] you should get the first missing number.
Easy, just need reset keys after array_diff:
...
$missing_numbers = array_values(missing_numbers);
print_r($missing_numbers);
IF you want do it only by using MySQL query and later get the result in PHP you need to do something like this :
Create temporray table with your example :
create table test123(id integer, Number varchar(100))
#insert into test123 (ID, Number) values (1, '3002');
#insert into test123 (ID, Number) values (2, '3004');
#insert into test123 (ID, Number) values (2000, '7545');
CREATE TEMPORARY TABLE WITH RANGE 3000 - 35000
CREATE TEMPORARY TABLE IF NOT EXISTS tableTest AS (
SELECT #row := #row + 1 AS row FROM
(select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) t,
(select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) t2,
(select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) t3,
(select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) t4,
(select 0 union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) t5,
(SELECT #row:=3000) numbers WHERE #row < 35000
)
AND FINALLY you can just select numbers that's not inside your table with data
select * from tableTest t where t.row not in ( select Number from test123)
The output will be select all numbers as you wanted.
You must fill out the form table 1 to 1,000,000:
Table view
CREATE TABLE `test` ( `id` INT UNSIGNED NOT NULL AUTO_INCREMENT , `value` INT UNSIGNED NOT NULL , PRIMARY KEY (`id`)) ENGINE = InnoDB;"
I wrote a function to do this, but adding data to the table is too slow, how can I improve the performance of insert data?
function InsertData(){
global $MySQL;
for($i = 1; $i != 1000000; $i++){
$MySQL->query("INSERT INTO `name` (`id`, `value`) VALUES ($i, $i);");
}
$MySQL->close();
}
You could use Transactions in order to only commit once every thousands inserts (or, if you are brave, after the million of queries). Here is the (brave) example:
function InsertData(){
global $MySQL;
// Start transactions
$MySQL->query('SET autocommit=0;');
$MySQL->query('START TRANSACTION;');
for($i = 1; $i != 1000000; $i++){
$MySQL->query("INSERT INTO `name` (`id`, `value`) VALUES ($i, $i);");
}
// So far, nothing as actually been saved to database
// Commit all inserts.
$MySQL->query('COMMIT;');
$MySQL->query('SET autocommit=1;');
$MySQL->close();
}
If this is too much for one single transactions due to some MySQL limit, you could perform the Commit every 10.000 inserts or so:
function InsertData(){
global $MySQL;
// Start transactions
$MySQL->query('SET autocommit=0;');
$MySQL->query('START TRANSACTION;');
for($i = 1; $i != 1000000; $i++){
$MySQL->query("INSERT INTO `name` (`id`, `value`) VALUES ($i, $i);");
if($i % 10000 == 0) {
$MySQL->query('COMMIT;');
$MySQL->query('START TRANSACTION;');
}
}
// So far, nothing as actually been saved to database
// Commit all inserts.
$MySQL->query('COMMIT;');
$MySQL->query('SET autocommit=1;');
$MySQL->close();
}
Pay attention to the eventual limit -> https://stackoverflow.com/a/2298325/2814721
And, of course, this is intended to be an experiment or one-shot script. Not advised to do in a production database.
How about moving the logic to the database, using a recursive CTE?
insert into name (id, value)
with recursive cte as (
select 1 id
union all select id + 1 from cte where i < 1000000
)
select id, id from cte
That may be to many rows to generate at once with recursion. An alternative is to generate just 10 rows, then multiply the rows:
insert into name (id, value)
with recursive cte as (
select 0 id
union all select id + 1 from cte where i < 9
)
select id, id
from (
select 1 + c0.id + c1.id * 10 + c2.id * 100 + c3.id * 1000 + c4.id * 10000 + c5.id * 100000 id
cte c0
cross join cte c1
cross join cte c2
cross join cte c3
cross join cte c4
cross join cte c5
) t
Try this query:
INSERT INTO `test` (`id`, `value`)
SELECT #row := #row + 1 AS row, #row
FROM (SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t1,
(SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t2,
(SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t3,
(SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t4,
(SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t5,
(SELECT 0 UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) AS t6,
(SELECT #row:=0) AS nums;
It's a "INSERT INTO... SELECT..." type of statement in which the SELECT statement itself is generating a million rows filled with pairs (1,1), (2,2), etc. Here is how it works:
The tables t1, t2, t3, t4, t5, t6 are 10 rows each. Cross joining them generates 10^6 = 1000000 combinations, so the resulting table will be with million rows;
For each of those rows, we SELECT the #row variable twice. And not only that but we also increment it with 1;
The nums table is used only to initialize the variable to 0 at the very beginning;
The resulting table is passed to the INSERT statement and the data is stored in the table.
A cleaner looking solution is to use recursive CTE with newer MySQL/MariaDB. It's the one that was submitted by user GMB:
INSERT INTO test (id, value)
WITH RECURSIVE temp AS (
SELECT 1 AS row
UNION SELECT row + 1
FROM temp
WHERE row < 1000000
)
SELECT row, row
FROM temp;
Based on my tests, it was a bit slower. I didn't monitor memory usage.
Simple mysql table:
CREATE TABLE `users` (
`id` INT NOT NULL AUTO_INCREMENT ,
`name` TEXT NOT NULL ,
`surname` text NOT NULL ,
`city` text NOT NULL ,
PRIMARY KEY ( `id` )
) ENGINE = MYISAM
And inserted like:
INSERT INTO `users` (`id`, `name`, `surname`, `city`) VALUES (1,`rock`,`fixed`,`london`);
How can i enter same table detail of one row 100 times with different primary id?
with php you can do it this way:
for($i=0; $i< 119; $i++)
{
//execute your query here, do not include your id column
INSERT INTO `users` (`name`, `surname`, `city`)
VALUES (`rock`,`fixed`,`london`);
}
OR
You can also do it with a mysql store procedure like:
//Loop stored procedure
DELIMITER $$
DROP PROCEDURE IF EXISTS test$$
CREATE PROCEDURE test()
BEGIN
DECLARE count INT DEFAULT 0;
WHILE count < 119 DO
INSERT INTO `users` (`name`, `surname`, `city`)
VALUES (`rock`,`fixed`,`london`);
SET count = count + 1;
END WHILE;
END$$
DELIMITER ;
execute the stored procedure
call test();
EDIT BASED ON COMMENTS:
Your code is not working because you are executing your query "outside" the loop, you code should look like this:
<?php
*/here mysql connection
for($x=0; $x<100; $x++)
{
$sql =INSERT INTO users (name, surname, city)
VALUES (rock,fixed,london);
$status = $conn->query($sql);
}
if ($status)
{ echo "New record created successfully"; }
else { echo "Error: " . $sql . "<br>" . $conn->error; }
$conn->close();
?>
try the following code
for ($i=0; $i<= 119; $i++){
//exclude id and do auto increament and primary key
$qry = "INSERT INTO `users` (`name`, `surname`, `city`) VALUES (`rock`,`fixed`,`london`)";
}
Doing it as a single SQL statement with no loop:-
INSERT INTO `users` (`id`, `name`, `surname`, `city`) VALUES (1,`rock`,`fixed`,`london`);
SELECT NULL, `rock`, `fixed`, `london`
FROM
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) units
CROSS JOIN
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9 UNION SELECT 10 UNION SELECT 11
) tens
The 2 sub queries just return ranges of numbers. With 10 in one and 12 in the other when cross joined they will return in total 120 rows. Those 120 rows are just your fixed values, but with NULL for the id which the auto increment will sort out into a unique value.
Note that you can do larger ranges like this. For example if you wanted to insert 653 rows (just a fairly random choice of number):-
SELECT NULL, `rock`, `fixed`, `london`
FROM
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) units
CROSS JOIN
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) tens
CROSS JOIN
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) hundreds
WHERE (units.aCnt + 10 * tens.aCnt + 100 * hundreds.aCnt) < 653
Or specifying the id field as the calculated value:-
SELECT 1 + units.aCnt + 10 * tens.aCnt + 100 * hundreds.aCnt, `rock`, `fixed`, `london`
FROM
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) units
CROSS JOIN
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) tens
CROSS JOIN
(
SELECT 0 AS aCnt UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9
) hundreds
WHERE (units.aCnt + 10 * tens.aCnt + 100 * hundreds.aCnt) < 653
Since the id has the property AUTO_INCREMENT this should generate a unique ID every time you enter data.
So your insert string can be like this:
INSERT INTO `users` (`name`, `surname`, `city`) VALUES (`rock`, `fixed`, `london`);
The id field will then auto-increment each time, giving you unique IDs
Just Use this query:
INSERT INTO users ( name, surname, city) VALUES ('rock','fixed','london');
I have the following details in MySQL as
user_id follow_user_id
1 2,3,3,3,3
5 1,2,3,3,3,3
6 1,2,3,3,3,3
i write the following code to get the unique code as follow:
SELECT LENGTH( follow_user_id ) - LENGTH( REPLACE( follow_user_id, ',', '' ) ) +1 AS no_of_follow FROM follow WHERE user_id =1;
But it provide the result:6
I need exactly unique rows: i.e:4
Apart from DB design questions you could use in PHP after fetching the row to $result:
count(array_unique(explode(",",$result["follow_user_id")));
$query="SELECT follow_user_id FROM follow WHERE user_id ='".$_POST['user_id']."' "; $query_run=mysql_query($query); $row= mysql_fetch_assoc($query_run);
$count= count(array_unique(explode(",",$row['follow_user_id'])));
$count;
This is better and faster ;)
count(array_flip(explode(",", $result["follow_user_id")));
Or doing it in SQL:-
SELECT COUNT(DISTINCT SUBSTRING_INDEX(SUBSTRING_INDEX(follow_user_id, ',', units.i + tens.i * 10), ',', -1) AS col1)
FROM sometable
CROSS JOIN (SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)units
CROSS JOIN (SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)tens
WHERE user_id = 1
(copes with up to 100 comma separated values).
But this would be so much easier with a properly normalised database design
I am trying to create php script that generates 11,000,000 million unique ids in sequential order. However, I am trying to do it very quickly within 20 min it should generate these 11 million unique ids. Also, once it reaches 12,000,000 it should wrap around and start back at zero.
Here is what I have so far. This script would would only return one id at a time. I just added a loop to see how long it would take to generate the ids.
while(true){
try {
$this->getAdapter()->query('INSERT INTO generate_ids (assigned_id) SELECT (MAX(assigned_id)+1) FROM generate_ids');
$id = $this->getAdapter()->lastInsertId();
$sql = 'SELECT assigned_id FROM generate_ids WHERE id = $id';
$query = $this->getAdapter()->query($sql);
$result = $query->fetchAll();
//Live system would return id here
$assigned_id = $result[0]['assigned_id'];
} catch (Exception $e) {
//do nothing
}
if($count == 11000000){
die();
}
$count++;
}
}
If you create the following table:
CREATE TABLE sequence (
sequence_id BIGINT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`sequence_id`)
)
Then issue these three queries one after the other:
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
SELECT LAST_INSERT_ID() AS sequence;
The third query is guaranteed to return a unique sequence number. This guarantee holds even if you have dozens of different client programs connected to your database. That's the beauty of AUTO_INCREMENT.
Instead of just generating eleven million of these sequence numbers up front, you can use these SQL queries to get a unique sequence number whenever you need it.
If you must wrap around at sequence number 12 million you can use these queries instead.
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
SELECT LAST_INSERT_ID() MOD 12000000 AS sequence;
The trick here is to use an auto-increment sequence number for uniqueness, but to also delete the rows in the table so it doesn't gobble up lots of space.
Note that you can also use the sequence number of LAST_INSERT_ID() for other purposes, like so for example.
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
INSERT INTO user (userid, username, phone)
VALUES (LAST_INSERT_ID() MOD 12000000, 'Joe', '800-555-1212');
SELECT LAST_INSERT_ID() MOD 12000000 AS sequence;
If you need to insert all your ids at once (for some reason) the fastest approach would be to do it in pure SQL
insert into generate_ids (assigned_id)
select N
from
(
select a.N + b.N * 10 + c.N * 100 + d.N * 1000 + e.N * 10000 + f.N * 100000 + g.N * 1000000 + h.N * 10000000 + 1 as N
from (select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) a
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) b
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) c
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) d
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) e
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) f
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) g
,(select 0 as N union all select 1) h
order by N
) q
where N <= 11000000
It takes less than a minute to complete on my laptop.