I have a temporary table with 100k rows. It comes from a csv file and I use LOAD DATA LOCAL INFILE to put it in temporary table and I need to process those raw data because it has duplication. So I make 3 tables which are product, product_images and product_description to process the raw data and put it there.
My sample code:
Insert Raw Products to products table
$data = $this->getRawDistinct(); // raw data I used sql group by here and get 2000 data
$insertedID = [];
foreach (array_chunk($data, 1000) as $key => $values) {
foreach($values as $value) {
$this->db->insert("{$this->prefix}products", [
'name' => $value['product_title'],
'brand_name' => $value['brand_name'],
'brand_logo' => $value['brand_logo_image'],
'status' => $value['product_status'],
'keywords' => $value['keywords'],
]);
array_push($insertedData, $this->db->insert_id); // store inserted data id
}
}
Insert Raw Products to product_image and product_description
$processedProduct = $this->db->get_results(
"SELECT id, name " .
"FROM {$this->prefix}products " .
"WHERE id IN ( ".implode(",", $insertedData)." ) "); // Get the data of those inserted earlier
foreach ($processedProduct as $key => $value) {
$rawProducts = $this->db->get_results(
"SELECT * " .
"FROM {$this->prefix}raw_products " .
"WHERE product_title = '$value->name'");
if($rawProducts) {
foreach ($rawProducts as $rawProduct) {
//insert to product_image
$this->db->insert("{$this->prefix}product_images", [
'product_id' => $value->id,
'thumbnail' => $rawProduct->thumbnail_image,
'color_swatch' => $rawProduct->color_swatch_image,
'product_image' => $rawProduct->product_image,
'front_flat' => $rawProduct->front_flat,
'back_flat' => $rawProduct->back_flat,
'front_model' => $rawProduct->front_model,
'back_model' => $rawProduct->back_model,
'side_model' => $rawProduct->side_model,
'three_q_model' => $rawProduct->three_q_model,
'color_square_image' => $rawProduct->color_square_image,
'color_product_image' => $rawProduct->color_product_image,
]);
//insert to product_descriptions
$this->db->insert("{$this->prefix}product_descriptions", [
'product_id' => $value->id,
'size' => $rawProduct->size,
'piece_weight' => $rawProduct->piece_weight,
'piece_price' => $rawProduct->piece_price,
'dozen_price' => $rawProduct->dozen_price,
'case_price' => $rawProduct->case_price,
'piece_sale_price' => $rawProduct->piece_sale_price,
'dozen_sale_price' => $rawProduct->dozen_sale_price,
'case_sale_price' => $rawProduct->case_sale_price,
'inventory_key' => $rawProduct->inventory_key,
'size_index' => $rawProduct->size_index,
'catalog_color' => $rawProduct->catalog_color,
'price_code' => $rawProduct->price_code,
'catalog_color' => $rawProduct->catalog_color,
]);
}
}
}
As you can see at the code, I distinct the records at first and get 2k of data because those are the products that I need to put in the products tables.
Don't get consfused of 100k then it comes to 2k, the other records are duplicated because it creates another row for different sizes, color, images and etc which is pertaining to one product only. And those sizes, images I need it to insert in product_image and product_description which both should have 100k records.
Insert Raw Products to products table is working fine but the inserting the raw product to product_image and product_description it insert data but not all because I reach max time execution and I already increase that to 5mins but I don't like that because the user will just for 5mins and still not processing all the data. How can I optimize it and make it work? or even handle more larger data than I'm facing now. Thanks.
Let the database do all the work.
Assuming that products has an autoincrement id field,and that each product_title has only one brand name, brand logo, status, and keywords value:
INSERT INTO products
SELECT DISTINCT product_title name, brand_name, brand_logo_image brand_logo, product_status status, keywords
FROM raw_products
Now we should have a good products table with a product_id field (the autoincrement field). Now, we need to create the relational tables product_images and product_descriptions. (side note, do these have a 1:1 relationship with each other? For the sake of argument I'll assume they don't and you've established that they need to be separate)
INSERT INTO product_images
SELECT p.id product_id, r.thumbnail_image thumbnail, r.color_swatch_image color_swatch, r.product_image, r.front_flat, r.back_flat, r.front_model, r.back_model, r.side_model, r.three_q_model, r.color_square_image, r.color_produt_image
FROM raw_products r
LEFT JOIN products p ON r.product_title=p.name
The left join is the magic here. For each row of raw products, the db will attempt to join the left joined table on the matching conditions. If it finds a match, it joins it; if no match is found, it puts nulls for the table's fields. (In this case there will be no rows with null values for products.)
Using the second insert as an example, it should be trivial for you to craft the third insert.
Related
Tell me about this issue, I can’t find it in the documentation. Is it possible to insert into a table by column names. For example, there are columns name, price, quantity, and you only need to insert the name and quantity. Using a method like this
$values = [
["Samsung", "10"],
];
$ValueRange = new Google_Service_Sheets_ValueRange();
$ValueRange->setValues($values);
$options = ['valueInputOption' => 'USER_ENTERED'];
$service->spreadsheets_values->update($spreadsheetId, $list . '!A' . $total, $ValueRange, $options);
The number of 10 should be inserted in the B column where the price is located. quantity in column C
Is it possible to somehow insert data by column names? Maybe some examples in php?
When outside the boxing thinking. I came up with the following solution:
$column = 'product';
$enum = '1';
$product_access = $dbh->prepare("UPDATE products_access SET {$column} = :enum WHERE products_access.id = :id");
$product_accessvar = trim($user['id']);
$product_access->bindParam(':id', $product_accessvar, PDO::PARAM_INT);
$product_access->bindParam(':enum', $enum, PDO::PARAM_INT);
//$product_access->bindParam(':product_enum', $enum);
//foreach($_POST["checkbox2"] as $loc_id)
$product_access->execute();
Thanks for your help, maybe i can help someone with my solution. It works for me now!
Below my question
I have a question about my follow project. I build an sql table where the users can add custom columns with a enum value. But the user can also update the value of this column. I can't set any column name because they all custom made by the user, so there is no column name.
My SQL:
tabel `products_access`
--
CREATE TABLE IF NOT EXISTS `products_access` (
`id` int(30) NOT NULL,
`user_id` int(30) NOT NULL,
`product 2` enum('0','1') NOT NULL COMMENT 'Dit is een product beschrijving van product 2.dgfdg'
)
Array:
Array
(
[id] => 17
[name] => product 2
[number] => 2002
[description] => Dit is een product beschrijving van product 2.
[mount] => 34
[price] => 6778
[deleted] => 0
[user_id] => 17
[product 2] => 1
)
Script:
<label>
<input name="clickedproduct[]" type="checkbox" value="<?php echo $avlue['id']; ?>" <?php echo (($_SERVER['REQUEST_METHOD'] == 'POST') ? ((isset($_POST[$avlue['name']])) ? ' value="'.$avlue[$avlue['name']].'" checked' : ' value="'.$avlue[$avlue['name']].'"') : (($avlue[$avlue['name']] == '1') ? ' value="'.$avlue[$avlue['name']].'"checked' : ' value="'.$avlue[$avlue['name']].'"')); ?>>
<?php echo $avlue['name']; ?>
</label>
PDO:
$product_access = $dbh->prepare('UPDATE products_access() VALUES(:id, :loc)');
$product_access->bindValue(':id', $id);
$product_access->bindParam(':loc', $loc_id);
foreach($_POST["checkbox2"] as $loc_id) $product_access->execute();
Below the array after saving.
Array
(
[username] => joshua
[rank] => Array
(
[0] => 0
)
[koppel] => Array
(
[0] => 1
)
[clickedproduct] => Array
(
[0] => 17
)
)
Can some one explain the solution?
This sort of application ordinarily is implemented using a metadata table, also known as a key / value store.
Each row of this metadata table identifies
the object it describes, with a product_id or similar foreign key.
the name of the data item it holds, e.g. 'price' or 'mount'
the value of the data item it holds, e,g, '6778' or '34'
optionally a code identifying the data type of the item ('money'? 'text'?)
It's easy to add metadata to an object. Insert a row into the metadata table giving the "column name" you want and the value. It's a little trickier to retrieve it. You need a query, for example, like this.
SELECT p.product_id, a.val as price, b.val as mount
FROM product p
LEFT JOIN metadata a ON a.product_id = p.product_id AND a.key='price'
LEFT JOIN metadata b ON b.product_id = p.product_id AND b.key='mount'
WordPress's wp_postmeta table setup is a good and widely used example of this data design pattern. It has a workable API.
It's ordinarily considered bad practice to use data definition language commands (like ALTER TABLE CHANGE colname newcolname INT) in production. For one thing these commands are quite slow and thread-unsafe. For another, when a schema contains all kinds of user-defined columns it's hard to troubleshoot. You're better off using application code, like what I have suggested here, to allow your users to create their own data keys and values.
Create a column named 'custom_column' in 'products_access'.(Add user
defined column name here)
Create another table named 'tbl_custom_column'
Add fields 'user_id', 'custom_column_value' (Add your values here)
Hope that will help.
I've using this to create a new user in the WordPress database...
// Add user to WP users table.
$user_table_name = $wpdb->prefix . "users";
$unique_string = substr(md5(rand(0, 1000000)), 0, 10);
$wpdb->insert( $user_table_name, array(
'user_login' => sanitize_text_field($_POST['email']),
'user_pass' => sanitize_text_field(MD5($unique_string)),
'user_email' => sanitize_text_field($_POST['email']),
'user_registered' => sanitize_text_field(date("Y-n-d G:i:s")),
'user_status' => $_POST['1'],
'display_name' => sanitize_text_field($_POST['first_name']) . " " . sanitize_text_field($_POST['last_name'])
) );
...which works fine, and let's pretend that the ID of that user turned out to be 1234 in the database table (thanks to auto increment).
So now I also need to add the corresponding user meta information into the usermeta table for that user, and this is where I get a little confused.
The code above is easy because it's just adding a row to a table. But the usermeta table is different because it will need - in this case - a bunch of rows with the corresponding user_ID of 1234 each respectively with:
nickname (I'll use the email address for this)
wp_capabilities (value to be a:1:{s:10:"subscriber";b:1;})
sales (a custom field I have - value will be set to the word "yes")
colour (another custom field I have - value will be set to the word "green")
I'm guessing the SQL statement is going to be similar to the one at the start of this post.
If anyone could show me, that would be awesome.
UPDATE:
So this is mostly done. This works:
// Add corresponding user metadata to WP users table.
$usermeta_table_name = $wpdb->prefix . "usermeta";
$last_id = $wpdb->insert_id;
$role = sanitize_text_field('a:1:{s:10:"subscriber";b:1;}');
$wpdb->query(
$wpdb->prepare(
"
INSERT INTO $usermeta_table_name (
`umeta_id`,
`user_id`,
`meta_key`,
`meta_value` )
VALUES (
NULL,
$last_id,
$usermeta_table_name . 'capabilities',
'$role' )
",
$last_id, $last_id
)
);
That will add one row to the usermeta table, but how can I add 2 more rows within the same statement?
Store the last inserted row's ID to a variable:
$last_id = $wpdb->insert_id;
now, use this $last_id variable for metadata insertion. The insert_id variable is provided by the wpdb class.
OK so I feel a little silly about this - though to be fair it's been a long time since I wrote any SQL :-)
The answer is simply to follow this method:
INSERT INTO tbl_name (a,b,c) VALUES(1,2,3),(4,5,6),(7,8,9);
I have an INSERT loop and I need to add a reference number to it. I need all of the reference numbers in the loop to be the same. I know that with MAX() I can select the highest number in the table. But if I loop it will just increase with each loop while I need it to stay the same.
Is there a way of doing this in the query itself? Or is the only way to save it in a variable outside of the loop?
Example code:
for($i=2;$i<=$row_count;$i++){ // Loops 3 times (example)
$part = $vehicle.'_part'.$i;
$description = $vehicle.'_description'.$i;
$imageName = $vehicle.'_image'.$i;
$parts[] = array(
'part' => $_SESSION[$part],
'image' => $_SESSION[$part],
'description' => $_SESSION[$description]);
}
foreach($parts as $onePart){
$queries[] = "INSERT INTO searches_tbl (ref_nr, vozila_id, korisnici_id, part, description, image)
VALUES (???, (SELECT id FROM vozila_tbl ORDER BY id DESC LIMIT 1),
(SELECT id FROM korisnici_tbl WHERE email = '".$email_address."' ORDER BY id DESC LIMIT 1), '".$onePart['part']."', '".$onePart['description']."', '".$onePart['image']."')";
}
You can create a new table, ie searches_ref with just an AUTO_INCREMENT Primary Key-column.
CREATE TABLE searches_ref ( id INT NOT NULL AUTO_INCREMENT PRIMARY
KEY );
You would then insert a new line in searches_ref before the loop, and get a new id you can use in the loop.
This way, you should not have concurrency issues (which you will probably get using MAX())
I'm writing a plugin and trying to insert a new row into the wp_term_relationships table inside of a foreach loop. I know the variables have values because of a var_dump, but for some reason, I'm getting an error consistently. This shows up about 600 times on the show_errors() function:
WordPress database error: [Duplicate entry '0-0' for key 1] INSERT
INTO wp_term_relationships
(object_id,term_taxonomy_id,term_order) VALUES ('','','')
My Code:
foreach ($cb_t2c_cat_check as $values) {
global $wpdb;
$prefix = $wpdb->prefix;
$table = $prefix . 'term_relationships';
$object_id = $values->object_id;
$taxo_id = $values->term_taxonomy_id;
$num_object_id = (int)$object_id;
$num_taxo_id = (int)$taxo_id;
//var_dump($num_object_id); //This produces values, so why are they not getting inserted into the table?
//var_dump($num_taxo_id); //This produces values, so why are they not getting inserted into the table?
$wpdb->insert(
$table,
array(
'object_id' => $num_object_id,
'term_taxonomy_id' => $num_taxo_id,
'term_order' => 0
), ''
);
//$wpdb->show_errors();
//$wpdb->print_error();
}
As for why it does not work: do not set third parameter of $wpdb->insert to empty string. It formats every field accordingly..
What it does now is equivalent to:
$wpdb->insert($table, array(
'object_id' => sprintf('', $num_object_id),
'term_taxonomy_id' => sprintf('', $num_taxo_id),
'term_order' => sprintf('', 0)
));
If you really want to set third parameter you should do:
$wpdb->insert($table, array(
'object_id' => $num_object_id,
'term_taxonomy_id' => $num_taxo_id,
'term_order' => 0
), array('%d', '%d', '%d'));
As for error: wp_term_relationships table has a unique primary key on (object_id, term_taxonomy_id). This means that you cannot have two rows in that table which have both same object_id and term_taxonomy_id.
Though this has happened because by setting third parameter of insert to empty string, you are trying to insert rows with object_id=0 and term_taxonomy_id=0 over and over again.
The answer above was correct in that the database needs to have unique keys and cannot insert a row where the key-value pair already exists, and the format of each new value needs to be set. In addition, specific to Wordpress, there was a problem I wasn't addressing, specifically dealing with the term_taxonomy table and updating the count.
First it's important to note that the plugin was designed to update certain categories for posts in the term_relationships table. This was actually accomplished using the $wpdb-> insert method. However, my test for determining whether the plugin actually inserted new rows in the term_relationships table was not to look at the table directly, but to go to the Wordpress dashboard, select categories, and see if the number of posts with that category was more than before. This didn't work, because the plugin never updated the count in the term_taxonomy table. I only discovered this by clicking 'view' next to a category in the Wordpress dashboard and seeing that there were multiple posts with that category, even though the official Wordpress "count" said there were none.
I confirmed that the term_taxonomy table, the 'count' column, needed to be updated as well by going straight to the database and putting WHERE = 'term_taxonomy_id' in the statement. Sure enough, there were over 1700 results, even though Wordpress thought there were none.
Lesson: Confirm the $wpdb->insert method is working by using PHPMyAdmin, not necessarily relying on the Wordpress dashboard.
With a few modifications, the code now works great. Here's an example:
foreach ($cb_t2c_objects as $values) {
global $wpdb;
$prefix = $wpdb->prefix;
$table = $prefix . 'term_relationships';
$object_id = $values->object_id;
$taxo_id = $values->cat_taxo;
$num_object_id = (int)$object_id;
$num_taxo_id = (int)$taxo_id;
//Need to check to see if row exists for each, if not, then insert.
$cb_t2c_get_row = $wpdb->get_row("
SELECT *
FROM ".$prefix."term_relationships
WHERE object_id = ".$num_object_id." AND term_taxonomy_id = ".$num_taxo_id."
GROUP BY object_id, term_taxonomy_id
", OBJECT);
//var_dump($cb_t2c_get_row);
if ( is_null($cb_t2c_get_row) ) {
//Insert the new values.
$wpdb->insert(
$table,
array(
'object_id' => $num_object_id,
'term_taxonomy_id' => $num_taxo_id,
'term_order' => 0
),
array(
'%d',
'%d',
'%d'
)
);
}
//Set the variables for the count update.
$cb_t2c_term_taxonomy_table = $prefix . 'term_taxonomy';
$cb_t2c_update_data = $wpdb->get_var("
SELECT count(term_taxonomy_id) as 'new_count'
FROM ".$prefix."term_relationships
WHERE term_taxonomy_id = ".$num_taxo_id."
",0,0); //returning NULL
//var_dump($cb_t2c_update_data);
//Update the count in the term_taxonomy table.
$wpdb->query("
UPDATE ".$prefix."term_taxonomy
SET count = ".$cb_t2c_update_data."
WHERE term_taxonomy_id = ".$num_taxo_id."
");