I am using ThingEngineer/PHP-MySQLi-Database-Class and I am trying to perform a multiple insert while using OnDuplicate. The goal is to insert a new product record if the 'sku' does not already exist. If the 'sku' does exist then the 'name' should be updated instead of creating a new entry.
MySQL Schema:
CREATE TABLE `products` (
`product_pk` bigint(9) NOT NULL,
`product_id` int(20) UNSIGNED NOT NULL,
`name` varchar(255) NOT NULL,
`sku` varchar(16) NOT NULL,
`category` int(10) DEFAULT NULL,
`last_update` timestamp NOT NULL ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;
ALTER TABLE `products`
ADD PRIMARY KEY (`product_pk`),
ADD UNIQUE KEY `sku` (`sku`);
ALTER TABLE `products`
MODIFY `product_pk` bigint(9) NOT NULL AUTO_INCREMENT;
PHP:
$sDate = date("Y-m-d H:i:s");
$lastid = $db->rawQuery('SELECT MAX( product_id ) AS max FROM products');
(!$lastid || !isset($lastid[0]['max'])) ? $pid = 0 : $pid = $lastid[0]['max']++;
foreach ($data as $item){
if (isset($item['sku']) && !null == $item['sku']){
$prod[$pid]['product_id'] = $pid;
$prod[$pid]['sku'] = $item['sku'];
$prod[$pid]['name'] = substr($item['product-name'],0,255);
$prod[$pid]['last_update'] = $sDate;
$pid++;
}
}
$utfEncodedArray =encodeArray($prod, 'UTF-8');
$db->onDuplicate('name', 'product_pk');
$db->insertMulti('products', $utfEncodedArray);
function encodeArray($array, $type)
{
foreach($array as $key => $value)
{
if (is_array($value)){ $array[$key] = encodeArray($value, $type);}else{ $array[$key] = mb_convert_encoding($value, $type);}
}
return $array;
}
The error I receive is:
Uncaught mysqli_sql_exception: Duplicate entry 'ABC123' for key 'sku'
Here is a sample of the array $utfEncodedArray used on the insertMulti call:
Array(
[1] => Array
(
[product_id] => 1
[sku] => ABC123
[name] => product1
[last_update] => 2018-09-08 18:55:20
)
[2] => Array
(
[product_id] => 2
[sku] => ABC124
[name] => product2
[last_update] => 2018-09-08 18:55:20
)
)
Steps I have tried so far:
Dropped the 'products' table and created it again. Multiple times.
Tried using 'sku' instead of 'product_pk' in the onDuplicate call.
Tried multiple collation types
Tried using unique key on both 'sku' and 'product_id'
When I attempted this method all entries were inserted correctly but when running it again it generated duplicates instead of updating the existing row. Not sure how this happened seeing as both 'sku' and 'product_id' are unique.
The $prod array currently contains the same values. So every time I run this I would expect to see the 'last_updated' column to be updated every time after the initial inserts.
This is my first experience using onDuplicate and despite hours of searching and reading docs I am still lost. I was trying to let the db class handle the multiple insert from the array but I am not against trying raw queries while iterating over my array of products instead.
Of course as soon as I posted this I find the issue...
Found a fork of the database class which resolved issues with insertMulti while using onDuplicate:
should fix insertMulti() if onDuplicate() is set
I have MySQL Query i want to write this query zend 2.
select billsec, call_status, Count(*)
from (
select
billsec,
if(ANSWERED_NUM is Null, 'Missed', 'Answered') call_status
from cust_info
where billsec in (
select id
from `users`
where `account_id` = 452 and `added_by` = 20694 and `status` = 'active'
)
)a
group by a.billsec,a.call_status
in zend 2 am trying to write like this but
$adapter = $this->getAdapter();
$resultset = $this->select( function( Select $select ) use ( $request, $adapter ) {
$sub1 = new Select( 'users' );
$sub1->columns( array( 'id' ) );
$sub1->where( array( 'account_id' => '452','added_by' => '20694','status' => 'active' ) );
$sub2 = new Select( 'cust_info' );
$sub2->columns(array("id","billsec", "if(ANSWERED_NUM is Null, 'Missed','Answered')"=>"call_status"));
$sub2->where('billsec IN(?)', new \Zend\Db\Sql\Expression( '?', array( $sub1 ) ));
var_dump( $sub2->getSqlString( $adapter->getPlatform() ) );die();
});
When am print this query output like this:
"SELECT `cust_info`.`id` AS `id`, `cust_info`.`billsec` AS `billsec`, `cust_info`.`call_status` AS `if(ANSWERED_NUM is Null, 'Missed','Answered')` FROM `cust_info` WHERE billsec IN('')"
Here am not able to write IN condition Query, thank in advance.
Let's say I have an array as follows (ID => Type):
$contentIndexes = [
32 => 'news',
40 => 'blog',
68 => 'blog',
109 => 'document',
124 => 'news'
]
And the following database table:
CREATE TABLE `ContentIndex` (
`ID` INT(11) NOT NULL AUTO_INCREMENT,
`ItemID` INT(11) NOT NULL,
`ItemType` VARCHAR(50) NOT NULL COLLATE 'utf8_unicode_ci',
//...
);
How would I retrieve each ContentIndex based on the combination of the 'ItemID' and 'ItemType' columns (preferably with just one query).
Using WHERE IN is not an option since it wouldn't take the combination in consideration:
ContentIndexQuery::create()
->filterByItemID(array_keys($contentIndexes))
->filterByItemType($contentIndexes)
->find();
Any ideas?
I don't know the Propel syntax, but the basic SQL syntax would be with OR.
WHERE ((ItemID = 32 AND ItemType = 'news')
OR
(ItemID = 40 AND ItemType = 'blog')
OR
(ItemID = 68 AND ItemType = 'blog')
OR
(ItemID = 109 AND ItemType = 'document')
OR
(ItemID = 124 AND ItemType = 'news')
)
For any Propel users encountering this problem in the future, this is what I came up with to create a query as Barmar stated:
$items = ContentIndexQuery::create();
$i = 0;
foreach ($contentIndexes as $id = > $type) {
if ($i > 0) $items->_or();
$items->condition('cond1'.$i, ContentIndexTableMap::COL_ITEM_ID . ' = ?', $id)
->condition('cond2'.$i, ContentIndexTableMap::COL_ITEM_TYPE . ' = ?', $type)
->where(['cond1'.$i, 'cond2'.$i], 'AND');
$i += 1;
}
$items = $items->find()->getData();
I have a project that i have inherited and was poorly designed and i am just trying to bring some sanity into it. Here is the issue i am having. I have a table that stores projects, contacts and another table that stores writers attached to that project(although there are still more tables). I have managed to write the below script.
$conditions = $this->paginate = array('joins' => array(
'table' => 'wp3_order_writers',
'alias' => 'OrderWriter',
'type' => 'LEFT',
'conditions' => array(
'OrderWriter.order_id=Order.id'
)
),
'conditions' => array(
array('Order.status_id' => array(5,6,7,8),
'OR' => array('Order.assigned_to_writer' => $this->_user['Contact']['id'],
'OrderWriter.writer_id' => $this->_user['Contact']['id'],
)
),
),
'fields' => array(
'Order.id', 'Order.urgent', 'Order.writer_deadline', 'Order.subject_title', 'Client.name', 'Client.email', 'Service.*', 'Order.writer_fee', 'Order.count_of_words', 'Status.*', 'Order.pay_status', 'Subject.*'
)
);
$this->paginate['limit'] = 100000;
$orders = $this->paginate($this->Order);
When i check the sql script generated it contains error. Below is what is generated
SELECT `Order`.`id`, `Order`.`urgent`, `Order`.`writer_deadline`, `Order`.`subject_title`, `Client`.`name`, `Client`.`email`, `Service`.*,`Order`.`writer_fee`, `Order`.`count_of_words`, `Status`.*, `Order`.`pay_status`, `Subject`.* FROM `wp3_orders` AS `Order` wp3_order_writers OrderWriter LEFT Array LEFT JOIN `wp3_contacts` AS `Client` ON (`Order`.`client_id` = `Client`.`id`) LEFT JOIN `wp3_keys_values` AS `Service` ON (`Order`.`service_id` = `Service`.`id` AND `Service`.`key` = 'services') LEFT JOIN `wp3_keys_values` AS `PayMethod` ON (`Order`.`pay_method_id` = `PayMethod`.`id` AND `PayMethod`.`key` = 'pay_methods') LEFT JOIN `wp3_subjects` AS `Subject` ON (`Order`.`subject_id` = `Subject`.`id`) LEFT JOIN `wp3_keys_values` AS `Status` ON (`Order`.`status_id` = `Status`.`id` AND `Status`.`key` = 'statuses') LEFT JOIN `wp3_clients_sessions` AS `ClientsSession` ON (`ClientsSession`.`current_order_id` = `Order`.`id`) WHERE ((`Order`.`status_id` IN (5, 6, 7, 8)) AND (((`Order`.`assigned_to_writer` = 1087) OR (`OrderWriter`.`writer_id` = '1087')))) AND `Order`.`completed` = 1 LIMIT 100000
With this error "Warning (512): SQL Error: 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'wp3_order_writers OrderWriter LEFT Array LEFT JOIN wp3_contacts AS Client ON' at line 1 [CORE\cake\libs\model\datasources\dbo_source.php, line 684]"
Your help will be greatly appreciated
I would make it with Containable (assuming that you have models like Order, OrderWriter, Writer and defined relations inside)
Then your query will look like:
$this->paginate['Order'] = array('contain' => array('Writer'));
$this->paginate['conditions'] = array(...);
$this->paginate['fields'] = array(...);
$orders = $this->paginate($this->Order);
I am not sure if the Order section should be like above or:
$this->paginate['Order'] = array('contain' => array('OrderWriter'=>array('Writer')));
Also I would advise you don't use 100000 limit, since it's probably a way to skip the pagination, but then just use find() instead of paginate.
Although avoiding pagination will slow down the request, since it will make a query for each row of Orders, so having default limit (20) or some reasonable number should speed-up the site/app.
$res (array)-> (count 50 (!) )
Example:
(
[1] => Array
(
[artistname] => Lady GaGa
[songname] => Love Games
[duration] => 3:31
[url] => 7e91a5ca16ae
[server] => 3
)
[2] => Array
(
[artistname] => DJ Layla
[songname] => Single Lady
[duration] => 3:20
[url] => f0906a3087eb
[server] => 3
)
[3] => Array
(
[artistname] => Lady Gaga
[songname] => Bad Romance (Bimbo Jones Clean Radio Remix)
[duration] => 3:59
[url] => 36e77d5a80357
[server] => 3
)
}
PHP code:
$massquery = '';
foreach($res as $value)
{
if(!get_magic_quotes_gpc())
{
$value['artistname'] = mysql_escape_string($value['artistname']);
$value['songname'] = mysql_escape_string($value['songname']);
$value['duration'] = mysql_escape_string($value['duration']);
$value['url'] = mysql_escape_string($value['url']);
$value['server'] = mysql_escape_string($value['server']);
}
$value['artistname'] = trim($value['artistname']);
$value['songname'] = trim($value['songname']);
$value['duration'] = trim($value['duration']);
$value['url'] = trim($value['url']);
$value['server'] = trim($value['server']);
$sh = mysql_query("SELECT `artistname`,`songname`,`server` FROM `music` WHERE `artistname`='".$value['artistname']."' AMD `songname`='".$value['songname']."' AND `server`='".$value['server']."' LIMIT 1");
if(!mysql_num_rows($sh))
{
$massquery .= '("'.$value['artistname'].'", "'.$value['songname'].'", "'.$value['duration'].'", "'.$value['url'].'", "'.$value['server'].'"),';
}
}
if(!empty($massquery))
{
$massquery = substr($massquery, 0, -1);
$query = mysql_query('INSERT INTO `music` (`artistname`, `songname`, `duration`, `url`, `server`) VALUES '.$massquery);
}
mysql_close($mysql);
It turns out 50 requests "SELECT" to the database, which is very bad = (
How can I optimize this code?
From answers:
CREATE TABLE `music` (
`id` int(50) NOT NULL auto_increment,
`artistname` varchar(50) NOT NULL,
`songname` varchar(50) NOT NULL,
`duration` varchar(6) NOT NULL,
`url` varchar(255) NOT NULL,
`server` int(5) NOT NULL,
PRIMARY KEY (`id`),
KEY `artistname` (`artistname`,`songname`,`server`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
INSERT INTO `music` VALUES ('test', 'btest', 1);
...
SELECT `artistname` , `songname` , `server`
FROM `music`
WHERE FALSE
OR (
`artistname` = 'test'
AND `songname` = 'btest'
AND `server` = '1'
)
OR (
`artistname` = 'sas'
AND `songname` = 'asf'
AND `server` = '1'
)
LIMIT 0 , 30
How do I INSERT those songs that are not yet in the database?
Sorry for bad english
You want to insert new records only if no other record with the tuple (artistname,songname,server) (already) exists.
If you create a unique index for these three fields MySQL won't insert a doublet. Then you can either use something like
INSERT IGNORE INTO
tablename
(a,b,c,x,y,z)
VALUES
(1,2,3,4,5,6),
(7,8,9,10,11,12),
...
(95,96,97,98,99,100)
or a prepared statement, e.g.
$pdo = new PDO("mysql:host=localhost;dbname=test", 'localonly', 'localonly');
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
/* test table */
$pdo->exec('
CREATE TEMPORARY TABLE foo (
id int auto_increment,
artistname varchar(64) not null,
songname varchar(64) not null,
duration varchar(16) not null,
url varchar(64) not null,
server int not null,
primary key(id),
unique key (artistname,songname,server)
)
');
$data = array(
array(':artistname' => 'Lady GaGa', ':songname' => 'Love Games', ':duration' => '3:31', ':url' => '7e91a5ca16ae', ':server' => 3),
array(':artistname' => 'DJ Layla', ':songname' => 'Single Lady', ':duration' => '3:20', ':url' => 'f0906a3087eb', ':server' => 3),
array(':artistname' => 'Lady Gaga', ':songname' => 'Bad Romance (Bimbo Jones Clean Radio Remix)', ':duration' => '3:59', ':url' => '36e77d5a80357', ':server' => 3)
);
/* the "actual" test script */
$stmt = $pdo->prepare('
INSERT IGNORE INTO
foo
(duration, artistname, songname, server, url)
VALUES
(:duration, :artistname, :songname, :server, :url)
');
// first run, all three records should be inserted
foreach( $data as $params ) {
$stmt->execute($params);
}
// second run
// same artist/songname, different server
$newData = $data[0]; $newData[':server'] = 4;
$data[] = $newData;
// and a completly new record
$data[] = array(':artistname' => 'xyz', ':songname' => 'The ABC song', ':duration' => '2:31', ':url' => 'whatever', ':server' => 2);
// again insert all records (including the three that have already been inserted)
foreach( $data as $params ) {
$stmt->execute($params);
}
/* fetch all records */
foreach( $pdo->query('SELECT * FROM foo', PDO::FETCH_NUM) as $row ) {
echo join(', ', $row), "\n";
}
prints
1, Lady GaGa, Love Games, 3:31, 7e91a5ca16ae, 3
2, DJ Layla, Single Lady, 3:20, f0906a3087eb, 3
3, Lady Gaga, Bad Romance (Bimbo Jones Clean Radio Remix), 3:59, 36e77d5a80357, 3
4, Lady GaGa, Love Games, 3:31, 7e91a5ca16ae, 4
5, xyz, The ABC song, 2:31, whatever, 2
The first three records have not been duplicated.
Create a single select for all the relevant cases like this, and verify the results by means of PHP:
$sh = "SELECT `artistname`,`songname`,`server` FROM `music` WHERE ";
$pq = ""
foreach($res as $value)
{
if(!get_magic_quotes_gpc())
{
$value['artistname'] = mysql_escape_string($value['artistname']);
$value['songname'] = mysql_escape_string($value['songname']);
$value['duration'] = mysql_escape_string($value['duration']);
$value['url'] = mysql_escape_string($value['url']);
$value['server'] = mysql_escape_string($value['server']);
}
$value['artistname'] = trim($value['artistname']);
$value['songname'] = trim($value['songname']);
$value['duration'] = trim($value['duration']);
$value['url'] = trim($value['url']);
$value['server'] = trim($value['server']);
$sh .= $pq . `(artistname`='".$value['artistname']."' AMD `songname`='".$value['songname']."' AND `server`='".$value['server']."')");
$pq = " OR ";
}
$res = mysql_query($sh);
The select query you wrote can't be inside the foreach, or of course it'll query each time. Turn your $res into a long WHERE clause:
$sql = "SELECT `artistname`,`songname`,`server` FROM `music` WHERE FALSE ";
foreach($res as $value)
{
// ...
$sql .= "OR (`artistname`='".$value['artistname']."' AND `songname`='".$value['songname']."' AND `server`='".$value['server'].")";
}
and then run that query against the database and build your INSERT query.