I have the following table in mysql:
CREATE TABLE `kampbs` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`user` varchar(20) COLLATE utf8_danish_ci DEFAULT NULL,
`k1` varchar(1) COLLATE utf8_danish_ci DEFAULT NULL,
`k1r` varchar(10) COLLATE utf8_danish_ci DEFAULT NULL,
`k2` varchar(1) COLLATE utf8_danish_ci DEFAULT NULL,
`k2r` varchar(10) COLLATE utf8_danish_ci DEFAULT NULL,
`k3` varchar(1) COLLATE utf8_danish_ci DEFAULT NULL,
`k3r` varchar(10) COLLATE utf8_danish_ci DEFAULT NULL,
`week` int(2) DEFAULT NULL,
`grp` varchar(11) COLLATE utf8_danish_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=5 DEFAULT CHARSET=utf8 COLLATE=utf8_danish_ci
I have a form, that INSERT INTO the fields in the database, but it should only be possible to insert once a week pr user. So it should check if the column 'user' and 'week' are unique together. Make sense? :/
Let's say the user 'freak' submit in week 13, he shouldn't be able to submit once again in week 13.
I use the following $sql:
$sql = "INSERT INTO kampbs(user, k1, k1r, k2, k2r, k3, k3r, week, grp) VALUES('$username', '$k1', '$k1r', '$k2', '$k2r', '$k3', '$k3r', '$week', '$grp')";
You can add a constraint to your table, so that the user / week couple is forced to be unique
ALTER TABLE `kampbs` ADD UNIQUE `unique_index`(`user`, `week`);
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I have a list of contents on my website and it is getting filled with records rapidly, for example after one month, there are 300K records in it. Because of this, the list page is responding slower and slower over time.
This page has this functionalities:
Searching
Paging
Also this table Left Joins other tables and it increases the run-time.
I run two queries every time this page is loaded, one for getting 10 limited records, and the other for getting number of all records.
How do I handle this amount of data with out jeopardizing user experience?
EDIT
Here is my Query:
SELECT *,
`note`.`attached_file` AS `attached_file`,
`note`.`description` AS `description`,
`note`.`id` AS `id`,
`note_type`.`title` AS `title`,
`note_goal`.`title` AS `goal`
FROM `note`
LEFT JOIN `contact` ON `note`.`id_contact`=`contact`.`id`
LEFT JOIN `contact_activity_field` ON `contact_activity_field`.`id_contact`=`contact`.`id`
LEFT JOIN `activity` ON `contact_activity_field`.`id_activity`=`activity`.`id`
LEFT JOIN `note_type` ON `note`.`title`=`note_type`.`id`
LEFT JOIN `note_goal` ON `note`.`goal`=`note_goal`.`id`
WHERE
( `note_type`.`title` LIKE '%$q%' OR
`firstname_eng` LIKE '%$q%' OR
`lastname_eng` LIKE '%$q%' OR
`firstname_per` LIKE '%$q%' OR
`lastname_per` LIKE '%$q%' OR
`company_name` LIKE '%$q%' OR
`company_name_per` LIKE '%$q%' OR
`description` LIKE '%$q%' OR
`note_goal`.`title` LIKE '%$q%' ) AND some other condition
GROUP BY `note`.`id`
ORDER BY `note`.`id` DESC
LIMIT $start_from, 10
EDIT 2
note table
CREATE TABLE `note` (
`id` int(10) NOT NULL AUTO_INCREMENT,
`id_user` int(10) NOT NULL,
`id_contact` int(10) NOT NULL,
`title` int(10) DEFAULT NULL,
`goal` int(10) DEFAULT NULL,
`register_date` date DEFAULT NULL,
`description` text COLLATE utf8_unicode_ci,
`attached_file` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`meeting_place` text COLLATE utf8_unicode_ci,
`start_time` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`finish_time` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3297 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
contact table
CREATE TABLE `contact` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`id_user` int(10) NOT NULL DEFAULT '0',
`id_user_registered` int(10) DEFAULT NULL,
`code` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`contact_type` varchar(20) COLLATE utf8_unicode_ci DEFAULT 'legal',
`firstname_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastname_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`firstname_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`lastname_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`gender` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`id_number` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`national_code` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`birth_date` date DEFAULT NULL,
`company_name` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`company_name_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`recommender_eng` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`recommender_per` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`company_type` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`registration_type` varchar(50) COLLATE utf8_unicode_ci DEFAULT NULL,
`registration_date` date DEFAULT '1900-01-01',
`registration_number` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`national_id` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`economic_code` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
`website` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`postal_code` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`address_eng` text COLLATE utf8_unicode_ci,
`address_per` text COLLATE utf8_unicode_ci,
`phone_number` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`fax` varchar(1000) COLLATE utf8_unicode_ci DEFAULT NULL,
`activity_comment` text COLLATE utf8_unicode_ci,
`level` varchar(50) COLLATE utf8_unicode_ci DEFAULT 'Basic',
`guild` varchar(500) COLLATE utf8_unicode_ci DEFAULT NULL,
`verify_comment` text COLLATE utf8_unicode_ci,
`status` int(1) NOT NULL DEFAULT '0',
`comment` text COLLATE utf8_unicode_ci,
`submitted` int(1) NOT NULL DEFAULT '0',
`assign_date` datetime DEFAULT NULL,
`created_date` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
`updated_date` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
KEY `id` (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=57357 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
And other tables just contain a simple id and title records.
Here is the result of EXPLAIN query:
These results are on my local database.
LIKE with a leading wild card is inefficient -- and will scan the entire table, hence the "slower as it gets bigger". Switch to FULLTEXT.
Pagination via OFFSET is inefficient because it must scan all the rows befoer the few desired. See http://mysql.rjweb.org/doc.php/pagination
Both of these have been discussed on stackoverflow repeatedly. Search for other discussions.
I'm trying to add job info to my DB from an API.
One of my feeds supplies a date(posted) and an expiration date.
I'm trying to insert both fields (along with others).
The date gets entered fine, however the expiry date gets set to the default 0000-00-00 00:00:00
example data from api:
{
"jobId":30081256,
"expirationDate":"22/08/2016",
"date":"11/07/2016"
}
I parse the dates ready to be entered like this:
$job_date = $job->date;
$timestamp = strtotime(str_replace('/', '-', $job_date));
$date = date("Y-m-d H:i:s", $timestamp);
//expiry
$job_expiry = $job->expirationDate;
$expiry_timestamp = strtotime(str_replace('/', '-', $job_expiry));
$expiry_date = date("Y-m-d H:i:s", $expiry_timestamp);
when I echo the results of $date and $expiry_date, i get:
2016-07-11 00:00:00
and
2016-08-22 00:00:00
to me all looks good and the dates are formatted as sql would expect, but the expiry date does not get set.
I'm not sure if it has something to do with my query (shortened to show the fields in question - all other fields insert correctly):
"INSERT INTO
jobs2 (jobref, date, expiry)
VALUES ('"
.$jobref
."','"
.$date
."','"
.$expiry_date
."') ON DUPLICATE KEY UPDATE
`date` = VALUES(date),
`expiry` = VALUES(expiry)"
Maybe the ON DUPLICATE part? The job may already be in my database, but sometimes it has been refreshed/ updated and the posted date or expiry may be different so I want to keep them updated with the latest values from the API.
whats going on here?
table structure as requested:
CREATE TABLE `jobs2` (
`date` datetime NOT NULL,
`title` varchar(200) COLLATE latin1_general_ci NOT NULL,
`company` varchar(100) COLLATE latin1_general_ci NOT NULL,
`email` varchar(100) COLLATE latin1_general_ci NOT NULL,
`url` varchar(1200) COLLATE latin1_general_ci NOT NULL,
`salarymin` int(20) DEFAULT NULL,
`salarymax` int(20) DEFAULT NULL,
`benefits` varchar(100) COLLATE latin1_general_ci NOT NULL,
`salary` varchar(200) COLLATE latin1_general_ci DEFAULT NULL,
`jobtype` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`full_part` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`salary_per` varchar(20) COLLATE latin1_general_ci DEFAULT NULL,
`location` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`country` varchar(30) COLLATE latin1_general_ci NOT NULL,
`description` varchar(3000) COLLATE latin1_general_ci NOT NULL,
`category` varchar(100) COLLATE latin1_general_ci DEFAULT NULL,
`image` varchar(150) COLLATE latin1_general_ci NOT NULL,
`latitude` float NOT NULL,
`longitude` float NOT NULL,
`town` varchar(50) COLLATE latin1_general_ci NOT NULL,
`county` varchar(50) COLLATE latin1_general_ci NOT NULL,
`location_id` varchar(255) COLLATE latin1_general_ci NOT NULL,
`jobref` varchar(100) COLLATE latin1_general_ci NOT NULL,
`partner` varchar(30) COLLATE latin1_general_ci DEFAULT NULL,
`company_id` int(20) DEFAULT NULL,
`featured` varchar(10) COLLATE latin1_general_ci DEFAULT NULL,
`solr_stored` int(1) DEFAULT NULL,
`expiry` datetime NOT NULL,
UNIQUE KEY `location_id` (`location_id`),
KEY `partner` (`partner`),
KEY `company` (`company`),
KEY `title` (`title`),
KEY `company` (`company`),
KEY `longitude` (`longitude`),
KEY `latitude` (`latitude`),
KEY `jobref` (`jobref`),
KEY `jobtype` (`jobtype`),
KEY `company_id` (`company_id`),
KEY `salarymin` (`salarymin`),
KEY `salarymax` (`salarymax`),
KEY `salary_per` (`salary_per`),
KEY `date` (`date`),
FULLTEXT KEY `title` (`title`),
FULLTEXT KEY `company` (`company`),
) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci
I've got two tables (NewProducts and OldProducts) that are being compared. NewProducts has about 68,000 records and OldProducts about 51,000. I'm using a covering index on each table, however the query is taking 20 minutes to execute, so I'm not using it properly. Does a covering index really apply with multiple tables? What am I doing wrong? Thank you.
Here is my query code and the indexes:
$querystring = "SELECT newProducts.Id, newProducts.SKU,
newProducts.Title, oldProducts.Title, oldProducts.product_Id
FROM
newProducts, oldProducts
WHERE
trim(newProducts.SKU)=trim(oldProducts.SKU) and
trim(newProducts.Title)=trim(oldProducts.Title) and
oldProducts.Position=1 and
oldProducts.Customer=$shop";
Indexes for NewProducts:
Primary: Id
Index: SKU, Title, customer (not unique)
Indexes for OldProducts:
Primary: Id
Index: Product_id (not unique)
Index: SKU, Title, Postition, Customer (not unique)
?>
CREATE TABLE `NewProducts` (
`Id` bigint(11) NOT NULL,
`Title` varchar(120) COLLATE utf8_unicode_ci NOT NULL,
`Category` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`Office` varchar(150) COLLATE utf8_unicode_ci NOT NULL,
`Rehashed` smallint(6) NOT NULL,
`Quantity` smallint(6) NOT NULL,
`Price1` decimal(7,2) NOT NULL,
`Price2` decimal(7,2) NOT NULL,
`Price3` decimal(7,2) NOT NULL,
`Price4` decimal(7,2) NOT NULL,
`created_at` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`OldQuantity` int(11) NOT NULL,
`SKU` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`Source` varchar(12) COLLATE utf8_unicode_ci NOT NULL,
`customer` varchar(70) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`Id`),
UNIQUE KEY `I-T-S` (`ItemId`,`Title`,`SKU`),
KEY `customer` (`customer`),
KEY `Title` (`Title`,`Rehashed`),
KEY `SKU` (`SKU`),
KEY `Title_2` (`Title`,`SKU`,`customer`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
CREATE TABLE `OldProducts` (
`barcode` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`compare_at_price` decimal(10,2) DEFAULT NULL,
`created_at` varchar(30) COLLATE utf8_unicode_ci DEFAULT NULL,
`fulfillment` varchar(35) COLLATE utf8_unicode_ci DEFAULT NULL,
`grams` decimal(10,2) DEFAULT NULL,
`id` bigint(11) NOT NULL,
`management` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`policy` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`size` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`color` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`type` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`position` int(11) DEFAULT NULL,
`price` varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL,
`product_id` bigint(11) NOT NULL,
`SKU` varchar(55) COLLATE utf8_unicode_ci DEFAULT NULL,
`Title` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`quantity` int(11) DEFAULT NULL,
`customer` varchar(70) COLLATE utf8_unicode_ci NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `P-S-T-PO-CUST`
(`product_id`,`SKU`,`Title`,`position`,`customer`),
KEY `product_id` (`product_id`),
TRIM is the villain. When you hide an indexed column (eg, SKU) inside a function (eg, TRIM), the the index cannot be used.
Clean up your data:
Fix the insertion code to TRIM before inserting (or as it inserts).
UPDATE tbl SET SKU = TRIM(SKU), title = TRIM(title); -- for each table
Change the SELECT: TRIM(SKU) --> SKU etc.
Even Better
oldProducts should have, in this order,
`INDEX(`position`,`customer` ,`SKU`,`Title`, `product_id`)
With this, the WHERE need look only at old rows for position=1 and customer =.... (Actually, the first 2 columns can be in any order; the last 3 in any order.)
While inserting first time it gives following error: "Result consisted of more than one row". When i try to insert record second time it gives error with message duplicate entry.
SQL=INSERT INTO `master_user` (`name`,`user_name`,`email`,`password`,`system_name_of_friend`,`system_no_of_friend`,`registered_from_site`,`registered_on`,`is_existing_user`) VALUES ('FirstName LastName','username','demo#mail.com','8c71eede42e38709e9e836021b0b9b9b','','','site','','1')
any one help will be appropriated and following is the table structure which will be get help to tracking this issue very easily and get the solution for this.
CREATE TABLE IF NOT EXISTS `master_user` (
`master_user_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`user_name` varchar(100) COLLATE utf8_unicode_ci DEFAULT NULL,
`email` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`password` varchar(255) COLLATE utf8_unicode_ci NOT NULL,
`system_name_of_friend` varchar(100) COLLATE utf8_unicode_ci NOT NULL,
`system_no_of_friend` varchar(50) COLLATE utf8_unicode_ci NOT NULL,
`registered_from_ip` varchar(25) COLLATE utf8_unicode_ci DEFAULT NULL,
`registered_from_site` varchar(25) COLLATE utf8_unicode_ci DEFAULT NULL,
`registered_on` datetime DEFAULT NULL,
`is_existing_user` bit(1) NOT NULL DEFAULT b'0',
PRIMARY KEY (`master_user_id`),
UNIQUE KEY `ukMasterUser_email` (`email`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci AUTO_INCREMENT=1293 ;
I have changed the few words from column Which are conflicting and resolved.
I am using the redbeanPHP ORM and mysql. I have the following table:
CREATE TABLE `mast` (
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`note` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`geolocation` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`location` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`zip` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`state` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
`app` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `UQ_84a93b55f688c94f73092dba1b9590c41a92cbf5` ('app')
) ENGINE=InnoDB AUTO_INCREMENT=98 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
I want to insert records into the 'mast' table providing they are unique with respect to both of the 2 fields listed above. In other words if either 'geolocation' or 'app' is a duplicate, I don't want to insert the associated record.
I am using following php code to create the 2 unique fields:
$mast= R::dispense('mast');
$mast->setMeta("buildcommand.unique" , array(array('geolocation')));
$mast ->import($resultsarray);
$mast->setMeta("buildcommand.unique" , array(array('app')));
$id = R::store($mast); // DUMMY BEAN
A unique index is only being created for the 'app' field . Is there a way to set them both as unique in redbean?
From the documentation it should work like the following:
$mast = R::dispense('mast');
$mast->setMeta("buildcommand.unique.0", array('geolocation', 'app'));
$mast->import($resultsarray);
$id = R::store($mast);
In your code you just overwrote the value for build command.unique.