MySQL php INSERT INTO taking very very long time - php

I want to insert 0 to 10000000 (10 million) of primary keys into my table.
My table is as follow:
p_key--> (INT[10]) AUTO_INCREMENT PRIMARY KEY UNIQUE
hash_value--> (SMALLINT)
req_count--> (SMALLINT)
only adding 100 records is taking 11 seconds
for 500 records it is giving 30 second timeout has been crossed.(wamp server localhost php mysql)
for ($i=0;$i<100;$i++){
$query='INSERT INTO hash_table (hash_value, req_count) VALUES (0,0)';
$result=$con->query($query);
if(!$result){
echo ("Error message:".$con->error." \n");
exit();
}
}
Am I doing something terribly wrong.

You should insert more then one records at one SQL query.
For example:
$values='';
for ($i=0;$i<100;$i++){
$values.='(0,0),';
}
$values=substr($values,0,-1);
$query='INSERT INTO hash_table (hash_value, req_count) VALUES '.$values;
$result=$con->query($query);
if(!$result){
echo ("Error message:".$con->error." \n");
exit();
}

Another option, doing it in a single SQL statement:-
INSERT INTO hash_table (hash_value, req_count)
SELECT 0, 0
FROM
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)units,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)tens,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)hundreds,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)thousands,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)tensthousands,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)hundredthousands,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9)millions,
(SELECT 0 i UNION SELECT 1)tensmillions
WHERE (units.i +
tens.i * 10 +
hundreds.i * 100 +
thousands.i * 1000 +
tensthousands.i * 10000 +
hundredthousands.i * 100000 +
millions.i * 1000000 +
tensmillions.i * 10000000) BETWEEN 0 AND 10000000

Related

How to manage table aliases in Laravel Query Builder functions from() and leftJoin()?

I have a complex query:
$sql = "SELECT dateQ.selected_date, apiQ.no_of_error AS error
FROM (
SELECT * from
(SELECT adddate('1970-01-01',t4.i*10000 + t3.i*1000 + t2.i*100 + t1.i*10 + t0.i) selected_date from
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t0,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t1,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t2,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t3,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t4) v WHERE selected_date BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()
)
dateQ LEFT JOIN
(
SELECT date(created_at) as ddate, COUNT(id) AS no_of_error
FROM `api_logs`
WHERE status_code <> 200
AND api_id = '" . $rowApi['id'] . "'
AND created_at BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()
GROUP BY date(created_at)
)
apiQ ON DATE(apiQ.ddate) = dateQ.selected_date GROUP BY dateQ.selected_date ORDER BY dateQ.selected_date DESC";
I want to split it using Laravel query builder.
The subquery:
SELECT date(created_at) as ddate, COUNT(id) AS no_of_error
FROM `api_logs`
WHERE status_code <> 200
AND api_id = '" . $rowApi['id'] . "'
AND created_at BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()
GROUP BY date(created_at)
is modified like this:
$apiQ = DB::table('api_logs')
->select('date(created_at) as ddate', DB::raw('COUNT(id) AS no_of_error'))
->where('api_id', $rowApi['id'])
->where('status_code', '<>', 200)
->whereRaw('created_at BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()')
->groupBy('date(created_at)');
Then the other subquery,
SELECT * from
(SELECT adddate('1970-01-01',t4.i*10000 + t3.i*1000 + t2.i*100 + t1.i*10 + t0.i) selected_date from
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t0,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t1,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t2,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t3,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t4)
v WHERE selected_date BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()
is modified like this:
$dateQ = DB::select('*')
->from("(SELECT adddate('1970-01-01',t4.i*10000 + t3.i*1000 + t2.i*100 + t1.i*10 + t0.i) selected_date from
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t0,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t1,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t2,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t3,
(SELECT 0 i UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) t4) v")
->whereRaw('selected_date BETWEEN CURDATE() - INTERVAL 90 DAY AND CURDATE()');
Now I finally trying to build up the entire query, like this:
$final= DB::select('dateQ.selected_date', 'apiQ.no_of_error AS error')
->from($dateQ, 'AS dateQ')
->leftJoin($apiQ, function($join){
$join->on(DB::RAW('DATE(apiQ.ddate'), '=', 'dateQ.selected_date')
})
->groupBy('dateQ.selected_date')
->orderBy('dateQ.selected_date', 'DESC');
But I cannot make the appropriate work out for the aliases like 'dateQ and apiQ'.

Raw Query Union All laravel

I have this query
select n as monthnum, monthname(100*n+1) as month, 0 as usr
from (
select 1 as n union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9 union all select 10 union all select 11 union all select 12
) q
UNION ALL
SELECT Monthnum, Month, SUM(usr) as jumlah
FROM
(
SELECT MONTH(t.created_at) AS monthnum, MONTHNAME(t.created_at) AS Month, 1 AS usr FROM job_applications t
WHERE YEAR(t.created_at) = YEAR(CURRENT_TIMESTAMP) AND MONTH(t.created_at) < 12 and job_id = 2
) q
GROUP BY monthnum, Month
ORDER BY monthnum
How i get data from laravel ?
I am already try
JobApplication:select(Raw(Query))
Always get error
u can
use DB
$test_query = "select n as monthnum, monthname(100*n+1) as month, 0 as usr
from (
select 1 as n union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9 union all select 10 union all select 11 union all select 12
) q
UNION ALL
SELECT Monthnum, Month, SUM(usr) as jumlah
FROM
(
SELECT MONTH(t.created_at) AS monthnum, MONTHNAME(t.created_at) AS Month, 1 AS usr FROM job_applications t
WHERE YEAR(t.created_at) = YEAR(CURRENT_TIMESTAMP) AND MONTH(t.created_at) < 12 and job_id = 2
) q
GROUP BY monthnum, Month
ORDER BY monthnum";
$test_results = DB::select($test_query);

How to separate the row value one by one with count function option in mysql

Mysql Table:
In My facility table is this
facility_name mbid date
yoga,aerobics,table tennis,tai chi, OM1111 2016-06-12
aerobics,tai chi, OM1111 2016-06-12
How to split row value one by one with mbid in mysql:
Facility_name mbid Number of count
yoga OM1111 1
aerobics OM1111 2
table tennis OM1111 1
tai chi OM1111 2
CREATE TABLE facility
(facility_name varchar(35), mbid varchar(6), date varchar(10))
;
INSERT INTO facility
(facility_name, mbid, date)
VALUES
('yoga,aerobics,table tennis,tai chi,', 'OM1111', '2016-06-12'),
('aerobics,tai chi,', 'OM1111', '2016-06-12')
;
Script :
Select T.VALUE,T.mbid,COUNT(T.VALUE)Cnt FROM (
SELECT SUBSTRING_INDEX(SUBSTRING_INDEX(t.facility_name, ',', n.n), ',', -1) value,mbid
FROM facility t CROSS JOIN
(
SELECT a.N + b.N * 10 + 1 n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
ORDER BY n
) n
WHERE n.n <= 1 + (LENGTH(t.facility_name) - LENGTH(REPLACE(t.facility_name, ',', ''))))T
WHERE T.VALUE <> ''
GROUP BY T.VALUE,T.mbid
ORDER BY T.value
How to pass the date function in where condition to get count of activity :
Select facility.mbid,membership.name,membership.organization,
membership.designation,membership.division, facility.VALUE `Facility Name`,
COUNT(facility.VALUE)`Number of Activite` FROM ( SELECT SUBSTRING_INDEX(SUBSTRING_INDEX(facility.facility_name, ',', n.n), ',', -1) value,mbid FROM facility CROSS JOIN
(
SELECT a.N + b.N * 10 + 1 n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
ORDER BY n
) n
WHERE n.n <= 1 + (LENGTH(facility.facility_name) -LENGTH(REPLACE(facility.facility_name, ',', ''))))T
facility Inner Join membership ON facility.mbid=membership.mbid
where facility.date Between '2016-06-04' and '2016-06-07' &&
facility.VALUE <> ''
GROUP BY facility.VALUE,facility.mbid ORDER BY facility.value
You have a very poor database structure. You should have one row per mbid and facility_name. In fact, I have no idea why mbid and date are the same in the two rows.
If I assume that you have facility names in another table, then you can use:
select fn.facility_name, f.mbid, count(*)
from facility f join
facility_names fn
on find_in_set(fn.facility_name, f.facility) > 0
group by fn.facility_name, f.mbid;
I should emphasize, though, that although you can do a query like this, you really need to fix your data structure. Storing lists of values in a string is the wrong way to store data in a SQL database.

Fastest way to generate 11,000,000 unique ids

I am trying to create php script that generates 11,000,000 million unique ids in sequential order. However, I am trying to do it very quickly within 20 min it should generate these 11 million unique ids. Also, once it reaches 12,000,000 it should wrap around and start back at zero.
Here is what I have so far. This script would would only return one id at a time. I just added a loop to see how long it would take to generate the ids.
while(true){
try {
$this->getAdapter()->query('INSERT INTO generate_ids (assigned_id) SELECT (MAX(assigned_id)+1) FROM generate_ids');
$id = $this->getAdapter()->lastInsertId();
$sql = 'SELECT assigned_id FROM generate_ids WHERE id = $id';
$query = $this->getAdapter()->query($sql);
$result = $query->fetchAll();
//Live system would return id here
$assigned_id = $result[0]['assigned_id'];
} catch (Exception $e) {
//do nothing
}
if($count == 11000000){
die();
}
$count++;
}
}
If you create the following table:
CREATE TABLE sequence (
sequence_id BIGINT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (`sequence_id`)
)
Then issue these three queries one after the other:
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
SELECT LAST_INSERT_ID() AS sequence;
The third query is guaranteed to return a unique sequence number. This guarantee holds even if you have dozens of different client programs connected to your database. That's the beauty of AUTO_INCREMENT.
Instead of just generating eleven million of these sequence numbers up front, you can use these SQL queries to get a unique sequence number whenever you need it.
If you must wrap around at sequence number 12 million you can use these queries instead.
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
SELECT LAST_INSERT_ID() MOD 12000000 AS sequence;
The trick here is to use an auto-increment sequence number for uniqueness, but to also delete the rows in the table so it doesn't gobble up lots of space.
Note that you can also use the sequence number of LAST_INSERT_ID() for other purposes, like so for example.
INSERT INTO sequence () VALUES ();
DELETE FROM sequence WHERE sequence_id < LAST_INSERT_ID();
INSERT INTO user (userid, username, phone)
VALUES (LAST_INSERT_ID() MOD 12000000, 'Joe', '800-555-1212');
SELECT LAST_INSERT_ID() MOD 12000000 AS sequence;
If you need to insert all your ids at once (for some reason) the fastest approach would be to do it in pure SQL
insert into generate_ids (assigned_id)
select N
from
(
select a.N + b.N * 10 + c.N * 100 + d.N * 1000 + e.N * 10000 + f.N * 100000 + g.N * 1000000 + h.N * 10000000 + 1 as N
from (select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) a
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) b
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) c
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) d
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) e
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) f
,(select 0 as N union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) g
,(select 0 as N union all select 1) h
order by N
) q
where N <= 11000000
It takes less than a minute to complete on my laptop.

MySQL inner join three tables to use in calendar

Following up my question where I used the answer to generate data on my calendar called maintenance calendar showing the aircraft's maintenance schedule. This is the MySQL query for it:
SELECT DISTINCT s.reg AS 'reg',
a.date AS 'date'
FROM (SELECT Curdate()
+ INTERVAL (a.a + (10 * b.a) + (100 * c.a)) day AS Date
FROM (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS a
CROSS JOIN (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS b
CROSS JOIN (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS c) a
INNER JOIN maintenance_sched s
ON a.date >= s.date_from
AND a.date <= s.date_to
WHERE Month(date) = '".$month."'
AND Dayofmonth(date) = '".$dayArray["mday"]."'
AND Year(date) = '".$year."'
Here is the maintenance_sched database:
And the calendar looks like this (based on the data from maintenance_sched):
Then, I have another calendar called reservation calendar with the same code as the maintenance calendar though with different query. This is the reservation calendar query: SELECT acode FROM reservation WHERE month(etd) = '".$month."' AND dayofmonth(etd) = '".$dayArray["mday"]."' AND year(etd) = '".$year."' ORDER BY etd".
The reservation table is this:
And the reservation calendar looks like this:
EDIT:
What I want to do is: have these two calendar in one calendar with the result of maintenance_sched query outputted as string with strikethrough. But I can't seem to make the two queries work out together as one.
I do think the answer to this question is to simply join the two queries. An example of this might be like below where you just null out any columns that aren't in your second table.
SELECT id, date, field3, description
FROM table1
UNION
SELECT id, date, field3, null
FROM table2
As there is no relationship among both the table we cannot go for joins, it would be better to go for UNION to combine the result.
This query uses group_concat so will generate common results in following form
2013-03-15 | RP-C1728, RP-C1086
2013-03-08 | RP-C1728, RP-C1086, RP-C143
If you dont want record in this format then just remove group_concat, group by clause from the query.
Query
SELECT a.date, group_concat(a.reg)
FROM
(SELECT DISTINCT s.reg AS 'reg',
a.date AS 'date'
FROM (SELECT Curdate()
+ INTERVAL (a.a + (10 * b.a) + (100 * c.a)) day AS Date
FROM (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS a
CROSS JOIN (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS b
CROSS JOIN (SELECT 0 AS a
UNION ALL SELECT 1
UNION ALL SELECT 2
UNION ALL SELECT 3
UNION ALL SELECT 4
UNION ALL SELECT 5
UNION ALL SELECT 6
UNION ALL SELECT 7
UNION ALL SELECT 8
UNION ALL SELECT 9) AS c) a
INNER JOIN maintenance_sched s
ON a.date >= s.date_from
AND a.date <= s.date_to
WHERE Month(date) = '".$month."'
AND Dayofmonth(date) = '".$dayArray["mday"]."'
AND Year(date) = '".$year."'
UNION ALL
SELECT acode as 'reg', date as 'date' //Add the date logic here as per your need
FROM reservation
WHERE month(etd) = '".$month."' AND
dayofmonth(etd) = '".$dayArray["mday"]."' AND
year(etd) = '".$year."' ORDER BY etd) a
GROUP BY a.date;
NOTE For the second query add the according date logic

Categories