how to use MySQL bitwise operations in php? - php

im trying to use MySQL bitwise operations for my query and i have this example:
table1
id ptid
1 3
2 20
3 66
4 6
table2
id types
1 music
2 art
4 pictures
8 video
16 art2
32 actor
64 movies
128 ..
...
now, the id = 3 from table1 is '66', witch means that it has 64 or movies and 2 or art
but
doesn't he also have 32 or actor twice and 2 or art ??
hope you see where my confusion is. How do i control what result i want back. In this case i want 64 or movies and 2 or art.
But sometimes i want three id's from table2 to belong to an id from table1
any ideas?
Thanks

Using bitwise OR
The following query returns all the items from table 2 in 66:
SELECT *
FROM table2
WHERE id | 66 = 66
But 32 + 32 = 64?
Though 32 + 32 = 64, it doesn't affect us.
Here's 64 in binary:
01000000
Here's 32 in binary:
00100000
Here's 2 in binary:
00000010
It's the position of the 1 that we use in this case, not the value. There won't be two of anything. Each flag is either on or off.
Here's 66 in binary. Notice that 64 and 2 are turned on, not 32:
01000010
Using bitwise AND instead of OR
Another way to write the query is with bitwise AND like this:
SELECT *
FROM table
WHERE id & 66 <> 0
Since 0 = false to MySQL, it can be further abbreviated like this:
SELECT *
FROM table
WHERE id & 66

select * from table2 where id & 66

Although the question on how to perform bitwise operations in MySQL has been answered, the sub-question in the comments about why this may not be an optimal data model remains outstanding.
In the example given there are two tables; one with a bitmask and one with a break down of what each bit represents. The implication is that, at some point, the two tables must be joined together to return/display the meaning of the various bits.
This join would either be explicit, e.g.
SELECT *
FROM Table1
INNER JOIN TABLE2
ON table1.ptid & table2.id <> 0
Or implicit where you might select the data from table1 into your application and then make a second call to lookup the bitmask values e.g.
SELECT *
FROM table2
WHERE id & $id <> 0
Neither of these options are ideas because they are not "sargable" that is, the database cannot construct a Search ARGument. As a result, you cannot optimize the query with an index. The cost of the query goes beyond the inability to leverage an index since for every row in the table, the DB must compute and evaluate an expression. This becomes very Memory, CPU and I/O intensive very quickly and it cannot be optimized without fundamentally changing the table structure.
Beyond the complete inability to optimize the query, it can also be awkward to read the data, report on the data, and you also potentially run into limits adding more bits (64 values in an 8 bit column might be fine now but not necessarily always so. They also make systems difficult to understand, and I would argue that this design violates first normal form.
Although using bitmasks in a database is often a sign of bad design, there are times when it's fine to use them. Implementing a many-to-many relationship really isn't one of those times.
The typical approach to implementing this type of relationship looks something like this:
table1
Id Val1 Val2
---------------------------
1 ABC DEF
2 ABC DEF
3 ABC DEF
4 ABC DEF
5 ABC DEF
6 ABC DEF
table2
id types
-------------
1 music
2 art
3 pictures
4 video
5 art2
6 actor
7 movies
table1-table2-relationshitp
table1ID Table2ID
---------------------
1 1
1 2
2 3
2 5
3 2
3 7
...
And you would query the data thusly
SELECT table1.*, table2.types
FROM table1
INNER JOIN table1-table2-relationship
ON table1.id = table1-table2-relationship.table1id
INNER JOIN table2
ON table1-table2-relationship.table2.id = table2.id
Depending on the access pattern of these tables, you would typically index both columns on the relationship table as a composite index (I usually treat them as a composite primary key.) This index would allow the database to quickly seek to the relevant rows in the relationship table and then seek to the relevant rows in table2.

After playing around with the answer from Marcus Adams, I thought I'd provide another example that helped me understand how to join two tables using bitwise operations.
Consider the following sample data, which defines a table of vowels, and a table of words with a single value representing the vowels present in that word.
# Create sample tables.
drop temporary table if exists Vowels;
create temporary table Vowels
(
Id int,
Letter varchar(1)
);
drop temporary table if exists Words;
create temporary table Words
(
Word varchar(20),
Vowels int
);
# Insert sample data.
insert into Vowels
select 1, 'a' union all
select 2, 'e' union all
select 4, 'i' union all
select 8, 'o' union all
select 16, 'u';
insert into Words
select 'foo', 8 union all
select 'hello', 10 union all
select 'language', 19 union all
select 'programming', 13 union all
select 'computer', 26;
We can now join the Vowel table to the Word table like so:
# List every word with its vowels.
select Word, Vowels, Letter, Id as 'Vowel Id'
from (
select *
from Words
) w
join Vowels v
where v.Id | w.Vowels = w.Vowels
order by Word, Letter;
And of course we can apply any conditions to the inner query.
# List the letters for just the words with a length < 6
select Letter
from (
select *
from Words
where length(Word) < 6
) w
join Vowels v
where v.Id | w.Vowels = w.Vowels
order by Word, Letter

Related

INNER JOIN too slow. how can it bee quicker

My code
SELECT * FROM andmed3 INNER JOIN test ON andmed3.isik like concat('%', test.isik, '%')
In andmed3 i have 130 000 rows and on test i have 10 000 rows, and it wont run.
When i limit it to 0,500 then it will query about 2-3 minutes.
How can it be better?
andmed3 table
id name number isik link stat else
-----------------------------------------------
1 john 15 1233213 none 11 5
8455666
7884555
test table
id isik
-----------
45 8455666
So i need all the rows from the andmed3 where is number what occures in test
The problem is the engine ill need to avalute the LIKE expression for each pair of rows in the join (130.000 X 10.000).
Also indexes are useless in this scenario because the expression need to be evaluated in order to accomplish the join (and you cannot put that expression INSIDE a index)
Maybe it's your architecture/schema the problem. When no one antecipated the need to join two tables based in a string expression.
Possible solution:
(It's a wild guess)
Hard to tell for sure from your example but if andmed3.isik contains all possible values to be used in the join you can try to put that in another table like it:
Andmed3Id isik
--------- -------
1 1233213
1 8455666
1 7884555
Of course to populate this table you ill need a strategy, possbile ones are: in the insert/update, in a batch in some late hour.
If this suits you just need to add one more join in your query.

Efficiently get diff of large data set?

I need to be able to diff the results of two queries, showing the rows that are in the "old" set but aren't in the "new"... and then showing the rows that are in the "new" set but not the old.
Right now, i'm pulling the results into an array, and then doing an array_diff(). But, i'm hitting some resource and timing issues, as the sets are close to 1 million rows each.
The schema is the same in both result sets (barring the setId number and the table's autoincrement number), so i assume there's a good way to do it directly in MySQL... but im not finding how.
Example Table Schema:
rowId,setId,userId,name
Example Data:
1,1,user1,John
2,1,user2,Sally
3,1,user3,Tom
4,2,user1,John
5,2,user2,Thomas
6,2,user4,Frank
What i'm needing to do, is figure out the adds/deletes between setId 1 and setId 2.
So, the result of the diff should (for the example) show:
Rows that are in both setId1 and setId2
1,1,user1,John
Rows that are in setId 1 but not in setId2
2,1,user2,Sally
3,1,user3,Tom
Rows that are in setId 2 but not in setId1
5,2,user2,Thomas
6,2,user4,Frank
I think that's all the details. And i think i got the example correct. Any help would be appreciated. Solutions in MySQL or PHP are fine by me.
You can use exists or not exists to get rows that are in both or only 1 set.
Users in set 1 but not set 2 (just flip tables for the opposite):
select * from set1 s1
where set_id = 1
and not exists (
select count(*) from set1 s2
where s1.user1 = s2.user1
)
Users that are in both sets
select * from set2 s2
where set_id = 2
and exists (
select 1 from set1 s1
where s1.setId = 1
and s2.user1 = s1.user1
)
If you only want distinct users in both groups then group by user1:
select min(rowId), user1 from set1
where set_id in (1,2)
group by user1
having count(distinct set_id) = 2
or for users in group but not the other
select min(rowId), user1 from set1
where set_id in (1,2)
group by user1
having count(case when set_id <> 1 then 1 end) = 0
What we ended up doing, was adding a checksum column to the necessary tables being diffed. That way, instead of having to select multiple columns for comparison, the diff could be done against a single column (the checksum value).
The checksum value was a simple md5 hash of a serialized array that contained the columns to be diffed. So... it was like this in PHP:
$checksumString = serialize($arrayOfColumnValues);
$checksumValue = md5($checksumString);
That $checksumValue would then be inserted/updated into the tables, and then we can more easily do the joins/unions etc on a single column to find the differences. It ended up looking something like this:
SELECT i.id, i.checksumvalue
FROM SAMPLE_TABLE_I i
WHERE i.checksumvalue not in(select checksumvalue from SAMPLE_TABLE_II)
UNION ALL
SELECT ii.id, ii.checksumvalue
FROM SAMPLE_TABLE_II ii
WHERE ii.checksumvalue not in(select checksumvalue from SAMPLE_TABLE_I);
This runs fast enough for my purposes, at least for now :-)

fetch the comma separate field in mysql query

My table id and skillid values are
1=>3
2 =>5
3 =>5,6,8
I have tried the query to fetch the value from my table
SELECT * FROM mytable WHERE skillid IN ('3', '5');
But i am getting only two rows, that was first and second row.
But i need the result three rows.
Thanks in advance.
The 'IN' comparison operator works like an equality operator. This:
expr IN ('3','5')
is equivalent to:
( expr = '3' OR expr = '5' )
That should be sufficient to explain why your query is not returning what you expect.
If your table has a character column that contains a comma separated list, like this:
id skill_id_list
-- -------------
1 3
2 5
3 5,6,8
Then one option, to search for a particular id value in the list is to make use of the MySQL FIND_IN_SET function. For example:
SELECT t.id
, t.skill_id_list
FROM mytable t
WHERE FIND_IN_SET('3', t.skill_id_list)
OR FIND_IN_SET('5', t.skill_id_list)
The MySQL FIND_IN_SET function is documented here:
https://dev.mysql.com/doc/refman/5.5/en/string-functions.html#function_find-in-set
Of course, that's only one option. There are several other ways to get an equivalent result.
If there are no spaces in the skill_id_list column, then another option is to use a LIKE operator, but taking care to handle the edge cases.
There are four cases you would need to check for the id value appearing:
as the only one in the list
at the beginning of the list
in the middle of the list
at the end of the list
To explicitly handle those four cases, you could check each one, for example:
WHERE skill_id_list LIKE '3' -- only one in list
OR skill_id_list LIKE '3,%' -- beginning of list
OR skill_id_list LIKE '%,3,%' -- middle of list
OR skill_id_list LIKE '%,3' -- end of list
Or, another way to approach that is to turn all of those cases into the single "middle of the list" by just appending a comma to the beginning and end of the list, and then doing a single check, for example:
WHERE CONCAT(',',skill_id_list,',') LIKE '%,3,%'
Note that spaces embedded in the list could cause a row not to match. The MySQL REPLACE function can be used to remove spaces, replacing all spaces with an empty string, something like this:
WHERE CONCAT(',',REPLACE(skill_id_list,' ',''),',') LIKE '%,3,%'
NOTE
The preceding attempts to to answer the question you asked. The following addresses a fundamentally different (though closely related) issue, concerning the relational dataa model and normalization.
Repeating attributes in normalized relational model are represented in separate table. That's the normative relational model. We typically avoid storing comma separated lists, and instead implement a separate table to store the repeating attributes.
For example, if a job has multiple skills, we would typically create another table to hold the list of job skills:
CREATE TABLE job
( id INT NOT NULL PRIMARY KEY
, description VARCHAR(10)
);
CREATE TABLE job_skill
( job_id INT NOT NULL COMMENT 'FK ref job.id'
, skill_id INT NOT NULL COMMENT ''
, PRIMARY KEY (job_id, skill_id)
, FOREIGN KEY job_skill_FK1 (job_id) REFERENCES job (id)
);
We'd represent the data in your model as five separate rows:
job_skill
job_id skill_id
------ --------
1 3
2 5
3 5
3 6
3 8
That's the normative pattern.
To get back a list of job ids that require skills 3 or 5:
SELECT s.job_id
FROM job_skill s
WHERE s.skill_id in (3,5)
GROUP BY j.job_id
To get back a list of job ids that require skills 3 and 5, there's several ways to do that, for example:
SELECT s.job_id
FROM job_skill s
ON s.job_id = t
WHERE s.skill_id in (3,5)
GROUP BY j.job_id
HAVING COUNT(DISTINCT s.skill_id) = 2
If it's more convenient for your use case to get back a comma separated list as a string, you can use the GROUP_CONCAT aggregate function:
SELECT j.id AS job_id
, GROUP_CONCAT(s.skill_id ORDER BY s.skill_id) AS skill_id_list
FROM job j
LEFT
JOIN job_skill s
ON s.job_id = j.id
WHERE ...
ORDER BY ...
Please check
SELECT * FROM mytable WHERE 5 IN (skillid) OR 3 IN (skillid);
SELECT * FROM mytable WHERE skillid LIKE '3%' OR skillid LIKE '5%';
Definetly this is NOT the way to do it...
YOU SHOULD NORMALIZE your table to avoid this kind of problems
This is not good practice, you should normalize your table table, but REGEXP would be helpful
Please try this way
SELECT * FROM mytable WHERE
skillid REGEXP '[[:<:]]3[[:>:]]' OR skillid REGEXP '[[:<:]]5[[:>:]]'

How to get single row from a query involving two tables. 2 Foreign Keys are repeated in one entry of child table

I have two tables
1. tbl_clubs
2. tbl_match_schedule
tbl_club has fields;
fld_id | fld_club_name
tbl_match_schedule has fields;
fld_id | fld_club_id_one | fld_club_id_two | fld_match_time.
Now the flow is that admin will
1. go to the add new match schedule.
2. Select the name of club from drop down 1
3. Select the name of the 2nd club (playing against) from drop down 2
4. Give Match time and save.
Now the problem is that I am unable to select the data properly. I need one row, but it
gives me two rows when I put the following query
SELECT
ms.fld_id, ms.fld_id_club_one , ms.fld_id_club_two , ms.fld_match_time, c.fld_club_name, c.fld_id
FROM
tbl_match_schedule as ms, tbl_club as c
WHERE
c.fld_id = ms.fld_id_club_one OR c.fld_id = ms.fld_id_club_two
Please help me, I want something in this form "club name one" vs "club name two" at "19:00"
I am using mysql
If i understand correctly this should do:
SELECT c1.fld_club_name AS 'club_name_one', c2.fld_club_name AS 'club_name_two', s.fld_match_time AS 'at' FROM tbl_match_schedule s
LEFT JOIN tbl_clubs c1 ON s.fld_club_id_one = c1.fld_id
LEFT JOIN tbl_clubs c2 ON s.fld_club_id_two = c2.fld_id
Of course you can add a WHERE or ORDER clause after this to suit your further needs as long as you use the c1. and c2. shorthands when you refer to the clubs as the full table name would be ambiguous.

Advice for Java HashMap alternative in PHP?

I have a database with this kind of a table, has more than 10 million rows.
ID colA colB Length
1 seq1 seq11 1
2 seq1 seq11 11
3 seq3 seq33 21
4 seq3 seq33 14
I want to loop though colA first, get the relevant colB value, and check if there are any other occurrences of the same value.
For example in colB (seq11) there are 2 occurrences of colA(seq1), this time I have to combine those and output the sum of the length. Similar to this:
ID colA colB Length
1 seq1 seq11 12
2 seq3 seq33 35
I am a bit Java guy, but because my colleague has written everything in PHP and this will be just an adding, I need a PHP solution.
With Java I would have used hashmap, so that I would have the colA data once and just increment the value of "Length Column".
I tried this query in order to group by occurences:
SELECT COUNT(*) SeqName FROM SeqTable GROUP BY SeqName HAVING COUNT(*)>0;
This is something easily achieved within SQL rather than in programming logic:
SELECT colA, colB, SUM(Length) as `length_sum`
FROM SeqTable
GROUP BY colA, colB
Of course you would still need PHP to iterate through the result set and do whatever it is you want to do with the data.
In PHP you can use an array like an hash map
$array = Array();
$array['seq1'] = Array();
$array['seq1']['seq11'] = 0;
$array['seq1']['seq11']++;
Or you can use an SQL query like this one:
select id,colA,colB,sum(Length) as Length from {tableName} group by colA,colB order by colA, colB;

Categories