MySql Query - optimization with varchars, indexs, taking over an hour to run - php

So I need to run a query that I do not know the UUID - but need to find it... so I am using the street num, street name, and a company UUID to find it
I have a few million records, and this took query is taking around an HOUR!!
any advice to speed it up?
gisPoints
UUID Indexed Unique varchar(36)
street_num int(11)
street_name varchar(128)
geoPoint_temp
UUID Indexed Unique varchar(36)
street_num int(11)
street_name varchar(128)
gcomUUID Indexed varchar(36)
update geoPoint_temp as temp JOIN gisPoints as `prod` on prod.gcomUUID=temp.gcomUUIDand prod.street_num=temp.street_num and prod.street_name REGEXP(temp.street_name)
set temp.UUID=prod.UUID,temp.customerUUID=prod.customerUUID WHERE temp.`uploadstate` = '1'";

Assuming you have the following values (in PHP):
$street_num = ...;//something
$street_name = ...;//something
$gcomUUID = ...;//something
If you run the following sql code:
$sql = "SELECT * FROM (
SELECT * FROM (
SELECT * FROM geoPoint_temp WHERE gcomUUID = $gcomUUID)
WHERE street_name = $street_name)
WHERE street_num = $street_num;"
You should obtain a list of rows (0 or more) from geoPoint_temp that have matching values, and it should be relatively fast even in a big table.
After obtaining those rows, you can check if the row count is greater than zero, and if so update the rows. If your using MySQL (PDO), you could do something similar to the following:
$count = $stmt->rowCount();
if ($count>0)
{
$rows = $stmt->fetchAll();
foreach ($rows as $row)
{
$sql = "UPDATE geoPoint_temp SET ... WHERE UUID = ".$row['UUID'];
$stmt = $conn->prepare($sql);
$stmt->execute();
}
}
Let me know if that helped.
EDITED:
Try the following as well and let me know if it works:
$sql = "
UPDATE geoPoint_temp SET ... WHERE UUID IN
(SELECT * FROM (
SELECT * FROM (
SELECT * FROM geoPoint_temp WHERE gcomUUID = $gcomUUID)
WHERE street_name = $street_name)
WHERE street_num = $street_num);"
And replace ... with the values you want updated.

This runs in 1.5 seconds opposed to the hours it was taking before
Much help to #Webeng for pointing us in the right direction!
$custquery="UPDATE geoPoint_temp as temp
join
(
select prod.name, prod.street_num, prod.street_name, prod.UUID,prod.customerUUID, prod.gcomUUID
FROM gisPoints as `prod`
JOIN
(
select t1.gcomUUID , t1.street_num, t1.street_name
FROM geoPoint_temp as t1
) as sub1 on prod.gcomUUID =sub1.gcomUUID and prod.street_num=sub1.street_num
) as sub2 on sub2.gcomUUID =temp.gcomUUID
and sub2.street_num=temp.street_num
AND sub2.street_name LIKE (CONCAT('%',temp.street_name,'%'))
set temp.customerUUID = sub2.customerUUID, temp.UUID=sub2.UUID";
$custre=mysql_query($custquery);
if (!$custre) { echo 'Could not run custre query: ' . mysql_error(); exit; }

Related

Copy data from one MySQL table to another when data intersect using PHP

I'm new to programming and would really appreciate your help. :) So, I have a USER table and a SALES table. On the SALES table I only have name and last name of the users. On the USER table I have name, last name, USER_ID, email and etc...
I need to copy the USER_ID from the USER table to the SALES table when NAME and LAST NAME are a match.
Here is the structure:
USER_TABLE_A
USER_ID_A
NAME_LASTNAME_A
SALES_TABLE_B
ROW_ID_B
NAME_B
LASTNAME_B
USER_ID_B (empty)
So far I got both tables to show data when they intersect but have no idea where to go from here. Could anyone please help?
$sql1 = mysql_query("SELECT name_B, lastname_B, user_id_B, row_id_B FROM sales_table_B WHERE name_B IS NOT NULL AND lastname_B IS NOT NULL", $db);
$sql2 = mysql_query("SELECT name_lastname_A, user_id_A FROM user_table_A WHERE name_lastname_A IS NOT NULL", $db);
$a1 = array();
while ($row = mysql_fetch_array($sql1)) {
$id = $row['row_id_B'];
$name1.$id = $row['name_B']." ".$row['lastname_B'];
array_push($a1, $name1.$id);
}
$a2 = array();
while ($row2 = mysql_fetch_array($sql2)) {
$id2 = $row2['user_id_A'];
$name2.$id2 = $row2['name_lastname_A'];
array_push($a2, $name2.$id2);
}
$result = array_intersect($a1,$a2);
print_r($result);
Thanks in Advance!
Under the assumption that NAME_LASTNAME_A column in USER_TABLE_A is a concatenation of NAME_B and LASTNAME_B columns of SALES_TABLE_B, following UPDATE query should update the ids:
update sales_table_b
set b.user_id_b = (
select user_id_a from user_table_a
where name_lastname_a = concat(b.name_b, ' ' , b.lastname_b)
limit 1
where exists(
select user_id_a from user_table_a
where name_lastname_a = concat(b.name_b, ' ' , b.lastname_b)
)
)
Please note that in case of multiple users having same first and last name, id of the first matching user will be considered.
Thank you so much Darshan!! Your answer was missing a ) after limit 1 but with the adjustment it worked beautifully! Here is the code that worked:
UPDATE sales_table_b
SET user_id_b = (SELECT user_table_a.user_id_a
FROM user_table_a
WHERE user_table_a.name_lastname_a = CONCAT(sales_table_b.name_b, ' ' , sales_table_b.lastname_b) LIMIT 1)
WHERE EXISTS (SELECT *
FROM user_table_a
WHERE user_table_a.name_lastname_a = CONCAT(sales_table_b.name_b, ' ' , sales_table_b.lastname_b))

Select row from a table that does not exist in other table

I want to select a row that exist in the images table but not in the images_viewed table. They both share the same name.
This is the code I am using. it works, but it making my website very slow.
$next = $mysqli->query("SELECT `name` FROM `images` WHERE `name` NOT IN (SELECT `name` FROM `images_viewed` WHERE `ip` = '$ip') ORDER BY RAND() LIMIT 1");
if($next->num_rows == 1){
$fetch_next = $next->fetch_assoc();
$next_name = $fetch_next['name'];
}
how can I fix this code? is there a better select statement that I can use?
Try:
SELECT images.name FROM images
LEFT OUTER JOIN images_viewed
ON images.name = images_viewed.name
WHERE images_viewed.ANY_OTHER_FIELD IS NULL AND images_views.ip = {$ip}

Getting total in while statement with UNION query

I am trying to calculate how much a user has earned so it reflects on the users home page so they know how much their referrals have earned.
This is the code I have.
$get_ref_stats = $db->query("SELECT * FROM `members` WHERE `referral` = '".$user_info['username']."'");
$total_cash = 0;
while($ref_stats = $get_ref_stats->fetch_assoc()){
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."' UNION SELECT * FROM `completed_repeat` WHERE `user` = '".$ref_stats['username']."'");
$countr_cash = $get_ref_cash->fetch_assoc();
$total_cash += $countr_cash['cash'];
$countr_c_rate = $setting_info['ref_rate'] * 0.01;
$total_cash = $total_cash * $countr_c_rate;
}
It worked fine when I just had
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."'");
but as soon as I added in the UNION it no longer calculated correctly.
For example, there is 1 entry in completed and 1 entry in completed_repeat both of these entries have a cash entry of 0.75. The variable for $countr_c_rate is 0.10 so $total_cash should equal 0.15 but instead it displays as 0.075 with and without the UNION it acts as if it is not counting from the other table as well.
I hope this makes sense as I wasn't sure how to explain the issue, but I am very unsure what I have done wrong here.
In your second query instead of UNION you should use UNION ALL since UNION eliminates duplicates in the resultset. That is why you get 0.075 instead of 0.15.
Now, instead of hitting your database multiple times from client code you better calculate your cash total in one query.
It might be inaccurate without seeing your table structures and sample data but this query might look like this
SELECT SUM(cash) cash_total
FROM
(
SELECT c.cash
FROM completed c JOIN members m
ON c.user = m.username
WHERE m.referral = ?
UNION ALL
SELECT r.cash
FROM completed_repeat r JOIN members m
ON r.user = m.username
WHERE m.referral = ?
) q
Without prepared statements your php code then might look like
$sql = "SELECT SUM(cash) cash_total
FROM
(
SELECT c.cash
FROM completed c JOIN members m
ON c.user = m.username
WHERE m.referral = '$user_info['username']'
UNION ALL
SELECT r.cash
FROM completed_repeat r JOIN members m
ON r.user = m.username
WHERE m.referral = '$user_info['username']'
) q";
$result = $db->query($sql);
if(!$result) {
die($db->error()); // TODO: better error handling
}
if ($row = $result->fetch_assoc()) {
$total_cash = $row['cash_total'] * $setting_info['ref_rate'];
}
On a side note: make use of prepared statements in mysqli instead of building queries with concatenation. It's vulnerable for sql-injections.
With $countr_cash = $get_ref_cash->fetch_assoc(); you only fetch the first row of your result. However, if you use UNION, you get in your case two rows.
Therefore, you need to iterate over all rows in order to get all values.
Ok, So there is only one row in members table. You are iterating only once on the members table. Then you are trying to get rows using UNION clause which will result in two rows and not one. Then you are just getting the cash column of the first row and adding it to the $total_cash variable.
What you need to do is iterate over the results obtained by executing the UNION query and add the $total_cash variable. That would give you the required result.
$get_ref_stats = $db->query("SELECT * FROM `members` WHERE `referral` = '".$user_info['username']."'");
$total_cash = 0;
while($ref_stats = $get_ref_stats->fetch_assoc()){
$get_ref_cash = $db->query("SELECT * FROM `completed` WHERE `user` = '".$ref_stats['username']."' UNION SELECT * FROM `completed_repeat` WHERE `user` = '".$ref_stats['username']."'");
while($countr_cash = $get_ref_cash->fetch_assoc()){
$total_cash += $countr_cash['cash'];
}
$countr_c_rate = $setting_info['ref_rate'] * 0.01;
$total_cash = $total_cash * $countr_c_rate;
}

PHP Recursive Find siblings

working with this for a while.. cant get my head right.. sooo... help here ;-)
It is quite simple I am sure..
Table:
CREATE TABLE IF NOT EXISTS `items` (
`item_id` bigint(20) NOT NULL AUTO_INCREMENT,
`item_parent_id` bigint(20) NOT NULL COMMENT 'itemid which this item belongs to',
`item_name` varchar(255) NOT NULL,
`item_serialnumber` varchar(255) NOT NULL,
`item_model` varchar(255) NOT NULL,
PRIMARY KEY (`item_id`),
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
I am trying to create an array of item_id and the item_id that it belongs to - via the item_parent_id - recursivly -
so that even if you find a child to a parent, check if the child is a parent to others.
Tried with something like this:
function get_item($item_id, $menu)
{
$sql = "
SELECT
items.*,
customers.*
FROM
assets
LEFT JOIN item_customer_rel USING(item_id)
LEFT JOIN customers USING(customer_id)
WHERE
items.item_parent_id = '".$parent."'
ORDER BY
items.item_name
";
$res = mysqli_query($db, $sql) or die("ERROR: SQL Select a2a ancestor", $sql, mysqli_error($db) , $_SESSION["u_id"]);
while ($items = mysqli_fetch_assoc($res))
$menu = build_ancestor_array($parent, $menu);
}
function build_ancestor_array($parent, $menu)
{
GLOBAL $db;
$sql = "
SELECT
items.*,
customers.*
FROM
items
LEFT JOIN item_customer_rel USING(item_id)
LEFT JOIN customers USING(customer_id)
WHERE
items.item_parent_id = '".$parent."'
";
$res = mysqli_query($db, $sql) or cc("ERROR: SQL Select a2a ancestor", $sql, mysqli_error($db) , $_SESSION["u_id"], $this_document);
while ($items = mysqli_fetch_assoc($res))
{
if ($ancestor_item_array[$parent] == $items["item_id"])
$menu = build_ancestor_array($parent, $menu);
$ancestor_item_array[$parent] = $items["item_id"];
// Creates entry into items array with current menu item id ie. $menu['items'][1]
$menu['items'][$items['item_id']] = $items;
$menu['items'][$items['item_id']]["connection_type"] = 2;
// Creates entry into connected_to array. connected_to array contains a list of all items with connected_to
$menu['connected_to'][$items['item_parent_id']][] = $items['item_id'];
}
return $menu;
} // end build item array
It only goes one "level" down.
Refer, the 2 links below, I had recently posted answers for these, this is done in pure SQL
Recursive MySQL Query with relational innoDB
and
How to find all child rows in MySQL?
Recursive worked.. Just needed to try manually with pen and paper ;-)
function get_item_data($parent, $menu, $ancestor_item_array = "")
{
GLOBAL $db;
$sql = "
SELECT
items.*,
customers.*
FROM
items
LEFT JOIN item_customer_rel USING(item_id)
LEFT JOIN customers USING(customer_id)
WHERE
items.item_parent_id = '".$parent."'
ORDER BY
items.item_name
";
$res = mysqli_query($db, $sql) or cc("ERROR: SQL Select a2a ancestor", $sql, mysqli_error($db) , $_SESSION["u_id"], $this_document);
while ($items = mysqli_fetch_assoc($res))
{
$ancestor_item_array[] = $items["item_id"];
if (!in_array($items["item_parent_id"], $ancestor_item_array))
$menu = get_item_data($items["item_id"], $menu, $ancestor_item_array);
// Creates entry into items array with current menu item id ie. $menu['items'][1]
$menu['items'][$items['item_id']] = $items;
$menu['items'][$items['item_id']]["connection_type"] = 2;
// Creates entry into connected_to array. connected_to array contains a list of all items with connected_to
$menu['connected_to'][$items['item_parent_id']][] = $items['item_id'];
}
}
It wont work on pure SQL.
You should take a look at stored procedures, the sql you're trying to make will only go 'inwards' one level because all the relations will be shown as if they're first level connections.
for example.
parent->son->grandson->ggson
parent.item_parent_id = null
son.item_parent_id = parent
grandson.item_parent_id = son
ggson.item_parent_id = grandson
even tough grandson is a lower level connection, he will show up as a first level connection.
it cant be done with pure sql, sadly..
that's one of the reasons that made me go to NOSQL databases.

Help construct a simple query Using 3 tables

Hey guys need some more help
I have 3 tables USERS, PROFILEINTERESTS and INTERESTS
profile interests has the two foreign keys which link users and interests, they are just done by ID.
I have this so far
$statement = "SELECT
InterestID
FROM
`ProfileInterests`
WHERE
userID = '$profile'";
Now I want it so that it selects from Interests where what it gets from that query is the result.
So say that gives out 3 numbers
1
3
4
I want it to search the Interests table where ID is = to those...I just don't know how to physically write it in PHP...
Please help.
Using a JOIN:
Best option if you need values from the PROFILEINTERESTS table.
SELECT DISTINCT i.*
FROM INTERESTS i
JOIN PROFILEINTERESTS pi ON pi.interests_id = i.interests_id
WHERE pi.userid = $profileid
Using EXISTS:
SELECT i.*
FROM INTERESTS i
WHERE EXISTS (SELECT NULL
FROM PROFILEINTERESTS pi
WHERE pi.interests_id = i.interests_id
AND pi.userid = $profileid)
Using IN:
SELECT i.*
FROM INTERESTS i
WHERE i.interests_id IN (SELECT pi.interests_id
FROM PROFILEINTERESTS pi
WHERE pi.userid = $profileid)
You are on the right track, lets say you execute the query above using this PHP code:
$statement = mysql_query("SELECT InterestID FROM `ProfileInterests`
WHERE userID = '$profile'");
Then you can use a PHP loop to dynamically generate an SQL statement that will pull the desired IDs from a second table. So, for example, continuing the code above:
$SQL = "";
while ($statementLoop = mysql_fetch_assoc($statement)) {
//Note the extra space on the end of the query
$SQL .= "`id` = '{$statementLoop['InterestID']}' OR ";
}
//Trim the " OR " off the end of the query
$SQL = rtrim($SQL, " OR ");
//Now run the dynamic SQL, using the query generated above
$query = mysql_query("SELECT * FROM `table2` WHERE {$SQL}")
I haven't tested the code, but it should work. So, this code will generate SQL like this:
SELECT * FROM `table2` WHERE `id` = '1' OR `id` = '3' OR `id` = '4'
Hope that helps,
spryno724
Most likely you want to join the tables
select
i.Name
from
ProfileInterests p
inner join
interests i
on
p.interestid = i.interestid
where
p.userid = 1

Categories