MySQL Union query taking too much time to execute - php

I'm trying to get a single MySQL query result from different queries working on different table and databases, this is my table structure with database names:
database ads1-
tables: clicks_214,requests
database ads2-
tables: clicks_255,requests
database ads3- tables: clicks_256,requests
and this is the query which I'm trying to execute
$query="";
for($i=1;$i<4;++$i)
{
$this->db_num=$i;
$this->selectDB($i);
$table=$this->getTableName();
$db=$this->getDatabaseName();
$query.="(SELECT r.kwd, count(c.rid) as cnum, count(r.kwd) as rnum
FROM $db.requests r LEFT JOIN $db.$table c ON r.id=c.rid
WHERE hid='$hid'
AND r.time BETWEEN '$date1 $hour1:00:00' AND '$date2 $hour2:59:59'
GROUP BY r.kwd
ORDER BY cnum DESC
LIMIT $limit,50)";
if($i<3)
{
$query.=" UNION ";
}
}
I'm certainly sure that this isn't the best way to do it, just because I have to wait about 5 minutes to get results. Is there any way to do this much faster? I've already set indexes on all of the 3 DBs
this is an EXPLAIN result:

a) You SHOULD NOT make such crossdatabase queries. With your DB structure it would be more appropriate to run 3 independent queries and combine the result using some simple script Or to make a temporary table in one of the databases where you can combine data from every one of those
b) On large databases put some indexes to improve the speed. It is not the solution though...
You can try to play with Temp tables:
$this->selectDB('temptableDB');
$maketemp = "
CREATE TEMPORARY TABLE temp_table_1 (
`kwd` int ,
`rid` int,
)
";
mysql_query($maketemp, $connection) or die ("Sql error : ".mysql_error());
for($i=1;$i<4;++$i)
{
$this->db_num=$i;
$this->selectDB($i);
$table=$this->getTableName();
$db=$this->getDatabaseName();
$inserttemp = "
INSERT INTO temptableDB.temp_table_1
(`kwd`, `rid`)
SELECT r.kwd, c.rid
FROM $db.requests r LEFT JOIN $db.$table c ON r.id=c.rid
WHERE hid='$hid'
AND r.time BETWEEN '$date1 $hour1:00:00' AND '$date2 $hour2:59:59'
";
mysql_query($inserttemp, $connection) or die ("Sql error : ".mysql_error());
}
$select = "
SELECT kwd, count(rid) as cnum, count(kwd) as rnum
FROM temp_table_1
GROUP BY r.kwd
ORDER BY cnum DESC
";
$export = mysql_query($select, $connection) or die ("Sql error : ".mysql_error());

Related

How do I connect two tables to see if a value exist in any of the two

I having problems connecting the below query together so that It works more efficient.Can someone please tell me how I can connect these two queries so that it is only one?
$rs_duplicate = mysql_query("select count(*) as total
from advertisers_account
where user_email='$user_email' ") or die(mysql_error());
list($total) = mysql_fetch_row($rs_duplicate);
}
$rs_duplicate_pub = mysql_query("select count(*) as total
from publishers_account
where user_email='$user_email' ") or die(mysql_error());
list($totalpub) = mysql_fetch_row($rs_duplicate_pub);
if ($totalpub || $total > 0)
{
echo "Not Available ";
} else {
echo "Available";
}
Use a UNION:
SELECT 'advertisers' AS which, count(*) AS total
FROM advertisers_account
WHERE user_email = '$user_email'
UNION
SELECT 'publishers' AS which, count(*) AS total
FROM publishers_account
WHERE user_email = '$user_email'
This query will return two rows, you can use the which column to tell whether it's advertisers or publishers.
This is how you could do it. you need to use joins, but you should make sure to not let any variables in the query be directly from an outside user like from a form submit. That will open you up to SQL Injection. Use Prepared Statements instead.
select count(*) as total from publishers_account INNER JOIN advertisers_account ON advertisers_account.user_email = publishers_account.user_email WHERE user_email='$user_email'
in response to:
Can someone please tell me how I can connect these two queries so that it is only one?
Why not:
Select
(select count(*) as total from advertisers_account where user_email='$user_email') +
(select count(*) as total from publishers_account where user_email='$user_email') as sumofCount
SELECT count(advertisers_account.id)
FROM publishers_account
LEFT JOIN advertisers_account ON publisher_account.email = advertisers_account.email
WHERE publisher_account.email = '$user_email';
If the count is greater than zero, then the email exists in both tables at least once. If it exists only in the left table (publishers), then the counter would be zero. If it doesn't exist at all in the left table, then you'll get no rows at all, even if it does exist in the right table (advertisers)

In a loop, some queries succeed, other queries fail 'No database selected'

I apply the same 2 queries to 3 tables, so just loop round and substitute the table names. The first INSERT query always works on test servers and on production. The second DELETE query works on two different test servers but is failing on production and throwing mysql error 1046 No database selected.
Affected code extract:
$queries = array(
"INSERT INTO knd_bkg.#table#
SELECT DISTINCT SQL_NO_CACHE
B.ASIN, Author, Title, EditorialReview, DetailPageURL, current_price, salesrank, number_reviews, erotica, G.BrowseNodeId, price_salesrank_checked_on
FROM bkd_books.club_books B
USE INDEX (free_quality)
RIGHT JOIN bkd_books.book_genre G on G.ASIN=B.ASIN
WHERE EXISTS
(
SELECT 1 FROM knd_bkg.genres K WHERE K.browsenode=G.BrowseNodeId
)
AND #priceWhere#
AND price_salesrank_checked_on > '#newerThan#'
AND B.number_reviews >= 4
AND B.rating >= 4.0
AND B.language = 'english'
AND B.hide_knd=0
AND B.public_domain=0;",
"DELETE T FROM knd_bkg.#table# T
INNER JOIN
(
SELECT N.* FROM knd_bkg.#table# N
INNER JOIN
(
SELECT DISTINCT ASIN, BrowseNodeId FROM knd_bkg.#table#
INNER JOIN knd_bkg.genres ON knd_bkg.genres.browsenode=knd_bkg.#table#.BrowseNodeId
WHERE isFiction=1
) F
ON F.ASIN=N.ASIN
INNER JOIN knd_bkg.genres
ON knd_bkg.genres.browsenode=N.BrowseNodeId
WHERE knd_bkg.genres.isNonFiction=1
) D
USING (ASIN, BrowseNodeId)
WHERE D.ASIN=T.ASIN AND D.BrowseNodeId=T.BrowseNodeId"
);
$tables = array(
"free_books" => array('priceWhere' => "B.current_price = 0", 'timePeriod' => $freeTimePeriod),
"99_books" => array('priceWhere' => "B.current_price > 0 AND B.current_price < 100", 'timePeriod' => $paidTimePeriod),
"399_books" => array('priceWhere' => "B.current_price >= 100 AND B.current_price < 400", 'timePeriod' => $paidTimePeriod),
);
connectSlaveDB();
foreach ($tables as $table => $data) {
$newerThan = date("Y-m-d H:i:s", strtotime("-" . $data['timePeriod'] . " hours"));
foreach ($queries as $query) {
$query = str_replace('#table#', $table, $query);
$query = str_replace('#newerThan#', $newerThan, $query);
$query = str_replace('#priceWhere#', $data['priceWhere'], $query);
Logger::_write(LOG_VERBOSE, "API - api/bkg/rebuild query: $query");
$result = mysql_query($query);
if (!$result) {
Logger::_write(LOG_ERR, "API - api/bkg/rebuild FAILED: " . mysql_error());
$success = false;
} else {
Logger::_write(LOG_VERBOSE, "API - api/bkg/rebuild SUCCESS");
}
}
}
And connectSlaveDB()
function connectSlaveDB()
{
global $dbSlave, $dbUser, $dbPass, $dbName;
$connSlave = mysql_connect($dbSlave, $dbUser, $dbPass, TRUE);
if (!$connSlave) {
Logger::_write(LOG_CRIT, "Unable to connect to slave DB");
include 'screens/error.inc.php';
exit();
}
mysql_set_charset('utf8', $connSlave);
mysql_select_db($dbName, $connSlave);
return TRUE;
}
The tables are MEMORY tables, but I get the same error with MYISAM, and only on production. This query is running on a replication slave and does not involve the master, but previous code TRUNCATES the tables so I don't believe it is down to permissions. When the query is taken out of the log file, it works fine in phpMyAdmin. And it's fine on the test servers.
I don't get the reason for the 1046 No database selected error. And after 3 hours I might be missing something really obvious. Especially as the failure pattern is like this, for one DB connection:
INSERT on free_books - success
DELETE on free_books - fail
INSERT on 99_books - success
DELETE on 99_books - fail
INSERT on 399_books - success
DELETE on 399_books - fail
Please excuse ugly globals and deprecated mysql* functions. This is a legacy app that is rather out of date. I know...
The syntax of your DELETE query may be causing the problem
DELETE T FROM
I know this has caused issues with UPDATE queries sometimes before due to a mysql bug.
Re-writing your query to avoid the T reference should resolve this issue.
Looks like the T alias in the DELETE was causing problems, but I'm not quiet sure why. The problem is now fixed by making the DELETE query to be:
"DELETE knd_bkg.#table# FROM knd_bkg.#table#
INNER JOIN
(
SELECT N.* FROM knd_bkg.#table# N
INNER JOIN
(
SELECT DISTINCT ASIN, BrowseNodeId FROM knd_bkg.#table#
INNER JOIN knd_bkg.genres ON knd_bkg.genres.browsenode=knd_bkg.#table#.BrowseNodeId
WHERE isFiction=1
) F
ON F.ASIN=N.ASIN
INNER JOIN knd_bkg.genres
ON knd_bkg.genres.browsenode=N.BrowseNodeId
WHERE knd_bkg.genres.isNonFiction=1
) D
USING (ASIN, BrowseNodeId)
WHERE D.ASIN=knd_bkg.#table#.ASIN AND D.BrowseNodeId=knd_bkg.#table#.BrowseNodeId"

Paginate while ignoring sub rows, with a Right Join

I have this basic query that returns a main row of a program title(eventdesc) and then lists program participants and info as multiple sub rows.
PROGRAM 1
Email 1, email 2, name, status (first participant)
Email 1, email 2, name, status (second participant)
PROGRAM 2
Email 1, Email 2, name, status (first participant)
ETC...
Basic query:
$query_perpage = "SELECT COUNT(*) FROM camps_events";
$result = mysql_query($query_perpage, $db) or die(mysql_error());
$r = mysql_fetch_row($result);
$numrows = $r[0];
$pages = new Paginator($query);
$pages->items_total = $numrows;
$pages->paginate();
$pages->mid_range = 7;
$query = mysql_query("SELECT ce.eventcode, ce.eventdesc, ce.date, r.participant_fname, r.participant_lname,
r.position, r.dob, r.status, r.email_primary, r.order_number
FROM camps_events ce
RIGHT JOIN registrations r on r.eventcode = ce.eventcode
WHERE ce.reg_status = 'Active'
AND r.status NOT IN ('Incomplete','Canceled')".getAllowedPrograms()."
ORDER BY ce.eventdesc, ce.eventcode, r.participant_lname ASC $pages->limit ") or die(mysql_error());
I've been trying to figure out how to paginate it by Program, no matter how many subrows.
I have a pagination class in place on another page, but that only pulls from one table.
I've been trying to implement the same pagination, but I can only get it to pull the first program, and then it limits the page based on the sub rows, not the main program row. And in the state I've posted it here, each following page displays only the first programs. The sub rows that come up are different on each page, but I can't tell if they are all in the first program.
The query works fine for pulling all results properly, but if it's a huge list, it's extremely inefficient, and can even timeout the browser.
Any help would be much appreciated, please let me know if I can include anything else.
I'm not too concerned with using the same pagination class, so I'm completely open to different approaches.
Thanks in advance
Try using a subselect:
SELECT ce.eventcode, ce.eventdesc, ce.date, r.participant_fname, r.participant_lname,
r.position, r.dob, r.status, r.email_primary, r.order_number
FROM camps_events ce
RIGHT JOIN registrations r on r.eventcode = ce.eventcode
WHERE ce.reg_status = 'Active'
AND r.status NOT IN ('Incomplete','Canceled')".getAllowedPrograms()."
AND ce.id in (select id from camps_events pgce order by ce.eventdesc, ce.eventcode $pages->limit)
ORDER BY ce.eventdesc, ce.eventcode, r.participant_lname ASC ")
or die(mysql_error());
Edit: unfortunately MySQL can not yet limit subqueries. :-(
So it has to be a temp-table:
mysql_query("drop temporary table if exists tmp_events", $db);
mysql_query("create temporary table tmp_events select id from camps_events order by eventdesc, eventcode " . $pages->limit, $db);
mysql_query("SELECT ce.eventcode, ce.eventdesc, ce.date, r.participant_fname, r.participant_lname,
r.position, r.dob, r.status, r.email_primary, r.order_number
FROM camps_events ce
inner join tmp_events pgce on pgce.id = ce.id
RIGHT JOIN registrations r on r.eventcode = ce.eventcode
WHERE ce.reg_status = 'Active'
AND r.status NOT IN ('Incomplete','Canceled')".getAllowedPrograms()."
ORDER BY ce.eventdesc, ce.eventcode, r.participant_lname ASC ")
or die(mysql_error());
There can be of course pages with less than paged items, because events without registrations will not show.

MySQL query returns rows in mysql but empty set in PHP

The following MySQL query runs in PHP without errors, but the resultset is empty. Directly outputting the query string to a file and running the query in the MySQL client using 'source [filename]' returns several rows of results, as expected.
Is there something that would cause this query not to work with PHP? categorylinks.cl_to and smw_spec2.value_string are both varbinary(255). Show create table indicates engine=InnoDB and default charset=binary.
Things I have tried without success:
$sql = preg_replace("/[\n\t]+/", " ", $sql);
Changing '_wpg' and 'Derp' to CAST('_wpg' AS BINARY(255))
Changing '_wpg' and 'Derp' to BINARY '_wpg'
I am using the MediaWiki DatabaseMysql class to execute the query and fetch rows, but it's a very thin abstraction, and I'm certain it's not the problem (see below).
SELECT
prop.name AS prop_name, prop.count AS prop_count, prop.type AS prop_type,
val.value AS val_value, val.unit AS val_unit, val.count AS val_count
FROM
(
SELECT
s_id, name, type, COUNT(foo.name) AS count
FROM (
(
SELECT
cl.cl_to AS cat_name, s.smw_id AS s_id, s.smw_sortkey AS name, spec.value_string AS type
FROM `smw_ids` s
INNER JOIN (`categorylinks` cl, `page` p, `smw_ids` s2, `smw_atts2` a)
ON (cl.cl_from = p.page_id AND
p.page_title = s2.smw_title AND
s2.smw_id = a.s_id AND
a.p_id = s.smw_id)
LEFT JOIN `smw_spec2` spec ON s.smw_id = spec.s_id
)
UNION ALL
(
SELECT
cl.cl_to AS cat_name, s.smw_id AS s_id, s.smw_sortkey AS name, '_wpg' AS type
FROM `smw_ids` s
INNER JOIN (`categorylinks` cl, `page` p, `smw_ids` s2, `smw_rels2` a)
ON (cl.cl_from = p.page_id AND
p.page_title = s2.smw_title AND
s2.smw_id = a.s_id AND
a.p_id = s.smw_id)
)
) AS foo
WHERE foo.cat_name = 'Derp'
GROUP BY name
ORDER BY count DESC
LIMIT 10
) AS prop
INNER JOIN
(
SELECT
bar.p_id AS p_id, bar.value AS value, bar.unit AS unit, COUNT(bar.value) AS count,
IF( #prev != p_id, #rownum := 1, #rownum := #rownum+1 ) AS rank,
#prev := p_id
FROM (
(SELECT a.p_id AS p_id, a.value_xsd AS value, a.value_unit AS unit FROM `smw_atts2` a)
UNION ALL
(SELECT r.p_id AS p_id, s.smw_sortkey AS value, NULL AS unit
FROM `smw_rels2` r INNER JOIN `smw_ids` s ON r.o_id = s.smw_id)
) AS bar
GROUP BY value, unit
ORDER BY count DESC
) AS val
ON prop.s_id = val.p_id
WHERE val.rank <= 50
ORDER BY prop_count DESC, prop_name, val_count DESC, val_value
Edit: The following test script outputs nothing. query.sql contains exactly the query above, written to file immediately preceding the mysql_query() call in MediaWiki's database class.
$db = mysql_connect('localhost', 'root', '');
mysql_select_db('mediawiki', $db);
$res = mysql_query(file_get_contents("query.sql"), $db);
while ($row = mysql_fetch_assoc($res)) {
var_dump($row);
}
echo mysql_error($db);
Edit: I imported a huge database dump and afterwards, when I loaded the PHP page, there was a noticeable wait that seemed to indicate that the query was running, but still no results showed. I ended up reworking the query, and I no longer have this problem.
Try this to detect and report errors better:
$db = mysql_connect('localhost', 'root', '');
mysql_select_db('mediawiki', $db);
$res = mysql_query(file_get_contents("query.sql"), $db);
if (!$res) {
print "SQL Error ".mysql_errno().":".mysql_error().", from query: '".file_get_contents("query.sql")."'";
} else {
while ($row = mysql_fetch_assoc($res)) {
var_dump($row);
}
}
Try this after mysql_connect:
mysql_query('SET NAMES utf8;');

PHP vs Phpmyadmin

I've got this code which i execute on phpmyadmin which works 100%
Create Temporary Table Searches ( id int, dt datetime);
Create Temporary Table Searches1 ( id int, dt datetime, count int);
insert into Searches(id, dt) select a.id, now() from tblSavedSearches a;
insert into Searches1(id, dt, count)
select
b.savedSearchesId,
(select c.dt from tblSavedSearchesDetails c where b.savedSearchesId = c.savedSearchesId order by c.dt desc limit 1) as 'dt',
count(b.savedSearchesId) as 'cnt'
from tblSavedSearchesDetails b
group by b.savedSearchesId;
insert into tblSavedSearchResults(savedSearchId,DtSearched,isEnabled)
select id,now(),0 from Searches where not id in (select savedSearchId from tblSavedSearchResults);
update tblSavedSearchResults
inner join Searches1 on tblSavedSearchResults.savedSearchId = Searches1.id
Set tblSavedSearchResults.DtSearched = Searches1.dt, tblSavedSearchResults.isEnabled = 1;
However when i put the same code in php as below it generates an error
$dba = DbConnect::CreateDbaInstance();
$query = "";
$query.="Create Temporary Table Searches ( id int, dt datetime); ";
$query.="Create Temporary Table Searches1 ( id int, dt datetime, count int); ";
$query.="insert into Searches(id, dt) select a.id, now() from tblSavedSearches a; ";
$query.="insert into Searches1(id, dt, count) ";
$query.="select ";
$query.=" b.savedSearchesId, ";
$query.=" (select c.dt from tblSavedSearchesDetails c where b.savedSearchesId = c.savedSearchesId order by c.dt desc limit 1) as 'dt', ";
$query.=" count(b.savedSearchesId) as 'cnt' ";
$query.="from tblSavedSearchesDetails b ";
$query.="group by b.savedSearchesId; ";
$query.="insert into tblSavedSearchResults(savedSearchId,DtSearched,isEnabled) ";
$query.="select id,now(),0 from Searches where not id in (select savedSearchId from tblSavedSearchResults); ";
$query.="update tblSavedSearchResults ";
$query.="inner join Searches1 on tblSavedSearchResults.savedSearchId = Searches1.id ";
$query.="Set tblSavedSearchResults.DtSearched = Searches1.dt, tblSavedSearchResults.isEnabled = 1; ";
$dba->DbQuery($query) or die(mysql_error());
I get the following error
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'Create Temporary Table Searches1 ( id int, dt datetime, count int) insert into S' at line 1
Please if someone could assist me with this ...
Thanks
If your $dba->DbQuery($query) method is actually using mysql_query (Which I suppose it does, as you are using mysql_error), then, you cannot execute more than one query per call (quoting) :
mysql_query() sends a unique query (multiple queries are not supported) to the currently active
database on the server that's
associated with the specified
link_identifier.
You'll have to either :
separate your queries, and call mysql_query once for each query
should be quite easy, here : instead of concatening all queries into $query, just execute them one by one.
or stop using mysql_*, and start working with MySQLi, which provides a mysqli_multi_query function
You can only execute queries one at a time via PHP. Call $dba->DbQuery() once per new query instead.

Categories