I would like to execute an SQL query like this:
SELECT a.*, (time - (SELECT time FROM activity
WHERE time < a.time ORDER BY time DESC LIMIT 1) ) AS czas
FROM activity a WHERE 1 LIMIT 6000
I have prepared the following code ZF:
$activity = new Application_Model_DbTable_Activity();
$subSql = $activity->select()
//->setIntegrityCheck(false)
->from(array('aw' => 'activity'), array('time'))
->where('time < a.time', '')
->order('time DESC')
->limit(1);
// main query
$sql = $activity->select()
//->setIntegrityCheck(false)
->from(array('a' => 'activity'), array('a.*', 'czas' => new Zend_Db_Expr('(a.time - (' . $subSql . '))')))
->where(1)
->limit(6000);
$this->rows = $bugs->fetchAll($sql);
Unfortunately, this solution does not work. What should I improve?
Sorry for my english
Related
I am trying to update many rows (100 000+) in my database but it's taking a while (over 10 mins and still not finished). I'm wondering if this is intended behavior or is there something wrong in my code. To prevent the database from hanging while performing the update I've been told to update one row at a time, not sure if this is how it should be implemented.
I am updating images in my song table to be null if those songs were played in my playlist table
private function updateBlogSongs ($blog_id) {
$db = Yii::app()->db;
$affectedRows = 0;
$sql = "SELECT *
FROM `firstdatabase`.song s
INNER JOIN `seconddatabase`.playlist p ON s.name LIKE p.song_name";
$dataReader = $db->createCommand($sql)->query(); // Rows from the song table that were played in the given blog
$row = $dataReader->read();
while ($row != false) {
$sql = "UPDATE `firstdatabase`.song s
SET s.image = NULL
WHERE s.song_id = " . $row['song_id'];
$affectedRows += $db->createCommand($sql)->execute();
$row = $dataReader->read();
}
return $affectedRows;
}
Edit: after reading The Dog's comment I made some changes:
With 500 000 rows in the song table it takes about 10 minutes if I increase my batchSize to 10000 (was taking 8 hours with the code above). At 250 at the batch size it's taking about 50 minutes. I chose 250 because the query takes about 1 second to run, and it's taking 10+ seconds to run at 10000 batch size (constraint is 1 second). I would like to make it faster but not sure what else to change
$batchSize = 250;
$lastSongID = 0;
$rowIndex = 0;
$affectedRows = 0;
$sql = "SELECT max(song_id) FROM `firstdatabase`.song";
$lastSongID = intval($db->createCommand($sql)->query()->read()['max(song_id)']);
echo($lastSongID . ' songs in table.' . PHP_EOL);
echo('Updating songs...' . PHP_EOL);
while($rowIndex <= $lastSongID) {
$startTime = microtime(true);
$sql = "UPDATE `firstdatabase`.song
SET image = NULL
WHERE song_id in (
SELECT song_id
FROM (
SELECT song_id, name
FROM `firstdatabase`.song
WHERE song_id > " . $rowIndex . "
LIMIT " . $batchSize . "
) s
INNER JOIN (
SELECT DISTINCT song_name
FROM `seconddatabase`.playlist
) p ON s.name LIKE p.song_name
ORDER BY s.song_id ASC
)";
$affectedRows += $db->createCommand($sql)->execute();
$rowIndex += $batchSize;
$endTime = microtime(true);
$elapsedTime = round($endTime - $startTime, 2);
}
This is really more a question for the SQL world instead of the PHP world but here's my recommendations:
Don't do this one row at a time in a while loop. Make a more complex update statement that can do it all in one database hit. Database commands are the slowest part of your php code, you want to limit the number of calls you do to the database.
When you are confident that you can get the operation done in one sql command, or even if you don't think it is possible then pull your code into a stored procedure in the database. Having complex sql queries as stored procedures can help a lot with maintaining your code.
Make sure you have indexes on your tables. You need to make sure your queries hit those indexes for best performance.
Here's an option for the single query:
update `firstdatabase`.song
set image = null
where song_id in (
select s.song_id
from `firstdatabase`.song s
INNER JOIN `seconddatabase`.playlist p
ON s.name LIKE p.song_name"
);
Obviously we don't have access to your database so you'll need to make changes where necessary but hopefully it can get you on the right track.
EDIT:
Try replacing your second code set with the following:
$lastSongID = 0;
$rowIndex = 0;
$affectedRows = 0;
$sql = "SELECT max(song_id) FROM `firstdatabase`.song";
$lastSongID = intval($db->createCommand($sql)->query()->read()['max(song_id)']);
echo($lastSongID . ' songs in table.' . PHP_EOL);
echo('Updating songs...' . PHP_EOL);
$startTime = microtime(true);
$sql = "
update `firstdatabase`.song
set image = null
where song_id in (
select s.song_id
from `firstdatabase`.song s
INNER JOIN `seconddatabase`.playlist p
ON s.name LIKE p.song_name"
)";
$affectedRows += $db->createCommand($sql)->execute();
$endTime = microtime(true);
$elapsedTime = round($endTime - $startTime, 2);
If it works, then let me know the time it takes to run, if it doesn't work, is it an issue with the SQL (again I can't see the tables so I'm guessing).
I have a query that uses PostgreSQL generate_series function but when it comes to large amounts of data, the query can be slow. An example of code the generates the query is below:
$yesterday = date('Y-m-d',(strtotime ( '-1 day' ) ));
$query = "
WITH interval_step AS (
SELECT gs::date AS interval_dt, random() AS r
FROM generate_series('$yesterday'::timestamp, '2015-01-01', '1 day') AS gs)
SELECT articles.article_id, article_title, article_excerpt, article_author, article_link, article_default_image, article_date_published, article_bias_avg, article_rating_avg
FROM development.articles JOIN interval_step ON articles.article_date_added::date=interval_step.interval_dt ";
if (isset($this -> registry -> get['category'])) {
$query .= "
JOIN development.feed_articles ON articles.article_id = feed_articles.article_id
JOIN development.rss_feeds ON feed_articles.rss_feed_id = rss_feeds.rss_feed_id
JOIN development.news_categories ON rss_feeds.news_category_id = news_categories.news_category_id
WHERE news_category_name = $1";
$params = array($category_name);
$query_name = 'browse_category';
}
$query .= " ORDER BY interval_step.interval_dt DESC, RANDOM() LIMIT 20;";
This series looks for only content that goes one day back and sorts the results in random order. My question is what are was that generate_series can be optimized to improve performance?
You don't need that generate_series at all. And do not concatenate query strings. Avoid it by making the parameter an empty string (or null) if it is not set:
if (!isset($this -> registry -> get['category']))
$category_name = '';
$query = "
select articles.article_id, article_title, article_excerpt, article_author, article_link, article_default_image, article_date_published, article_bias_avg, article_rating_avg
from
development.articles
inner join
development.feed_articles using (article_id)
inner join
development.rss_feeds using (rss_feed_id)
inner join
development.news_categories using (news_category_id)
where
(news_category_name = $1 or $1 = '')
and articles.article_date_added >= current_date - 1
order by
date_trunc('day', articles.article_date_added) desc,
random()
limit 20;
";
$params = array($category_name);
Passing $yesterday to the query is also not necessary as it can be done entirely in SQL.
If $category_name is empty it will return all categories:
(news_category_name = $1 or $1 = '')
Imho, try removing that random() in your order by statement. It probably has a much larger performance impact than you think. As things are it's probably ordering the entire set by interval_dt desc, random(), and then picking the top 20. Not advisable...
Try fetching e.g. 100 rows ordered by interval_dt desc instead, then shuffle them per the same logic, and pick 20 in your app. Or wrap the entire thing in a subquery limit 100, and re-order accordingly along the same lines.
I have a search function in a forum where the search results will be displayed 10 at the time. The user can then look at the next or previous 10 search results. The results show different topics where the searched words are to be found. Everything works like I want it to.
The issue is when I want the user to be able to click a result and end up on the right page of that topic. For instance post nr 14 in a certain topic must be viewed on page 2 ( using LIMIT 10,10 in the SQL query on the topic page). I send the LIMIT parameter as a $_GET in the link.
How can I retrieve the row number of each topic in the results out of the total numbers of that specific topic when ordering by the date it was posted? Everything is always displayed in that order. I would like to use $nr = $nr-1; //and then
$limit = floor($nr / 10) * 10;
on that number to be able to send the right LIMIT parameter with the link in the search result.
Here's the PDO used to get the search results:
$query = 'SELECT t.topic_id, t.topic_cat, t.topic_subject, p.post_content, p.post_id, UNIX_TIMESTAMP(p.post_date) AS post_date
FROM posts p
LEFT JOIN topics t ON p.post_topic = t.topic_id
WHERE p.post_content LIKE :search OR t.topic_subject LIKE :search ORDER BY UNIX_TIMESTAMP(p.post_date) DESC LIMIT :start, :size';
$statement = $db->prepare($query);
$statement->bindValue(':search', '%'.$search.'%');
$statement->bindValue(':start', $start2, PDO::PARAM_INT);
$statement->bindValue(':size', $pagesize, PDO::PARAM_INT);
$statement->execute();
while ($row = $statement->fetch(PDO::FETCH_ASSOC)) {
$array[$row['topic_subject']][] = $row['post_id'];
$nr = count($array[$row['topic_subject']]) - 1;
echo '<div class="search_result"><div class="search_topic"><a href="topic_view.php?id=' . $row['topic_id'] . '&cat_id=' . $row['topic_cat'] . '
&start=' . floor($nr / 10) * 10 . '#anchor_' . $row['post_id'] . '">' . $row['topic_subject'] . '</a><span style="float:right;color:#696969">'
. date("M d, Y",$row['post_date']) . '</span></div><div class="search_post">' . $row['post_content'] . '</div></div>';
}
} $statement->closeCursor();
It is the start parameter in the link that I somehow need to grab in the query so I don't have to do a new DB call for each post_id in the while loop.
Assuming, that you only know, the ID of the post, I would go like this:
SELECT
count(*)
FROM
`posts`
WHERE
date <= ( SELECT date FROM `posts` WHERE post_id = '$id' )
ORDER BY
date DESC;
This will give you the number of row this post is. After that, just do some php code like:
$start = floor( $nr / 10 ) * 10;
$end = ceil( $nr / 10 ) * 10;
For multiple IDs:
SELECT topic_id, (
SELECT
count(*)
FROM
`posts`
WHERE
date <= ( SELECT date FROM `posts` u1 WHERE u1.topic_id = u2.topic_id )
) AS row
FROM
`posts` u2
WHERE
topic_id IN ( '$id1', '$id2', '$id3' )
ORDER BY
date DESC;
$query = 'SELECT :start as start, t.topic_id, t.topic_cat, t.topic_subject,
p.post_content, p.post_id, UNIX_TIMESTAMP(p.post_date) AS post_date
FROM posts p
LEFT JOIN topics t ON p.post_topic = t.topic_id
WHERE p.post_content LIKE :search OR t.topic_subject LIKE :search
ORDER BY UNIX_TIMESTAMP(p.post_date) DESC LIMIT :start, :size';
then access through $start = $row['start']
$nr = count($array[$row['topic_subject']])+$start - 1;
I had a problem as you said. I think this links is useful for you.
I have done these successfully, I hope be useful you:
1. http://www.awcore.com/dev/1/3/Create-Awesome-PHPMYSQL-Pagination_en
2. http://www.9lessons.info/2009/09/pagination-with-jquery-mysql-and-php.html
I tried myself for another queries but this one is more complex for me as i am new to zend. Please help me i tried different ways but not worked.
Tour Id fetching from another query
$tourId = $row2 ['test_public_id'];
$query = select count(ms.test_public_id) as total_views, ms1.recent_views from test_stats
ms join (select count(test_stats.test_public_id) as recent_views
from test_stats where test_stats.test_public_id = '$tourId'
and test_stats.updated_on > DATE_SUB(CURDATE(), INTERVAL 7 DAY)) ms1
where ms.test_public_id ='$tourId'" ;
Something like that should work:
$subselect = $dbAdapther->select()->from(
array('test_stats' => 'test_stats'),
array(
'(COUNT(test_public_id)) AS recent_views'
)
)->where(
$dbAdapther->quoteInto('test_stats.test_public_id = ?', $tourId)
)->where(
'test_stats.updated_on > DATE_SUB(CURDATE(), INTERVAL 7 DAY)'
);
$select = $dbAdapther->select()->from(
array('ms' => 'test_stats'),
array(
'(COUNT(ms.test_public_id)) AS total_views' // COUNT should be in brackets to preevent Zend from interpreting it as a field name
)
)->join(
array('ms1' => $subselect),
'',
array(
'ms1.recent_views'
)
)->where(
$dbAdapther->quoteInto('ms.test_public_id = ?', $tourId)'
);
Although I'd have your query broken into two separate ones or, more precisely, write a universal "get number of views" query with a date as its parameter, and then I'd be calling it twice, with or without the date.
But if you still need to get those two figures in one go in a single row (i.e. you can't use UNION instead of your unnecessary JOIN), I'd recommend you to use the following code instead:
$select = $dbAdapther->select()->from(
array('ms' => 'test_stats'),
array(
'(COUNT(ms.test_public_id)) AS total_views',
'(
COUNT(
CASE
WHEN ms.updated_on > DATE_SUB(CURDATE(), INTERVAL 7 DAY)) THEN ms.test_public_id
ELSE NULL
END
)
) AS recent_views'
)
)->where(
$dbAdapther->quoteInto('ms.test_public_id = ?', $tourId)
);
I'm new too in Zend, but I've tried this sample and it's works.
See this tutorial, I hope it'll help you:
http://framework.zend.com/manual/en/zend.db.select.html
or you can do this:
$db = Zend_Db_Table_Abstract::getDefaultAdapter();
$stmt = $db->query($query);
$result = $stmt->fetchAll();
select a.id, b.title, b.start_time, b.end_time from tv_channel a
left join tv_program b on a.id = b.tv_channel_id and b.start_time >= ‘2011-09-23 12:00:00′ and b.end_time <= '2011-09-23 14:30:00'
order by a.code
limit 0, 10;
–pager object
tnx
What's the question ? Do you want to write this SQL query in Propel ?
<?php
TvChannelQuery::create('a')
->joinTvProgram('b')
->addJoinCondition('b', 'b.StartTime >= 2011-09-23 12:00:00')
->addJoinCondition('b', 'b.EndTime >= 2011-09-23 14:30:00')
->orderByCode()
->limit(10)
;
Something like that should work but be be careful about values passed in addJoinCondition, there is no binding here and if you want to change these values you have to use it to prevent SQL injections or other security issues.
$c = new Criteria();
$c->addLeftJoin(tv_channel.id = tv_program .channel_id);
$c->add(tv_program.start_time, '2011-09-23 12:00:00', Criteria::GREATER_EQUAL);
$c->add(tv_program.end_time, '2011-09-23 14:30:00', Criteria::LESS_EQUAL);
$c->addAscendingOrderByColumn(tv_channel.code);
$c->setLimit(10);
$rs = DoSelect($c); //actual select execution here...