In a database there are 2 tables - 10 records in each of them;
I want to retrieve all the data, so there will be 20 records in a response;
On each page should be only 5 records listed, so there will be 4 pages at the bottom to switch between;
When I retrieve data from database I must query each table in a separate query.
And there is a problem, because when I set limits and offset afted choosing a page number 4 there won't be records to retrieve from each tables, and each query will give me no data, because limit is 5 and offset is 15 then, but in each tables there is only 10 records;
Is it possible to solve such a problem? I'm not searching for implementation, only for text explanation;
I don't use any framework, just simple mysqli_query during db connection (it's a very old project)
Thanks for advices!
Its possible with UNION but that problem represent very bad database architecture and you should change it (merge into one table) before bigger problems comes
Have you tried using UNION and split each table query into it's own query?
Example:
([query_from_first_table]) UNION ([query_from_second_table]) LIMIT 5;
Related
I have a very large database (~10 million rows) and I want to list these things as fast as possible in a table. I have few options :
I can limit the rows from Mysql - Not Preferred as I want to count the rows with specific type of data say attachment
Fetch all rows and use while loop to limit 1000 records each time - I think it's good to do but calling 10 million rows in memory looks insane and I am quite sure that it must have worse performance.
Count the total data and then list using limit - but mysql count is a deal breaker as inspite of unique and indexed id I have faced bad time with mysql count.
What is the best way to do this?
If I just want to list 10 million rows and parsing data using php to stop it and display each time 1000 rows it is a bad idea ?
Theres some things to consider:
Is the database optimized? if yes, skip
Indexing columns you want to filter the search from
Select the columns you require from it only (instead of select *)
If you want to count the total and the id is sequencial, you get select the latest row and count based on the id if it's 'that slow'
If you're looking at some sort of pagination, you can count the rows and select only a few records based of an user input (select with limit 1000, skip '1000' when its page 2, etc)
You wouldn't want 10million data in your "memory" when you'd be using 0.1% of it right?
Hi, I am creating a simple news website and I need to count news views. Currently I have 25000 rows and 25 columns. The hits count increases per page reload like Joomla. How should I structure the tables?
I have 2 approaches to this issue:
Create column named hits in the content table.
Create a new table that has 2 columns: content_id and hits.
I used the first approach and I think that slows my site.Will the second approaches perform better than the first one? Is there a better approach?
Well I don't know what's your logic in MySQL or PHP or what is you current table structure for news but I would suggest you to use Stored Procedure in MySQL as
Begin
update tblnews set hits = hits+1;
select news from tblnews;
End
and off course use PHP PDO Prepared Statement for performance
and if you are trying to get last 10 news or something like that then must set Indexing for content_id say like Primary key with auto_increment for better retrieval of query otherwise don't even use content_id column. I don't think there should be any hard structre for table. This would definitely increase performance and more than 100000000 rows would not make any big difference I hope. I don't think there would be any other better solution because these 2 queries needs to be performed at every page view.
Option 1 sounds the best. Option 2 is redundant because your just storing the hits. Again you would a join query to pull the hits
I'm sorry, I am a newbie in PHP. Now I want to learn how to write a pagination effect in PHP. I know there are many tutorials for this on the net. I searched it and have a reference to them. When I try it by myself and don't have a reference to other resources, though, I still don't know how to write a pagination effect with PHP.
Now, I want to know what's the basic and important thing of writing a pagination in PHP. If anyone could give me more details, thank you.
For this example, I will use MySQL because it's a popular database to use with PHP. The general principle applies to other databases, but the precise way you use SQL to achieve the results is different.
The essence of pagination is that you have a number records, say 105, and you only want to display a smaller, more manageable number of records at a time, say 10 of them. In order to do this, you need to know how to find out the total number of records, and you need to know how to select just a subset of the records.
First, find out how many records you have. For this, you use COUNT(*) to count the records in the table. If your table is called Users:
SELECT COUNT(*) FROM Users;
Since we said there are 105 records in the table, this will return the result 105. We said each page has 10 records, so figure out how many pages there are in total: 105 / 10 = 10.5; round up to 11 because the extra records after page 10 need their own page. So, there are 11 pages in total. Your web page will display controls that enable the user to select a page from 1 to 11.
Instead of fetching all the records, you will only fetch one page at a time. In MySQL you generally do this using the LIMIT keyword; this allows you to choose a range of records. LIMIT syntax works like this: LIMIT $SKIP, $COUNT, where $SKIP is the number of records in the result set to skip, and $COUNT is the number to return. (Actually, you will get up to $COUNT records, in case fewer are available.)
The user has requested page 6. This means that I need to skip 5 pages before the results I want to show. With pages of 10 records each, this means I will skip 5 * 10 records, i.e. 50 records. In other words, $SKIP = ($PAGE_NUMBER - 1) * $PAGE_SIZE. You query would look like this:
SELECT User_ID, UserName, City, State from Users LIMIT 50, 10;
This gives you records 51-60 of the table.
See the documentation to learn more about how LIMIT works:
http://dev.mysql.com/doc/refman/5.5/en/select.html
Now you have a single page of records. However, the numbering of the records is dependent on their ordering. The query above will give you unpredictable results because the ordering is undefined. (A database can return records in any order it chooses unless you command it to use a certain ordering.) Thus, to do pagination, you need to define the order of the records. Use ORDER BY to define the ordering:
SELECT User_ID, UserName, City, State from Users ORDER BY UserName LIMIT 50, 10;
This allows you to show different pages of results, and always know that when you show page 6, it contains the records that come after the ones on page 5 and before the ones on page 7.
(If your records don't have any natural ordering, you will have to define an arbitrary one to make this work correctly.)
I have avoided mentioning PHP at all in this answer because I don't think it's necessary to do so to explain pagination. Pagination is entirely a matter of handling data. If you understand this, I think you can go read the tutorials and they will make sense to you and then you can figure how to write the PHP to do this.
Get page number
Know how many items to display on each page.
Using mysql command "LIMIT", get the offset and select how many rows you want.
Want to use a class? http://net.tutsplus.com/tutorials/php/how-to-paginate-data-with-php/
Or just a plain script to modify? http://pastebin.com/J8nTk1q5
If you are new to PHP, then you need to learn more PHP or risk future confusion.
I'm running a sql query to get basic details from a number of tables. Sorted by the last update date field. Its terribly tricky and I'm thinking if there is an alternate to using the UNION clause instead...I'm working in PHP MYSQL.
Actually I have a few tables containing news, articles, photos, events etc and need to collect all of them in one query to show a simple - whats newly added on the website kind of thing.
Maybe do it in PHP rather than MySQL - if you want the latest n items, then fetch the latest n of each of your news items, articles, photos and events, and sort in PHP (you'll need the last n of each obviously, and you'll then trim the dataset in PHP). This is probably easier than combining those with UNION given they're likely to have lots of data items which are different.
I'm not aware of an alternative to UNION that does what you want, and hopefully those fetches won't be too expensive. It would definitely be wise to profile this though.
If you use Join in your query you can select datas from differents tables who are related with foreign keys.
You can look of this from another angle: do you need absolutely updated information? (the moment someone enters new information it should appear)
If not, you can have a table holding the results of the query in the format you need (serving as cache), and update this table every 5 minutes or so. Then your query problem becomes trivial, as you can have the updates run as several updates in the background.
What are some of the strategies being used for pagination of data sets that involve complex queries? count(*) takes ~1.5 sec so we don't want to hit the DB for every page view. Currently there are ~45k rows returned by this query.
Here are some of the approaches I've considered:
Cache the row count and update it every X minutes
Limit (and offset) the rows counted to 41 (for example) and display the page picker as "1 2 3 4 ..."; then recompute if anyone actually goes to page 4 and display "... 3 4 5 6 7 ..."
Get the row count once and store it in the user's session
Get rid of the page picker and just have a "Next Page" link
I've had to engineer a few pagination strategies using PHP and MySQL for a site that does over a million page views a day. I persued the strategy in stages:
Multi-column indexes I should have done this first before attempting a materialized view.
Generating a materialized view. I created a cron job that did a common denormalization of the document tables I was using. I would SELECT ... INTO OUTFILE ... and then create the new table, and rotate it in:
SELECT ... INTO OUTFILE '/tmp/ondeck.txt' FROM mytable ...;
CREATE TABLE ondeck_mytable LIKE mytable;
LOAD DATA INFILE '/tmp/ondeck.txt' INTO TABLE ondeck_mytable...;
DROP TABLE IF EXISTS dugout_mytable;
RENAME TABLE atbat_mytable TO dugout_mytable, ondeck_mytable TO atbat_mytable;
This kept the lock time on the write contended mytable down to a minimum and the pagination queries could hammer away on the atbat materialized view. I've simplified the above, leaving out the actual manipulation, which are unimportant.
Memcache I then created a wrapper about my database connection to cache these paginated results into memcache. This was a huge performance win. However, it was still not good enough.
Batch generation I wrote a PHP daemon and extracted the pagination logic into it. It would detect changes mytable and periodically regenerate the from the oldest changed record to the most recent record all the pages to the webserver's filesystem. With a bit of mod_rewrite, I could check to see if the page existed on disk, and serve it up. This also allowed me to take effective advantage of reverse proxying by letting Apache detect If-Modified-Since headers, and respond with 304 response codes. (Obviously, I removed any option of allowing users to select the number of results per page, an unimportant feature.)
Updated:
RE count(*): When using MyISAM tables, COUNT didn't create a problem when I was able to reduce the amount of read-write contention on the table. If I were doing InnoDB, I would create a trigger that updated an adjacent table with the row count. That trigger would just +1 or -1 depending on INSERT or DELETE statements.
RE page-pickers (thumbwheels) When I moved to agressive query caching, thumb wheel queries were also cached, and when it came to batch generating the pages, I was using temporary tables--so computing the thumbwheel was no problem. A lot of thumbwheel calculation simplified because it became a predictable filesystem pattern that actually only needed the largest page numer. The smallest page number was always 1.
Windowed thumbweel The example you give above for a windowed thumbwheel (<< 4 [5] 6 >>) should be pretty easy to do without any queries at all so long as you know your maximum number of pages.
My suggestion is ask MySQL for 1 row more than you need in each query, and decide based on the number of rows in the result set whether or not to show the next page-link.
MySQL has a specific mechanism to compute an approximated count of a result set without the LIMIT clause: FOUND_ROWS().
MySQL is quite good in optimizing LIMIT queries.
That means it picks appropriate join buffer, filesort buffer etc just enough to satisfy LIMIT clause.
Also note that with 45k rows you probably don't need exact count. Approximate counts can be figured out using separate queries on the indexed fields. Say, this query:
SELECT COUNT(*)
FROM mytable
WHERE col1 = :myvalue
AND col2 = :othervalue
can be approximated by this one:
SELECT COUNT(*) *
(
SELECT COUNT(*)
FROM mytable
) / 1000
FROM (
SELECT 1
FROM mytable
WHERE col1 = :myvalue
AND col2 = :othervalue
LIMIT 1000
)
, which is much more efficient in MyISAM.
If you give an example of your complex query, probably I can say something more definite on how to improve its pagination.
I'm by no means a MySQL expert, but perhaps giving up the COUNT(*) and going ahead with COUNT(id)?