I already read some article but I am still confused.
I n pagination it will execute a query when load a page but what happen in chunk ?
I read https://laravel-school.com/posts/laravel-pagination-vs-chunk-51 this article and so on.
Now pagination works on a per page basis. So if you for example have 100 records and you retrieve 10 records per page you need to perform 10 queries to get all results. However since it's pagination you have a page number you use to determine which records you will return. For example page 3 will return the records 21-30 and so on. Here you will always only load 10 records in memory and return them. Query wise this works with a limit and offset to determine the correct records.
Now looking at chunk it's actually a bit different. So if we still take the 100 records and we chunk them by 10 records we still parse all the 100 records. However it will only load 10 records in memory at a time. So it's using a similar query approach using limit and offset, however it will eventually return all results, but in batches/chunks instead of only return 1 set of records.
The documentation also does a good job of explaining this: https://laravel.com/docs/9.x/eloquent#chunking-results
Related
I’m stuck in silly thing but don’t know how to do it, whenever I want to get the average of a column without retrieving all the data from de database I just run the following code:
Result::where('test_id', $test->id)->avg('grade')
The answer for a particular example is 37, now when I want to get the first 10 rows, I will run this:
Result::select('grade')->where('test_id', $test->id)
->orderBy('grade', 'asc')->limit(10)->get();
This off course will give me the first 10 rows, and if I use a calculator and get the average of the grades I will get 33, but if I run the following I:
Result::where('test_id', $test->id)->orderBy('grade','asc')->limit(10)->avg('grade');
I get the same 37 as with the total dataset. I'm pretty sure it is a silly thing but I can’t figure out what it is.
I'm kind of guessing, because I'm not set up to test this right now, but what I'm thinking is, you're using the builder avg method, which does an SQL aggregate function, and the limit(10) is being applied after the aggregation has already happened and there's only one row anyway.
I think if you do avg after you get the limited results, then you'll be using the collection avg method instead, and averaging over the 10 rows the query returned.
Result::where('test_id', $test->id)
->orderBy('grade','asc')->limit(10)->get()->avg('grade');
I'm working on an application that uses a scroll load system of 20 or so results loading into a feed at a time as you scroll. This feed consists of constantly added user generated content. Which means the result set can change per query that is offset by X.
So let's say we load 20 results, then scroll, another 20, and then before scrolling more to load the next 20, another user has uploaded a new piece of content, which effectively would present a duplicate in the next set of 20 results for the feed because we're using OFFSET to get additional results, and the total result set is getting shifted by 1 with this addition of new content that falls into the conditions of the query.
What is the best and most efficient way around this? We've dabbled with using the id of a row in a where condition to prevent duplicate results, and only using limit without offset for new results fetched.. so we could do WHERE id < 170 LIMIT 20, WHERE id < 150 LIMIT 20, WHERE id < 130 LIMIT 20, etc.. to control and prevent duplciates... HOWEVER, this does not work in every possible scenario as our result sets aren't always ordered with the id column ordered by DESC..
Soo.. what other options are there?..
Why are you using the where clause instead of limit with the offset option? Limit can take two arguments. The offset argument seems to do exactly what you want. For instance:
limit 100, 20
Takes 20 rows starting at the 101st row. Then:
limit 120, 20
Takes 20 rows starting at the 121st row. (The offsets start at 0 rather than 1 in MySQL counting.)
The one enhancement that you need to make is to ensure that the sort order for the records is stable. A stable sort is, essentially, one where there are no sort keys with the same value. To make this happen, just make the id column the last column in the sort. It is unique, so if there are any duplicate sort keys, then the addition of the id makes the sort stable.
You might want to try a database solution. On the initial request, create and populate a table. Use that table for the feed.
Make sure the tablename starts with something consistent, like TableToFeedMyApp, and ends with something guaranteed to make it unique. Then set up a scheduled job to find and drop all these tables that were created earlier than whatever you deem to be a certain interval.
I'm running a contest where I count up all the daily comments on a webpage per unique commenter each day. At the end of the month, the person with the most comment-votes wins. I have logic written for this that worked perfectly ... until today.
When a page has more than 900 or so comments, the vote-counter stops going up. I figured this had to do with pagination, so I altered my FQL query to use LIMIT and OFFSET to parse 100 comments at a time and combine the results for my vote-counting function, but when running my queries (even by hand, putting the https://graph.facebook.com/comments url in the address bar) it craps out somewhere between 500 and 600 comments, so the last 400 or so comments on the webpage never even return!
Since this method is failing is there a more reliable way to get the comment data?
UPDATE: I'm supplying some code, but I've found the same problems persist in FQL as in the Graph API. Here's my graph API call:
https://graph.facebook.com/comments/?ids=http://www.burlesquebitch.com/article.php?id=538&limit=500
If you crank the limit to over ~600 where more than 900 comments exist (or add an offset that would put the results past that range) it begins to fail. The earlier comments disappear. When I count up the results they're somewhere between 500-600.
https://graph.facebook.com/comments/?ids=http://www.burlesquebitch.com/article.php?id=538&limit=1000
Then I finally got fql working:
$fql_query_result = file_get_contents("https://graph.facebook.com/fql?q=SELECT+object_id,+id,+text,+time+,+fromid+FROM+comment+WHERE+object_id='366760226746431'+LIMIT+1000");
$data = json_decode($fql_query_result,true);
It also fails, but more like around 800-900. In any case the bad behavior is the same. I even did some experiments where I limited the query by time hoping that by isolated my results to a date range I would keep myself within it's failing threshold:
$fql_query_result = file_get_contents("https://graph.facebook.com/fql?q=SELECT+object_id,+id,+text,+time,+fromid+FROM+comment+WHERE+object_id='366760226746431'+AND+time>1365984000+AND+time<1366416000+LIMIT+500");
$data = json_decode($fql_query_result,true);
But no dice. If I flip the < to > in my time comparison the entire query fails and returns nothing. Idealy I would want to get the paging information, as suggested in the answer below, but I can't SELECT it in my query... if I try and get anything that ends in _cursor the query returns empty.
I am forced to conclude that whatever horrendous bug lives in the Graph API code its extended into the FQL code as well. I know of no other way to get data from a facebook object this size.
You should try out cursor pagination, it's recommended as explained at https://developers.facebook.com/docs/reference/api/pagination/
Returned results under cursor paging more consistently match the limit
requested, even after hiding any records for which you do not have
permissions to view (eg. if you request 10 records, but do not have
permissions to see 3 of those records, 3 additional records will be
pulled transparently, so that a full 10 records are pulled).
Example with post_id_cursor:
SELECT text, post_id, post_id_cursor FROM comment WHERE post_id='22707976849_10151395520781850' ORDER BY time DESC limit 50
You get the post_id_cursor of the last comment, then navigate next page with >post_id_cursor symbol
SELECT text, post_id, post_id_cursor FROM comment WHERE post_id='22707976849_10151395520781850' AND post_id_cursor>'Mjg3NA==' ORDER BY time DESC limit 50
Example with object_id_cursor is same:
SELECT text, post_id, object_id_cursor FROM comment WHERE object_id='10151395520696850' ORDER BY time DESC limit 50
SELECT text, post_id, time, object_id_cursor FROM comment WHERE object_id='10151395520696850' AND object_id_cursor>'Mjg3NA==' ORDER BY time DESC limit 50
Make sure you enabled "July 2013 Breaking Changes:" field at your app advanced settings,
https://developers.facebook.com/apps/YOUR_APP_ID/advanced. More info at https://developers.facebook.com/roadmap
I am trying to implement the pagination in php. I am using the Mysql as back end database. I am trying to implement the pagination logic.
I would be having lots of record. But the user will see only 10 at a time.
Now to show the first page, i do a
SELECT * from USERS LIMIT 10.
Now to get the next 10 and the subsequent 10 records i am not able to write a query. Please help me fetch the in between records to support pagination logic. Also provide if any other suggestions for pagination.
You should use the OFFSET option.
SELECT * FROM Users LIMIT 10 OFFSET 10 (or 20, or 30);
That way you just pass the start position in the request when you hit next (or the page number) and you'll retrieve the records you want.
MySQL's limit feature can take two arguments:
select * from USERS limit 10,10
The above would retrieve 10 rows starting at row 10. Bear in mind that the MySQL row offset is 0 based, not 1. The first argument is the starting row, the second is the page size.
Also, if your page size is consistent, all you need to do is pass in the current page (default to zero). That would then allow you to specify the start row as a page * size.
I mean what the most efficient way to get information about the quantity of your page's items and make sql query with LIMIT that you need. or I should get all items and then crop array with php functions?
now I do 2 queries: first to count all items and second to get items that I need with LIMIT.
OK, I'll be more concrete. For example I need to show a question on my page and 20 answers to this question. At the bottom there shold be page control: links to the next, prev page and so on. I want to show proper number of links (number of answers/20) and when I go to any link I want to recieve proper answers (for example 41 to 60 on the 3d page). So what's the best way to get number of items (answers) to show proper number of links and to get proper answers for each link?
I guess your'e trying to say you want to know how many items/answers there is in the query but only read up to 20 items at at time, for pagination.
Firstly: You really should look for a pagination package; lots and lots of people have had the same problem before and there probably exists both free/opensource and proprietary solutions for your programming language and framework. (If you say what language you are using I'm sure someone can reccomend a solution for you.)
Anyway, I know I like to know how things work, so this is how it usually does:
As far as I know the pagination code calculates the pages by doing one query using select count(*) from tblX where something divide this number with the items-per-page number and use ceiling (e.g. 4.1 => 5).
For listing the results per page a new query is required; don't worry the count query is terribly much faster than getting every result discarding the ones you don't need DO NOT DO THAT (that's the recipie for becoming the top story on this page). Something like select * from tblX where something limit Y offset Z where Y is the number of items per page, and Z is the the (requested_page - 1)*Y; page 1 will have offset 0, page 2 have offset 20 (if thats what Y are) etc..
But do not try to implement this manually, it's unneccesary, tedious and error prone, much better to use your time customizing a readymade solution.
I'm assuming you want a count of the number of rows you'll be reading so as to do some pagination or similar? I don't understand your need for the LIMIT in the context of your question. However, if you just want a count of how many rows have been found, use one of the following.
You select the count of all rows such as:
select count(*) as counted, name, address
from contact
Or found rows:
SELECT SQL_CALC_FOUND_ROWS, name, address
from contact
This may be mysql specific I'm not sure.
Update:
For pagination you would do something like the following - (Psuedocode)
$rows = array($result)
$num_rows = sql_calc_found_rows
$per_page = 20
$pages = ceil($num_rows / $per_page)
page
$rows_this_page = array()
$rows_this_page = get_values($rows, (min index)$page_number * $per_page - $per_page, (max index)$page_number * $per_page - 1)