I have wrote universal script to get all rows from Doctrine Table Models, but if rows amount is too large, i get exception:
Cannot define NULL as part of query when defining 'offset'.
Running script:
$table = new JV_Model_StoreOrder();
$this->data['list'] = $table->getTable()->findAll()->toArray();
I understand from the error above is due to the large number of entries in the table (> 20 000). So I decided to make a paginator to break records on the pages of 100 pieces.
Could you help me, how can I do something like that:
...
$total_amount = $table->getTable()->count();
$this->data['list'] = $table->getTable()->offset(0)->limit(100)
->findAll()->toArray();
...
Doctrine has it's own Pagination component:
http://www.doctrine-project.org/documentation/manual/1_0/ru/utilities:pagination
Example from Doctrine manual:
// Defining initial variables
$currentPage = 1;
$resultsPerPage = 50;
// Creating pager object
$pager = new Doctrine_Pager(
Doctrine_Query::create()
->from( 'User u' )
->leftJoin( 'u.Group g' )
->orderby( 'u.username ASC' ),
$currentPage, // Current page of request
$resultsPerPage // (Optional) Number of results per page. Default is 25
);
Related
So this is my case. I have a main table payment_transactions having almost 1 million records.
I am getting the data from this table with joins and where clauses and there is Laravel paginate method,
But this method takes too much time and after investigation, I found that its count method is taking time like 4 to 5 seconds just for counting.
So how can I optimize and speed up this query, especially is there any way to improve paginate method speed?
Note: I can't use simplePaginate because there is a datatable on frontend and I need a total count for that.
So for paginate, two queries run
1 is the main query and other one is for count, and I felt that, the count query is taking much time.
Here is the count query after getQueryLog
select count(*) as aggregate from `payment_transactions`
left join `users` as `U` on `U`.`id` = `payment_transactions`.`user_id`
left join `coupons` as `C`
on `C`.`id` = `payment_transactions`.`coupon_id`
where `payment_transactions`.`refund_status` = 'NO_REFUND'
and `payment_transactions`.`transaction_type`
in ('BOOKING','SB_ANPR','QUERCUS_ANPR','CANDID_ANPR','SB_TICKET',
'ORBILITY_TICKET','TOPUP,CREDIT','DEBIT','GIFT')
and `payment_transactions`.`status` != 'INITIATED'
Here is my code example:
//Get Transactions data
public function adminTransactions(Request $request)
{
$selectableFields = [
'payment_transactions.id', 'payment_transactions.transaction_id AS transaction_id',
'payment_transactions.refund_status',
'payment_transactions.created_at', 'payment_transactions.response_data', 'payment_transactions.status',
'payment_transactions.transaction_type', 'payment_transactions.payment_mode','payment_transactions.payment_source',
'payment_transactions.subtotal', 'payment_transactions.service_fees', 'C.coupon_code','C.amount AS coupon_value',
DB::raw("IF(payment_transactions.refund_remarks='NULL','-NA-',payment_transactions.refund_remarks) as refund_remarks"),
DB::raw("IF(payment_transactions.transaction_type='TOPUP' AND payment_transactions.coupon_id IS NOT NULL
AND payment_transactions.coupon_id!=0,
payment_transactions.amount + C.amount,payment_transactions.amount) as amount"),
DB::raw("CONCAT(U.first_name,' ',U.last_name) AS username"), 'U.id AS user_id',
DB::raw("JSON_UNQUOTE(json_extract(payment_transactions.response_data, '$.description')) AS description"),
DB::raw("payment_transactions.invoice_id"),
DB::raw("JSON_UNQUOTE(json_extract(payment_transactions.response_data, '$.Data.PaymentID')) AS upay_payment_id"),
];
return PaymentTransactions::select($selectableFields)
->with('homeScreenMessages:payment_transaction_id,from_name,message,amount')
->leftJoin('users AS U', 'U.id', '=', 'payment_transactions.user_id')
->leftJoin('coupons AS C', 'C.id', '=', 'payment_transactions.coupon_id')
->where(DB::raw("CONCAT(U.first_name,' ',U.last_name)"), 'like', "%{$request->input('query')}%")
->orWhere('U.id', $request->input('query'))
->orWhere("U.phone_number", "LIKE", "%" . $request->input('query') . "%")
->orWhere("U.email", "LIKE", "%" . $request->input('query') . "%")
->orWhere('payment_transactions.id', $request->input('query'))
->orWhere('payment_transactions.transaction_id', $request->input('query'));
}
//Paginate function
public function paginationCalculate($queryObject, $request) {
$draw = $request->get('draw');
$start = $request->get("start");
$rowperpage = $request->get("length"); // Rows display per page
$columnIndex_arr = $request->get('order');
$columnName_arr = $request->get('columns');
$order_arr = $request->get('order');
$columnIndex = $columnIndex_arr[0]['column']; // Column index
$columnName = $columnName_arr[$columnIndex]['name']; // Column name
$columnSortOrder = $order_arr[0]['dir']; // asc or desc
$pageNumber = ($start + $rowperpage) / $rowperpage;
if(!empty($columnName)) {
$queryObject->orderBy($columnName, $columnSortOrder);
}
$records = $queryObject->paginate($rowperpage, ['*'], 'page', $pageNumber)->toArray();
return array(
"draw" => intval($draw),
"recordsFiltered" => $records['total'],
"recordsTotal" => $records['total'],
"data" => $records['data']
);
}
I don't think you want "LEFT". With LEFT, the COUNT will count payment transactions even if there is no matching user and/or coupon.
Or maybe there are multiple users or coupons for each payment_transaction? In this case, the COUNT will be inflated. To fix this, simply remove the two extra tables.
Regardless of the situation, add this composite (and covering) index to payment_transactions:
INDEX(refund_status, transaction_type, status, user_id, coupon_id)
refund_type must be first since it is tested with '='.
If that is not "fast enough", then I must ask you "Who cares what the exact number when there are a million rows?" Do you ever see a search engine giving an exact number of hits? Does that imprecision bother you?
That is, rethink the requirement to calculate the exact number of rows. Or even an approximate number.
You mentioned "pagination". Please show us how that is done. OFFSET is an inefficient way since it rescans the previously seen rows again and again. Instead, 'remember where you left off': Pagination
You HAVE to adding INDEX to your MySQL database. Makes it lighting fast!
If you are using Navicat, click on the Index tab, and add the Column name and Name it the same. You WILL notice a huge difference.
I am using server-side rendering in DataTable.
I have a variable that determines limit to number of rows from database.
I am trying to fetch, say, 100 rows, and from DataTable with pagelength of 25, paginate it upto 4 pages.
Also, if the limit variable is 4 or 10, or anything less than pagelength value, then all records will be displayed in a single page.
Here's my code:
#get datatable params
$draw = $this->request->input('draw');
$start = $this->request->input('start');
$dt_limit = $this->request->input('length'); // datatable pagination limit
$limit_rows = $this->request->input('limit');
# query
$result_product_urls = DB::table('product_match_unmatches')
->select('r_product_url', 'r_image_url_main')
->selectRaw('count(distinct s_product_url, r_product_url) as frequency,
GROUP_CONCAT(CONCAT(s_image_url, "&s_product_url=", s_product_url) SEPARATOR " | ") as source_products')
->groupBy('r_product_url')
->where('request_id', '=', $reqId)
->orderBy('frequency', 'DESC')
->take($limit_rows);
$recordsTotal = $result_product_urls->get()->count();
$urls = $result_product_urls->offset($start)->limit($dt_limit)->get();
This will always fetch 25 rows from the database, since the dataTable page-length is 25.
But I need to limit the rows based on $limit_rows value, and not $dt_limit.
If I only do $urls = $result_product_urls->get();, then if $limit_rows value is 100, all 100 rows are displayed in the same page.
How can I fetch limit rows from database, and then paginate through it?
I hope I make my post clear.
I'll take a stab at this, but it's not entirely clear to me the exact thing you are attempting to accomplish based on the code sample.
According to Laravel docs take and limit appear to be equivalent.
https://laravel.com/docs/5.8/queries#ordering-grouping-limit-and-offset (bottom of this section)
What it looks like the code is doing:
Using $limit_rows value with ->take() method in the $result_product_urls query builder object:
# query
$result_product_urls = DB::table('product_match_unmatches')
->select('r_product_url', 'r_image_url_main')
->selectRaw('count(distinct s_product_url, r_product_url) as frequency,
GROUP_CONCAT(CONCAT(s_image_url, "&s_product_url=", s_product_url) SEPARATOR " | ") as source_products')
->groupBy('r_product_url')
->where('request_id', '=', $reqId)
->orderBy('frequency', 'DESC')
->take($limit_rows); /* <------------- limit_rows HERE */
But then also using $dt_limit value with ->limit() method when setting the $urls variable:
$recordsTotal = $result_product_urls->get()->count();
$urls = $result_product_urls->offset($start)->limit($dt_limit)->get();/*<-- dt_limit HERE */
If you don't want to use $dt_limit as a limit then don't use it...? I think maybe what you are trying to do is have a query with a limit on the result set, and then a sub limit on that? I can't figure out a reason to want to do that, so I might be missing something. I don't think you can use ->take() and ->limit() on the same query.
Shouldn't this accomplish your goal?:
# query
$result_product_urls = DB::table('product_match_unmatches')
->select('r_product_url', 'r_image_url_main')
->selectRaw('count(distinct s_product_url, r_product_url) as frequency,
GROUP_CONCAT(CONCAT(s_image_url, "&s_product_url=", s_product_url) SEPARATOR " | ") as source_products')
->groupBy('r_product_url')
->where('request_id', '=', $reqId)
->orderBy('frequency', 'DESC');
/* ->take($limit_rows); remove this */
$recordsTotal = $result_product_urls->get()->count();
$urls = $result_product_urls->offset($start)
->limit($limit_rows) /* set limit to $limit_rows */
->get();
If not, then maybe you can clarify.
use offset alongwith limit i.e.
$users = DB::table('users')->skip(10)->take(5)->get();
or
$users = DB::table('users')->offset(10)->limit(5)->get();
So I've been working on a pagination system that pulls data externally, after finally figuring out through trial and error I got the solution that works. While going through it something struck me as odd, as per the documentation $paginator = Paginator::make($items, $totalItems, $perPage);
I was wondering what's the actual use of the $perPage parameter? You would think that what ever number is specified would show that many items. But with manual pagination you have to limit the results that are passed into $items in order for it to work, otherwise you get the output of all items (as shown in code block below). Is manual pagination flawed? because if $perPage doesn't match the total number of items in the array $items it shows everything.
Example: Paginator::make( array('10xarray') ), 10, 2); it would show 5 pages with 2 items per page? where in reality it actually shows 10 items with 5 pages that all show the same 10 items.
<?php
class MainController extends BaseController {
public function library()
{
$this->layout->title = 'testing';
$this->layout->main = View::make('library/layout');
// Pagination data
$media = array(
array('title' => 'test'),
array('title' => 'test'),
array('title' => 'test'),
array('title' => 'test')
);
$perPage = 2;
$currentPage = Input::get('page', 1);
$pagedData = array_slice($media, ($currentPage - 1) * $perPage, $perPage);
$this->layout->main->paginated = Paginator::make($pagedData, count($media), $perPage);
if(Request::ajax()) {
return Response::json(
View::make(
'library/layout',
array('paginated' => $this->layout->main->paginated)
)->render()
);
}
}
}
it isn't flawed. it is intended behavior.
this manual paginator is just a container. nothing more, nothing less. how you implement it, is upon you.
without a LIMIT query, you can never do a pagination. when you use pagination(), laravel does the work under the hood. when you go for manual, you have to do it manually and get greater control.
why do you think it is called manual pagination in the first place?
I'm using Yii's Dataprovider to output a bunch of users based on the column "points";
It works fine now but I have to add a feature so if the user is online, he gets an extra 300 points.
Say Jack has 100 points, Richmond has 300 points, However Jack is online, so Jack should rank higher than Richmond.
Here is my solution now:
$user=new Rank('search');
$user->unsetAttributes();
$user->category_id = $cid;
$dataProvider = $user->search();
$iterator = new CDataProviderIterator($dataProvider);
foreach($iterator as $data) {
//check if online ,update points
}
However, this CDataProviderIterator seems change my pagination directly to the last page and I can't even switch page anymore. What should I do?
Thank you very much!
Here is the listview:
$this->widget('zii.widgets.CListView', array(
'id'=>'userslist',
'dataProvider'=>$dataProvider,
'itemView'=>'_find',
'ajaxUpdate'=>false,
'template'=>'{items}<div class="clear"></div><div style="margin-right:10px;"><br /><br />{pager}</div>',
'pagerCssClass'=>'right',
'sortableAttributes'=>array(
// 'create_time'=>'Time',
),
));
Updated codes in Rank.php model
$criteria->with = array('user');
$criteria->select = '*, (IF(user.lastaction > CURRENT_TIMESTAMP() - 1800, points+300, points)) as real_points';
$criteria->order = 'real_points DESC';
However, it throws me error:
Active record "Rank" is trying to select an invalid column "(IF(user.lastaction > CURRENT_TIMESTAMP() - 1800". Note, the column must exist in the table or be an expression with alias.
CDataProviderIterator iterates every dataprovider value, and stops at the end. I don't know all about this classes, but think the reason is in some internal iterator, that stops at the end of dataprovider after your foreach.
Iterators are used when you need not load all data (for large amounts of data) but need to process each row.
To solve your problem, just process data in your view "_find". Add points there if online.
Or if you want place this logic only in the model (following MVC :) ), add method to your model:
public function getRealPoints() {
return ($this->online) ? ($this->points + 300) : $this->points;
}
And you can use $user->realPoints to get points according to user online status
update: To order your list by "realPoints" you need to get it in your SQL.
So use your code:
$user=new Rank('search');
$user->unsetAttributes();
$user->category_id = $cid;
$dataProvider = $user->search();
and modify $user->search() function, by adding:
$criteria->select = '*, (IF(online='1', points+300, points)) as real_points';
$criteria->order = 'real_points DESC';
where online and points - your table columns.
I have a query that running way too slow. the page takes a few minutes to load.
I'm doing a table join on tables with over 100,000 records. In my query, is it grabbing all the records or is it getting only the amount I need for the page? Do I need to put a limit in the query? If I do, won't that give the paginator the wrong record count?
$paymentsTable = new Donations_Model_Payments();
$select = $paymentsTable->select(Zend_Db_Table::SELECT_WITH_FROM_PART);
$select->setIntegrityCheck(false)
->from(array('p' => 'tbl_payments'), array('clientid', 'contactid', 'amount'))
->where('p.clientid = ?', $_SESSION['clientinfo']['id'])
->where('p.dt_added BETWEEN \''.$this->datesArr['dateStartUnix'].'\' AND \''.$this->datesArr['dateEndUnix'].'\'')
->join(array('c' => 'contacts'), 'c.id = p.contactid', array('fname', 'mname', 'lname'))
->group('p.id')
->order($sortby.' '.$dir)
;
$payments=$paymentsTable->fetchAll($select);
// paginator
$paginator = Zend_Paginator::factory($payments);
$paginator->setCurrentPageNumber($this->_getParam('page'), 1);
$paginator->setItemCountPerPage('100'); // items pre page
$this->view->paginator = $paginator;
$payments=$payments->toArray();
$this->view->payments=$payments;
Please see revised code below. You need to pass the $select to Zend_Paginator via the correct adapter. Otherwise you won't see the performance benefits.
$paymentsTable = new Donations_Model_Payments();
$select = $paymentsTable->select(Zend_Db_Table::SELECT_WITH_FROM_PART);
$select->setIntegrityCheck(false)
->joinLeft('contacts', 'tbl_payments.contactid = contacts.id')
->where('tbl_payments.clientid = 39')
->where(new Zend_Db_Expr('tbl_payments.dt_added BETWEEN "1262500129" AND "1265579129"'))
->group('tbl_payments.id')
->order('tbl_payments.dt_added DESC');
// paginator
$paginator = new Zend_Paginator(new Zend_Paginator_Adapter_DbTableSelect($select));
$paginator->setCurrentPageNumber($this->_getParam('page', 1));
$paginator->setItemCountPerPage('100'); // items pre page
$this->view->paginator = $paginator;
Please see revised code above!
In your code, you are :
first, selecting and fetching all records that match your condition
see the select ... from... and all that
and the call to fetchAll on the line just after
and, only the, you are using the paginator,
on the results returned by the fetchAll call.
With that, I'd say that, yes, all your 100,000 records are fetched from the DB, manipulated by PHP, passed to Zend_Paginator which has to work with them... only to discard almost all of them.
Using Zend_Paginator, you should be able to pass it an instance of Zend_Db_Select, and let it execute the query, specifying the required limit.
Maybe the example about DbSelect and DbTableSelect adapter might help you understand how this can be achieved (sorry, I don't have any working example).
I personally count the results via COUNT(*) and pass that to zend_paginator. I never understood why you'd deep link zend_paginator right into the database results. I can see the pluses and minuses, but really, its to far imho.
Bearing in mind that you only want 100 results, you're fetching 100'000+ and then zend_paginator is throwing them away. Realistically you want to just give it a count.
$items = Eurocreme_Model::load_by_type(array('type' => 'list', 'from' => $from, 'to' => MODEL_PER_PAGE, 'order' => 'd.id ASC'));
$count = Eurocreme_Model::load_by_type(array('type' => 'list', 'from' => 0, 'to' => COUNT_HIGH, 'count' => 1));
$paginator = Zend_Paginator::factory($count);
$paginator->setItemCountPerPage(MODEL_PER_PAGE);
$paginator->setCurrentPageNumber($page);
$this->view->paginator = $paginator;
$this->view->items = $items;