I have table (~150 columns with ~150k records) and project on Symfony 3 with Doctrine. In project is clasic filter to show results.
If you submit form i collect data in object $selectedInputOptions and build query looks like:
$query = $repository
->createQueryBuilder('t')
->select('t.idkatcountry', 't.idkatlocality', 't, MIN(t.price) AS priceFrom'......);
if(count($selectedInputOptions->getCountry()) > 0)
$query->andWhere('t.idkatcountry IN (:idkatcountry )')->setParameter('idkatcountry ', $selectedInputOptions->getCountry());
if(count($selectedInputOptions->getLocality()) > 0)
$query->andWhere('t.idkatlocality IN (:idkatlocality )')->setParameter('idkatlocality ', $selectedInputOptions->getLocality());
price column have decimal(15,2) datatype
Before i have in $repository->select('t.price') and everything was OK but after change this to 't, MIN(t.price) AS priceFrom' query execution time was increased +40% and in few cases (any input in form be blank = checks all records) +900%.
So my questions:
How i can cut execution time? (Is there some idexes for this?, Will help change datetype range let's say to decimal(6,2)?)
And bonus question :) Table has ~150columns but query for filtering using ~10-15 columns can i set some type of index for quicker selects?
EDIT:
changed column price to ineger - did not help
added index to column pricte - did not help
SOLUTION!
It was little mistake in select parameter using MIN().
Insted of:
't, MIN(t.price) AS priceFrom'
I used:
'MIN(t.price) AS priceFrom')'
Because t takes ALL columns (~150 in my case) and I didn't notice this... So now is everything OK and time is normal.
Here you can do one thing, stop loading unwanted data in entity, by using unset in jsonSerialize() method.
Related
So basically, I got 2 tables.
The first table contains 1 million rows or something, with an empty field called 'telefon'.
Now, I got a second table, which has the field values for 'telefon' in the other table.
I came up with this solution, but this takes forever. It has been an hour, and when inspecting the database table, only 1600 rows are done. Is there any faster ways of doing this? Thanks in advance.
DB::table('phones') -> orderBy('id') -> chunk(100, function($old) {
foreach ($old as $x) {
DB::table('companies')
-> where('organisasjonsnummer', $x -> businessCode)
-> update([
'telefon' => $x -> contact
]);
}
});
Huh, foreach + queries is almost always bad. If I am not mistaken, you would like to do this:
UPDATE companies, phones SET companies.telefon = phones.contact WHERE companies.organisasjonsnummer = phones.businessCode
It may be very slow if there's no index on companies.organisasjonsnummer and phones.businessCode columns, but it can take a lot of time to index them now as well, so I'm not sure if there's any benefit to index them now if they won't be used later. Anyway, using a single query should be faster at least to some extent.
Always remember, when you use Eloquent/SQL inside a loop, you will run a command for each round.
The same applies for lazy loading.
In this case you should use \DB::statement("put your sql here"); or in this case \DB::update("put your update here");, let the database do the service for you and be happy!.
I'm using a geoNear() to calculate distances between objects in my Mongo database.
The query works perfectly, even with addition field filters such as ->field('name')->equals($name) etc...
This automatically populates a mapped field #ODM\Distance on the object.
$this->getQueryBuilder()
->geoNear((float) $query['near_longitude'], (float) $query['near_latitude'])
->spherical(true)
->distanceMultiplier(self::EARTH_RD_KM);
If I add an ->field('id')->in($array) however this distance is suddenly 0.
I'm not really sure where the information is lost.
Is this a limitation on how $in works on MongoDB?
You need to use "maxDistance" function
like this;
$this->getQueryBuilder()
->geoNear((float) $query['near_longitude'], (float)$query['near_latitude'])
->spherical(true)
->maxDistance(self::YOUR_DESIRED_KM_RADIUS / self::EARTH_RD_KM)
->distanceMultiplier(self::EARTH_RD_KM);
That way it sets a radius for your query.
(updates at bottom)
I'm trying to get the latest entry in my table called "VersionHistory", and since the ID is set to auto increment, I was trying to get the max id. Trying to stay away from sorting the whole table in descending order and taking the top as I want to minimize the computation required for this query as the table grows, and this table will probably get pretty huge fast.
class VersionHistoryQuery extends BaseVersionHistoryQuery {
public function getLatestVersion() {
return $this
->withColumn('MAX(version_history.id)')
->limit(1);
}
}
I'm calling the function in my VersionHistory constructor as below:
class VersionHistory extends BaseVersionHistory {
public function __construct($backupPath = "") {
$lastVersion = VersionHistoryQuery::create()
->getLatestVersion()
->find();
$lastVersion->setBackupPath("backup/" . $backupPath);
$lastVersion->save();
parent::setImportedAt(date("Y-m-d H:i:s"));
}
}
This outputs a "Allowed memory size exhausted" error in php. Any idea why? Commenting out the query in the VersionHistory constructor fixes the error, so it's somewhere in the query. I tried setting up a custom query following the instructions here: http://propelorm.org/documentation/03-basic-crud.html#using-custom-sql. But I couldn't get that to work. Running:
SELECT * FROM version_history WHERE id = (SELECT MAX(id) FROM version_history)
From MySQL workbench works fine and quickly.
Any ideas of what I'm doing wrong?
What I tried
Updated the code to:
public function getLatestVersion() {
return $this
->orderById('desc')
->findOne();
}
Still get the same memory allocation error.
Updated the code to:
$lastVersion = VersionHistoryQuery::create()
->orderById('desc')
->findOne();
Removed the custom function, turned on propel debug mode, it outputs that this query is run:
[2015-10-11 17:26:54] shop_reporting_db.INFO: SELECT `version_history`.`id`, `version_history`.`imported_at`, `version_history`.`backup_path` FROM `version_history` ORDER BY `version_history`.`id` DESC LIMIT 1 [] []
Still runs into a memory overflow.
Thats all:
SELECT * FROM version_history ORDER BY id DESC LIMIT 1;
From the documentation, withColumn does the following:
Propel adds the 'with' column to the SELECT clause of the query, and
uses the second argument of the withColumn() call as a column alias.
So, it looks like you are actually querying every row in the table, and also every row is querying the max ID.
I don't know anything about propel (except what I just googled), but it looks like you need a different way to specify your where condition.
Your raw SQL and your Propel Query are different / not equivalent.
In the propel query merely added a column, whereas is your raw SQL your actually have two queries with one being a sub-query to the other.
So you need to do the equivalent in Propel:
$lastVersionID = VersionHistoryQuery::create()
->withColumn('MAX(id)', 'LastVersionID')
->select('LastVersionID')
->findOne();
$lastVersion = VersionHistoryQuery::create()
->filterByVersionHistory($lastVersionID)
->find();
Note the ->select('LatestVersionID') since you only need a scalar value and not an entire object, as well as the virtual column (alias in SQL) using withColumn()
I have a problem with Laravel's ORM Eloquent chunk() method.
It misses some results.
Here is a test query :
$destinataires = Destinataire::where('statut', '<', 3)
->where('tokenized_at', '<', $date_active)
->chunk($this->chunk, function ($destinataires) {
foreach($destinataires as $destinataire) {
$this->i++;
}
}
echo $this->i;
It gives 124838 results.
But :
$num_dest = Destinataire::where('statut', '<', 3)
->where('tokenized_at', '<', $date_active)
->count();
echo $num_dest;
gives 249676, so just TWICE as the first code example.
My script is supposed to edit all matching records in the database. If I launch it multiple times, it just hands out half the remaining records, each time.
I tried with DB::table() instead of the Model.
I tried to add a ->take(20000) but it doesn't seem to be taken into account.
I echoed the query with ->toSql() and eveything seems to be fine (the LIMIT clause is added when I add the ->take() parameter).
Any suggestions ?
Imagine you are using chunk method to delete all of the records. The table has 2,000,000 records and you are going to delete all of them by 1000 chunks.
$query->orderBy('id')->chunk(1000, function ($items) {
foreach($items as $item) {
$item->delete();
}
});
It will delete the first 1000 records by getting first 1000 records in a query like this:
SELECT * FROM table ORDER BY id LIMIT 0,1000
And then the other query from chunk method is:
SELECT * FROM table ORDER BY id LIMIT 1000,2000
Our problem is here, that we delete 1000 records and then getting results from 1000 to 2000. Actually we are missing first 1000 records and this means that we are not going to delete 1000 records in first step of chunk! This scenario will be the same for other steps. In each step we are going to miss 1000 records and this is the reason that we are not getting best result in these situations.
I made an example for deletion because this way we could know the exact behavior of chunk method.
UPDATE:
You can use chunkById() for deleting safely.
Read more here:
http://laravel.at.jeffsbox.eu/laravel-5-eloquent-builder-chunk-chunkbyid
https://laravel.com/api/5.4/Illuminate/Database/Eloquent/Builder.html#method_chunkById
Quick answer: Use chunkById() instead of chunk().
When updating or deleting records while iterating over them, any changes to the primary key or foreign keys could affect the chunk query. This could potentially result in records not being included in the results.
The explanation can be found in the Laravel documentation:
If you are updating database records while chunking results, your chunk results could change in unexpected ways. If you plan to update the retrieved records while chunking, it is always best to use the chunkById method instead. This method will automatically paginate the results based on the record's primary key.
Example usage of chunkById():
DB::table('users')->where('active', false)
->chunkById(100, function ($users) {
foreach ($users as $user) {
DB::table('users')
->where('id', $user->id)
->update(['active' => true]);
}
});
(end of the update)
Below is the original answer which used the cursor() method instead of the chunk() method to solve the problem:
I had the same problem - only half of the total results were passed to the callback function of the chunk() method.
Here is the code which had the same problem - half of the transactions were not processed:
Transaction::whereNull('processed')->chunk(100, function ($transactions) {
$transactions->each(function($transaction){
$transaction->process();
});
});
I used Laravel 5.4 and managed to solve the problem replacing the chunk() method with cursor() method and changing the code accordingly:
foreach (Transaction::whereNull('processed')->cursor() as $transaction) {
$transaction->process();
}
Even though the answer doesn't address the problem itself, it provides a valuable solution.
For anyone looking for a bit of code that solves this, here you go:
while (Model::where('x', '>', 'y')->count() > 0)
{
Model::where('x', '>', 'y')->chunk(10, function ($models)
{
foreach ($models as $model)
{
$model->delete();
}
});
}
The problem is in the deletion / removal of the model while chunking away at the total. Including it in a while loop makes sure you get them all! This example works when deleting Models, change the while condition to suit your needs!
When you fetch data using chunk the same SQL query is being executed only the offset is different. Actually increasing as specified on the chunk method param. For example:
SELECT * FROM users WHERE status = 0;
Let's say there are 200 records(let's suppose that is a lot so we want to retrieve these data as chunks). So this looks like:
SELECT * FROM users WHERE status = 0 LIMIT 50 OFFSET 0(offset has a dynamic value
which means next time is 50, after that 100, and the last time 150).
The problem when using laravel chunk while updating is that we are only changing the offset. And this means the number of results is different each time we try to retrieve a chunk of data. So the first time there are 200 records that match the where condition. But if we update the status, for example to 1(status = 1) this means the next time when we try to fetch data we still execute the same query:
SELECT * FROM users WHERE status = 0 LIMIT 50 OFFSET 50(offset has a dynamic value
which means next time 100 and the last time 150).
We only have 150 records that match this query since we updated the table status = 1 for 50rows. Also we said the offset on the second time is going to be 50. And what is going to happen is that we skip 50 rows from 150rows since the offset is 50. And do the same update to these data. This means rows from 50->100 status is being updated to 1(status = 1) from the total of 150 rows.
The third time we run this query:
SELECT * FROM users WHERE status = 0 LIMIT 50 OFFSET 150(offset is going to be 150).
But the result of the query is 100 users in total that have status = 0. So no more data to go through.
This is not what you would expect to happen on the first thought. But this is how it works and why only half of data are being updated and the other part of data is being skipped.
I have table of dogs in my DB and I want to retrieve N latest added dogs.
Only way that I found is something like this:
Dogs:all()->where(time, <=, another_time);
Is there another way how to do it? For example something like this Dogs:latest(5);
Thank you very much for any help :)
You may try something like this:
$dogs = Dogs::orderBy('id', 'desc')->take(5)->get();
Use orderBy with Descending order and take the first n numbers of records.
Update (Since the latest method has been added):
$dogs = Dogs::latest()->take(5)->get();
My solution for cleanliness is:
Dogs::latest()->take(5)->get();
It's the same as other answers, just with using built-in methods to handle common practices.
Dogs::orderBy('created_at','desc')->take(5)->get();
You can pass a negative integer n to take the last n elements.
Dogs::all()->take(-5)
This is good because you don't use orderBy which is bad when you have a big table.
You may also try like this:
$recentPost = Article::orderBy('id', 'desc')->limit(5)->get();
It's working fine for me in Laravel 5.6
I use it this way, as I find it cleaner:
$covidUpdate = COVIDUpdate::latest()->take(25)->get();
Ive come up with a solution that helps me achieve the same result using the array_slice() method. In my code I did array_slice( PickupResults::where('playerID', $this->getPlayerID())->get()->toArray(), -5 ); with -5 I wanted the last 5 results of the query.
The Alpha's solution is very elegant, however sometimes you need to re-sort (ascending order) the results in the database using SQL (to avoid in-memory sorting at the collection level), and an SQL subquery is a good way to achieve this.
It would be nice if Laravel was smart enough to recognise we want to create a subquery if we use the following ideal code...
$dogs = Dogs::orderByDesc('id')->take(5)->orderBy('id')->get();
...but this gets compiled to a single SQL query with conflicting ORDER BY clauses instead of the subquery that is required in this situation.
Creating a subquery in Laravel is unfortunately not simply as easy as the following pseudo-code that would be really nice to use...
$dogs = DB::subQuery(
Dogs::orderByDesc('id')->take(5)
)->orderBy('id');
...but the same result can be achieved using the following code:
$dogs = DB::table('id')->select('*')->fromSub(
Dogs::orderByDesc('id')->take(5)->toBase(),
'sq'
)->orderBy('id');
This generates the required SELECT * FROM (...) AS sq ... sql subquery construct, and the code is reasonably clean in terms of readability.)
Take particular note of the use of the ->toBase() function - which is required because fromSub() doesn't like to work with Eloquent model Eloquent\Builder instances, but seems to require a Query\Builder instance). (See: https://github.com/laravel/framework/issues/35631)
I hope this helps someone else, since I just spent a couple of hours researching how to achieve this myself. (I had a complex SQL query builder expression that needed to be limited to the last few rows in certain situations).
For getting last entry from DB
$variable= Model::orderBy('id', 'DESC')->limit(1)->get();
Imagine a situation where you want to get the latest record of data from the request header that was just inserted into the database:
$noOfFilesUploaded = count( $request->pic );// e.g 4
$model = new Model;
$model->latest()->take($noOfFilesUploaded);
This way your take() helper function gets the number of array data that was just sent via the request.
You can get only ids like so:
$model->latest()->take($noOfFilesUploaded)->puck('id')
use DB;
$dogs = DB::select(DB::raw("SELECT * FROM (SELECT * FROM dogs ORDER BY id DESC LIMIT 10) Var1 ORDER BY id ASC"));
Dogs::latest()->take(1)->first();
this code return the latest record in the collection
Can use this latest():
$dogs = Dogs::latest()->take(5)->get();