Doctrine 2 conditional multiple row update with QueryBuilder - php

The question has some answers on SO, but non of them seems to help accomplish, actually, a simple task.
I need to update multiple rows based on condition in one query, using Doctrine2 QueryBuilder. Most obvious way is supposed to be wrong:
$userAgeList = [
'user_name_a' => 30,
'user_name_b' => 40,
'user_name_c' => 50,
];
//Array of found `User` objects from database
$usersList = $this->getUsersList();
foreach($usersList as $user)
{
$userName = $user->getName();
$user->setAge($userAgeList[$userName]);
$this->_manager->persist($user);
}
$this->_manager->flush();
It will create an update query for each User object in transaction, but I need only one query. This source suggests that instead you should rely on the UPDATE query, because in case of it we only execute one SQL UPDATE statement, so I've made like this:
$userAgeList = [
'user_name_a' => 30,
'user_name_b' => 40,
'user_name_c' => 50,
];
$builder = $this->_manager->getRepository(self::REPOSITORY_USER)
->createQueryBuilder(self::REPOSITORY_USER);
foreach($userAgeList as $userName => $age)
{
$builder->update(self::REPOSITORY_USER, 'user')
->set('user.age', $builder->expr()->literal($age))
->where('user.name = :name')
->setParameter('name', $userName)
->getQuery()->execute();
}
But that also makes (obviously) a bunch of updates instead of one. If I assign result of getQuery() to a variable, and try to execute() it after foreach loop, I see that $query understands and accumulates a set(), but it does no such thing for WHERE condition.
Is there any way to accomplish such task in QueryBuilder?
UPDATE - similar questions:
Multiple update queries in doctrine and symfony2 - this one does not assume UPDATE in one query;
Symfony - update multiple records - this also says, qoute, 'one select - one update';
Update multiple columns with Doctrine in Symfony - WHERE condition in this query is always the same, not my case;
Doctrine 2: Update query with query builder - same thing as previous, only one WHERE clause;
http://doctrine-orm.readthedocs.org/en/latest/reference/batch-processing.html - doctrine batch processing does not mention conditions at all...
In MySQL I used to do it using CASE-THEN, but that's not supported by Doctrine.

Related

Doctrine Paginator selects entire table (very slow)?

This is related to a previous question here: Doctrine/Symfony query builder add select on left join
I want to perform a complex join query using Doctrine ORM. I want to select 10 paginated blog posts, left joining a single author, like value for current user, and hashtags on the post. My query builder looks like this:
$query = $em->createQueryBuilder()
->select('p')
->from('Post', 'p')
->leftJoin('p.author', 'a')
->leftJoin('p.hashtags', 'h')
->leftJoin('p.likes', 'l', 'WITH', 'l.post_id = p.id AND l.user_id = 10')
->where("p.foo = bar")
->addSelect('a AS post_author')
->addSelect('l AS post_liked')
->addSelect('h AS post_hashtags')
->orderBy('p.time', 'DESC')
->setFirstResult(0)
->setMaxResults(10);
// FAILS - because left joined hashtag collection breaks LIMITS
$result = $query->getQuery()->getResult();
// WORKS - but is extremely slow (count($result) shows over 80,000 rows)
$result = new \Doctrine\ORM\Tools\Pagination\Paginator($query, true);
Strangely, count($result) on the paginator shows the total number of rows in my table (over 80,000) but traversing the $result with foreach outputs 10 Post entities, as expected. Do I need to do some additional configuration to properly limit my paginator?
If this is a limitation of the paginator class what other options do I have? Writing custom paginator code or other paginator libraries?
(bonus): How can I hydrate an array, like $query->getQuery()->getArrayResult();?
EDIT: I left out a stray orderBy in my function. It looks like including both groupBy and orderBy causes the slowdown (using groupBy rather than the paginator). If I omit one or the other, the query is fast. I tried adding an index on the "time" column in my table, but didn't see any improvement.
Things I Tried
// works, but makes the query about 50x slower
$query->groupBy('p.id');
$result = $query->getQuery()->getArrayResult();
// adding an index on the time column (no improvement)
indexes:
time_idx:
columns: [ time ]
// the above two solutions don't work because MySQL ORDER BY
// ignores indexes if GROUP BY is used on a different column
// e.g. "ORDER BY p.time GROUP BY p.id is" slow
You should simplify your query. That would shave off some execution time. I can't test your query but here are a few pointers:
don't do sort while executing count()
you could sort by orderBy('p.id', 'DESC'), index would be used
instead of leftJoin() you could use join() if at least one record always exists at joined table. Else that record is skipped.
KNP/Paginator uses DISTINCT() to read only distinct records, but that could lead to using disk tmp table
$query->getArrayResult() uses array hidration mode, which returns multidimension array and it is way faster than object hidration for large result set
you could use partial select('partial p.{id, other used fields}'), this way you would load only needed fields, maybe skip unneded relations when using object hydration
check SF profiler EXPLAIN on a given query under doctrine section, maybe indexes are not used
does p.hashtags and p.likes return only one row or is oneToMany, which multiplies result
maybe some Posts design changes, that would remove some joins:
have p.hashtags field defined as #ORM\Column(type="array") and have stored string values of tags. Later maybe using full text search on serialized array.
have p.likesCount field defined as #ORM\Column(type="integer") which would have count of likes
I use KnpLabs/KnpPaginatorBundle and can also have speed issues for complex queries.
Usually using LIMIT x,z is slow for DB, because it runs COUNT on whole dataset. If indexes are not used it is painfully slow.
You could use different approach and do some custom pagination by ID advancing, but that would complicate your approach. I have used this with large datasets like SYSLOG tables. But you loose sorting and total record count functionality.
At the end of the day, many of the queries used in my application are too complex to make proper use of the Paginator, and I wasn't able to use array hydration mode with the Paginator.
According to MySQL documentation, ORDER BY cannot be resolved by indexes if GROUP BY is used on a different column. Thus, I ended up using a couple post-processing queries to populate my base results (ORDERed and LIMITed) with one-to-many relations (like hashtags).
For joins that load a single row from the joined table, I was able to join the desired values in the base ordered query. For example, when loading the "like status" for a current user, only one like from the set of likes needs to be loaded to indicate whether or not the current post has been liked. Similarly, the presence of only one author for a given post produces a single joined author row. e.g.
$query = $em->createQueryBuilder()
->select('p')
->from('Post', 'p')
->leftJoin('p.author', 'a')
->leftJoin('p.likes', 'l', 'WITH', 'l.post_id = p.id AND l.user_id = 10')
->where("p.foo = bar")
->addSelect('a AS post_author')
->addSelect('l AS post_liked')
->orderBy('p.time', 'DESC')
->setFirstResult(0)
->setMaxResults(10);
// SUCCEEDS - because joins only join a single author and single like
// no collections are joined, so LIMIT applies only the the posts, as intended
$result = $query->getQuery()->getArrayResult();
This produces a result in the form:
[
[0] => [
['id'] => 1
['text'] => 'foo',
['author'] => [
['id'] => 10,
['username'] => 'username',
],
['likes'] => [
[0] => [
['post_id'] => 1,
['user_id'] => 10,
]
],
],
[1] => [...],
...
[9] => [...]
]
Then in a second query I load the hashtags for posts loaded in the previous query. e.g.
// we don't care about orders or limits here, we just want all the hashtags
$query = $em->createQueryBuilder()
->select('p, h')
->from('Post', 'p')
->leftJoin('p.hashtags', 'h')
->where("p.id IN :post_ids")
->setParameter('post_ids', $pids);
Which produces the following:
[
[0] => [
['id'] => 1
['text'] => 'foo',
['hashtags'] => [
[0] => [
['id'] => 1,
['name'] => '#foo',
],
[2] => [
['id'] => 2,
['name'] => '#bar',
],
...
],
],
...
]
Then I just traverse the results containing hashtags and append them to the original (ordered and limited) results. This approach ends up being much faster (even though it uses more queries), as it avoids GROUP BY and COUNT, fully leverages MySQL indexes, and allows for more complex queries, such as the one I posted here.
You can configure the paginator to use a simpler 'count' sql strategy by doing one or more of the optimizations below.
$paginator = new Paginator($query, false);
$paginator->setUseOutputWalkers(false);
If results are unexpected you may want to do a DISTINCT select (select('DISTINCT p'))
For us it made massive improvements and we had no need to write or use a custom paginator.
More details can be found on this site. Note that I am owner of that website.

Laravel 5 updating single row limit doesn't work

I need to update single matching record in db (PostgreSQL), but Limit method doesn't work with Update method. This code will update all records matching Where condition instead of single record.
DB::table("records")
->where('need_moderate','=','no')
->where('locked_per_time','<',$date_now->format("Y-m-d H:i:s"))
->limit(1)
->update(["locked_per_time"=>$locked_per->format("Y-m-d H:i:s"),'locked_by'=>$mdkey]);
How do I work around this so only single record would be updated?
Unlike with Oracle or MySQL update statements, using LIMIT directly on PostgreSQL update statements is not possible. So chaining the limit(1) method to the Query Builder instance does nothing, because the compileUpdate method from Laravel's PostgresGrammar class that is responsible for compiling the query, only compiles the where statements.
You could however overcome this by having a condition that uses a subquery which only returns one row that will be updated. Something like this should work:
DB::table("records")->whereIn('id', function ($query) use ($date_now) {
$query->from('records')
->select('id')
->where('need_moderate', '=', 'no')
->where('locked_per_time', '<', $date_now->format("Y-m-d H:i:s"))
->limit(1);
})->update(["locked_per_time" => $locked_per->format("Y-m-d H:i:s"), 'locked_by' => $mdkey]);
The whereIn('id', ...) condition assumes your table has a column named id that can be used as a unique identifier so it can find the first row that matches your conditions in the subquery.

Laravel - multi-insert rows and retrieve ids

I'm using Laravel 4, and I need to insert some rows into a MySQL table, and I need to get their inserted IDs back.
For a single row, I can use ->insertGetId(), however it has no support for multiple rows. If I could at least retrieve the ID of the first row, as plain MySQL does, would be enough to figure out the other ones.
It's mysql behavior of
last-insert-id
Important
If you insert multiple rows using a single INSERT statement, LAST_INSERT_ID() returns the value generated for the first inserted row only. The reason for this is to make it possible to reproduce easily the same INSERT statement against some other server.
u can try use many insert and take it ids or after save, try use $data->id should be the last id inserted.
If you are using INNODB, which supports transaction, then you can easily solve this problem.
There are multiple ways that you can solve this problem.
Let's say that there's a table called Users which have 2 columns id, name and table references to User model.
Solution 1
Your data looks like
$data = [['name' => 'John'], ['name' => 'Sam'], ['name' => 'Robert']]; // this will insert 3 rows
Let's say that the last id on the table was 600. You can insert multiple rows into the table like this
DB::begintransaction();
User::insert($data); // remember: $data is array of associative array. Not just a single assoc array.
$startID = DB::select('select last_insert_id() as id'); // returns an array that has only one item in it
$startID = $startID[0]->id; // This will return 601
$lastID = $startID + count($data) - 1; // this will return 603
DB::commit();
Now, you know the rows are between the range of 601 and 603
Make sure to import the DB facade at the top using this
use Illuminate\Support\Facades\DB;
Solution 2
This solution requires that you've a varchar or some sort of text field
$randomstring = Str::random(8);
$data = [['name' => "John$randomstring"], ['name' => "Sam$randomstring"]];
You get the idea here. You add that random string to a varchar or text field.
Now insert the rows like this
DB::beginTransaction();
User::insert($data);
// this will return the last inserted ids
$lastInsertedIds = User::where('name', 'like', '%' . $randomstring)
->select('id')
->get()
->pluck('id')
->toArray();
// now you can update that row to the original value that you actually wanted
User::whereIn('id', $lastInsertedIds)
->update(['name' => DB::raw("replace(name, '$randomstring', '')")]);
DB::commit();
Now you know what are the rows that were inserted.
As user Xrymz suggested, DB::raw('LAST_INSERT_ID();') returns the first.
According to Schema api insertGetId() accepts array
public int insertGetId(array $values, string $sequence = null)
So you have to be able to do
DB::table('table')->insertGetId($arrayValues);
Thats speaking, if using MySQL, you could retrive the first id by this and calculate the rest. There is also a DB::getPdo()->lastInsertId(); function, that could help.
Or if it returened the last id with some of this methods, you can calculate it back to the first inserted too.
EDIT
According to comments, my suggestions may be wrong.
Regarding the question of 'what if row is inserted by another user inbetween', it depends on the store engine. If engine with table level locking (MyISAM, MEMORY, and MERGE) is used, then the question is irrevelant, since thete cannot be two simultaneous writes to the table.
If row-level locking engine is used (InnoDB), then, another possibility might be to just insert the data, and then retrieve all the rows by some known field with whereIn() method, or figure out the table level locking.
$result = Invoice::create($data);
if ($result) {
$id = $result->id;
it worked for me
Note: Laravel version 9

Doctrine 2 delete with query builder

I have two Entities with relation OneToMany, Project and Services. Now i want to remove all the services by project_id.
First attempt:
$qb = $em->createQueryBuilder();
$qb->delete('Services','s');
$qb->andWhere($qb->expr()->eq('s.project_id', ':id'));
$qb->setParameter(':id',$project->getId());
This attempt fails with the Exception Entity Service does not have property project_id. And it's true, that property does not exists, it's only in database table as foreign key.
Second attempt:
$qb = $em->createQueryBuilder();
$qb->delete('Services','s')->innerJoin('s.project','p');
$qb->andWhere($qb->expr()->eq('p.id', ':id'));
$qb->setParameter(':id',$project->getId());
This one generetate a non valid DQL query too.
Any ideas and examples will be welcome.
You're working with DQL, not SQL, so don't reference the IDs in your condition, reference the object instead.
So your first example would be altered to:
$qb = $em->createQueryBuilder();
$qb->delete('Services', 's');
$qb->where('s.project = :project');
$qb->setParameter('project', $project);
If you really can't get project object and you have to handle only with id you can use this.
Citation from doctrine documentation:
There are two possibilities for bulk deletes with Doctrine. You can either issue a single DQL DELETE query or you can iterate over results removing them one at a time. (Below I paste only first solution)
DQL Query
The by far most efficient way for bulk deletes is to use a DQL DELETE query.
Example that worked in my project
$q = $em->createQuery('delete from AcmeMyTestBundle:TemplateBlock tb where tb.template = '.intval($templateId));
$numDeleted = $q->execute();
In entity TemplateBlock I have property called template which is mapped to template_id in db.
But I agree that highly preferred way of doing it is using objects.

How get normal SQL query from cake query array?

I have array like this
$conditions = array("Post.title" => "This is a post");
And using $conditions array in this method.
$this->Post->find('first', array('conditions' => $conditions));
I want convert the $conditions array to normal sql query.
I want use
$this->Post->query($converted_query);
instead of
$this->Post->find('first', array('conditions' => $conditions));
$null=null;
echo $this->getDataSource()->generateAssociationQuery($this, NULL, NULL, NULL, NULL, $query_array, false,$null);
To do what you want you could do two things:
1) Combine your $conditions arrays and let CakePHP build your new query so you can simply use $this->Model->find() again.
2) Use this. It's an expansion for the mysql datasource that adds the option to do $this->Model->find('sql', array('conditions' => $conditions)) which will return the SQL-query. This option might cause trouble, because for some find calls (especially when you're fetching associated models) CakePHP uses multiple queryies to fetch the associated models (especially in case of hasMany-associations).
If at all possible, option 1 will probably cause the least trouble. Another problem with going with 2 is that if you're trying to combine two queries with conflicting conditions (like 'name = Hansel' in query 1 and 'name = Gretel' in query 2) you will just find nothing unless you plan on writing extra code to parse the resulting queries and look for conflicts..
Going with 1 will probably be a lot simpler and will probably avoid lots of problems.

Categories