I have a PHP ActiveRecord model in which I have a function that requires the number of rows a query will return. I obtain the number of rows using the built in static::count($conditions) function. This works well and good but the issue arises when I include a GROUP BY statement. When I include this the count returns 1. I examined the resulting SQL and it was similar to
SELECT COUNT(*)
FROM TABLE
/* JOINS */
/* WHERE CONDITIONS */
GROUP BY `field`
When I ran the query manually I get
1
1
1
.
.
.
1
(1,000 times since there are 1,000 rows in the DB)
When I remove the GROUP BY statement, I get the value 1,000 like I should.
Obviously this occurs since COUNT is an aggregate function and it doesn't play well with group by. So with that being said, how can I return the correct number of rows using activerecord with a group by?
I had the same problem. I followed the example set by #jvenema in this question, wherein one defines a BaseModel class to override default ActiveRecord\Model behavior. Your models will then extend the BaseModel class.
class BaseModel extends ActiveRecord\Model
{
public static function count(/* ... */)
{
$args = func_get_args();
$options = static::extract_and_validate_options($args);
// Call the original function if $options['group'] is undefined
if ( !array_key_exists('group', $options) )
return call_user_func_array( 'parent::count', func_get_args() );
// This might fail if the table has a `counts` column
$options['select'] = 'COUNT(*) as counts';
if (!empty($args) && !is_null($args[0]) && !empty($args[0]))
{
if (is_hash($args[0]))
$options['conditions'] = $args[0];
else
$options['conditions'] = call_user_func_array('static::pk_conditions',$args);
}
$table = static::table();
$sql = $table->options_to_sql($options);
$values = $sql->get_where_values();
// Again, this might fail if there is a table named `tmp`
$wrapper = "SELECT COUNT(counts) FROM ({$sql->to_s()}) as tmp";
// Casting to (int) is optional; remove if it causes problems
return (int) static::connection()->query_and_fetch_one($wrapper,$values);
}
}
This function will fire only if $options['group'] is set. Additionally, note that this executes a COUNT() of rows created by GROUP BY rather than a SUM(). This is meant to account for cases when $has_many and $options['joins'] are in play, so as to prevent double-counting when INNER JOIN returns multiple results for an association.
Related
I need to get the sum of the field "valor", from the table "orcamentos".
I'm using this and it is working, but I know that this is not the right way:
//function index() from OrcamentosController.php
$orcamentoSubprojetosTotal = $this->Orcamento->query(
"SELECT
SUM(Orcamento.valor) AS TotalOrcamentoSuprojetos
FROM
orcamentos AS Orcamento
WHERE
Orcamento.subprojeto_id IS NOT NULL;"
);
$this->set(compact('orcamentoSubprojetosTotal'));
I have found this question cakephp sum() on single field (and others sum() function in cakephp query, using virtual fields to sum values in cakephp), but in the moment I add this line to my controller:
$this->Orcamento->virtualFields['total'] = 'SUM(Orcamento.valor)';
The paginate() stops working and display only one entry, like so:
Page 1 of 1, showing 1 records out of 2 total, starting on record 1, ending on 2
This is my index() function:
public function index($tipoOrcamento = null) {
$this->Orcamento->recursive = 0;
/*
$orcamentoSubprojetosTotal = $this->Orcamento->query(
"SELECT
SUM(Orcamento.valor) AS TotalOrcamentoSuprojetos
FROM
orcamentos AS Orcamento
WHERE
Orcamento.subprojeto_id IS NOT NULL;"
);
$this->set(compact('orcamentoSubprojetosTotal'));
*/
$this->set(compact('tipoOrcamento'));
if($tipoOrcamento == 'subtitulo'){
$this->set('orcamentos', $this->Paginator->paginate('Orcamento', array('Orcamento.subtitulo_id IS NOT NULL')));
}elseif($tipoOrcamento == 'subprojeto'){
$this->set('orcamentos', $this->Paginator->paginate('Orcamento', array('Orcamento.subprojeto_id IS NOT NULL')));
}else{
$this->set('orcamentos', $this->Paginator->paginate('Orcamento'));
}
}
Can I use the query() or someone can help me with the virtual field?
Thank you.
Do not use a virtual field
A virtual field is intended for things like:
public $virtualFields = array(
'name' => 'CONCAT(User.first_name, " ", User.last_name)'
);
If a virtual field is used from something which does not belong to the/a single row - it will not do what you're expecting as evident by the secondary effects on the find call paginate generates.
Use field
Instead, use the field method and pass in the expression:
$integer = $Model->field(
'SUM(valor)',
array('NOT' => array('subprojeto_id' => null))
);
Which will execute:
SELECT SUM(valor) from x where NOT (subprojecto_id IS NULL);
This will also return a scalar value, whereas calling query as shown in the question will return a nested array.
of course when you use SUM you'll get a single record
there are two things you can do:
create the virtualField just before the find() call and unset it just after the query.
using 'fields' => arra(...) in your paginator setting and list just the fields you need to retrieve and not the virtualField when you don't want to SUM
Here is how I query my database for some words
$query = $qb->select('w')
->from('DbEntities\Entity\Word', 'w')
->where('w.indictionary = 0 AND w.frequency > 3')
->orderBy('w.frequency', 'DESC')
->getQuery()
->setMaxResults(100);
I'm using mysql and I'd like to get random rows that match the criteria, I would use order by rand() in my query.
I found this similar question which basically suggests since ORDER BY RAND is not supported in doctrine, you can randomize the primary key instead. However, this can't be done in my case because I have a search criteria and a where clause so that not every primary key will satisfy that condition.
I also found a code snippet that suggests you use the OFFSET to randomize the rows like this:
$userCount = Doctrine::getTable('User')
->createQuery()
->select('count(*)')
->fetchOne(array(), Doctrine::HYDRATE_NONE);
$user = Doctrine::getTable('User')
->createQuery()
->limit(1)
->offset(rand(0, $userCount[0] - 1))
->fetchOne();
I'm a little confused as to whether this will help me work around the lack of support for order by random in my case or not. I was not able to add offset after setMaxResult.
Any idea how this can be accomplished?
The Doctrine team is not willing to implement this feature.
There are several solutions to your problem, each having its own drawbacks:
Add a custom numeric function: see this DQL RAND() function
(might be slow if you have lots of matching rows)
Use a native query
(I personally try to avoid this solution, which I found hard to maintain)
Issue a raw SQL query first to get some IDs randomly, then use the DQL WHERE x.id IN(?) to load the associated objects, by passing the array of IDs as a parameter.
This solution involves two separate queries, but might give better performance than the first solution (other raw SQL techniques than ORDER BY RAND() exist, I won't detail them here, you'll find some good resources on this website).
Follow these steps:
Define a new class at your project as:
namespace My\Custom\Doctrine2\Function;
use Doctrine\ORM\Query\Lexer;
class Rand extends \Doctrine\ORM\Query\AST\Functions\FunctionNode
{
public function parse(\Doctrine\ORM\Query\Parser $parser)
{
$parser->match(Lexer::T_IDENTIFIER);
$parser->match(Lexer::T_OPEN_PARENTHESIS);
$parser->match(Lexer::T_CLOSE_PARENTHESIS);
}
public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
{
return 'RAND()';
}
}
Register the class config.yml:
doctrine:
orm:
dql:
numeric_functions:
Rand: My\Custom\Doctrine2\Function\Rand
Use it directly as:
$qb->addSelect('RAND() as HIDDEN rand')->orderBy('rand()'); //Missing curly brackets
In line with what Hassan Magdy Saad suggested, you can use the popular DoctrineExtensions library:
See mysql implementation here: https://github.com/beberlei/DoctrineExtensions/blob/master/src/Query/Mysql/Rand.php
# config.yml
doctrine:
orm:
dql:
numeric_functions:
rand: DoctrineExtensions\Query\Mysql\Rand
Tested in Doctrine ORM 2.6.x-dev, you can then actually do:
->orderBy('RAND()')
Or you could do this -->
$words = $em->getRepository('Entity\Word')->findAll();
shuffle($words);
Of course this would be very inefficient if you have many records so use with caution.
Why not to use repository?
<?php
namespace Project\ProductsBundle\Entity;
use Doctrine\ORM;
class ProductRepository extends ORM\EntityRepository
{
/**
* #param int $amount
* #return Product[]
*/
public function getRandomProducts($amount = 7)
{
return $this->getRandomProductsNativeQuery($amount)->getResult();
}
/**
* #param int $amount
* #return ORM\NativeQuery
*/
public function getRandomProductsNativeQuery($amount = 7)
{
# set entity name
$table = $this->getClassMetadata()
->getTableName();
# create rsm object
$rsm = new ORM\Query\ResultSetMapping();
$rsm->addEntityResult($this->getEntityName(), 'p');
$rsm->addFieldResult('p', 'id', 'id');
# make query
return $this->getEntityManager()->createNativeQuery("
SELECT p.id FROM {$table} p ORDER BY RAND() LIMIT 0, {$amount}
", $rsm);
}
}
For me, the most useful way was to create two arrays where i say order type and different properties of the Entity. For example:
$order = array_rand(array(
'DESC' => 'DESC',
'ASC' => 'ASC'
));
$column = array_rand(array(
'w.id' => 'w.id',
'w.date' => 'w.date',
'w.name' => 'w.name'
));
You could add more entries to array $column like criteria.
Afterwards, you can build your query with Doctrine adding $column and $order inside ->orderBy. For example:
$query = $qb->select('w')
->from('DbEntities\Entity\Word', 'w')
->where('w.indictionary = 0 AND w.frequency > 3')
->orderBy($column, $order)
->getQuery()
->setMaxResults(100);
This way improved the performance of my application. I hope this helps someone.
Shuffling can be done on the query (array) result, but shuffling does not pick randomly.
In order to pick randomly from an entity I prefer to do this in PHP, which might slow the random picking, but it allows me to keep control of testing what I am doing and makes eventual debugging easier.
The example below puts all IDs from the entity into an array, which I can then use to "random-treat" in php.
public function getRandomArt($nbSlotsOnPage)
{
$qbList=$this->createQueryBuilder('a');
// get all the relevant id's from the entity
$qbList ->select('a.id')
->where('a.publicate=true')
;
// $list is not a simple list of values, but an nested associative array
$list=$qbList->getQuery()->getScalarResult();
// get rid of the nested array from ScalarResult
$rawlist=array();
foreach ($list as $keyword=>$value)
{
// entity id's have to figure as keyword as array_rand() will pick only keywords - not values
$id=$value['id'];
$rawlist[$id]=null;
}
$total=min($nbSlotsOnPage,count($rawlist));
// pick only a few (i.e.$total)
$keylist=array_rand($rawlist,$total);
$qb=$this->createQueryBuilder('aw');
foreach ($keylist as $keyword=>$value)
{
$qb ->setParameter('keyword'.$keyword,$value)
->orWhere('aw.id = :keyword'.$keyword)
;
}
$result=$qb->getQuery()->getResult();
// if mixing the results is also required (could also be done by orderby rand();
shuffle($result);
return $result;
}
#Krzysztof's solution is IMHO best here, but RAND() is very slow on large queries, so i updated #Krysztof's solution to gives less "random" results, but they are still random enough. Inspired by this answer https://stackoverflow.com/a/4329492/839434.
namespace Project\ProductsBundle\Entity;
use Doctrine\ORM;
class ProductRepository extends ORM\EntityRepository
{
/**
* #param int $amount
* #return Product[]
*/
public function getRandomProducts($amount = 7)
{
return $this->getRandomProductsNativeQuery($amount)->getResult();
}
/**
* #param int $amount
* #return ORM\NativeQuery
*/
public function getRandomProductsNativeQuery($amount = 7)
{
# set entity name
$table = $this->getClassMetadata()
->getTableName();
# create rsm object
$rsm = new ORM\Query\ResultSetMapping();
$rsm->addEntityResult($this->getEntityName(), 'p');
$rsm->addFieldResult('p', 'id', 'id');
# sql query
$sql = "
SELECT * FROM {$table}
WHERE id >= FLOOR(1 + RAND()*(
SELECT MAX(id) FROM {$table})
)
LIMIT ?
";
# make query
return $this->getEntityManager()
->createNativeQuery($sql, $rsm)
->setParameter(1, $amount);
}
}
I hope this would help others:
$limit = $editForm->get('numberOfQuestions')->getData();
$sql = "Select * from question order by RAND() limit $limit";
$statement = $em->getConnection()->prepare($sql);
$statement->execute();
$questions = $statement->fetchAll();
Note here the table question is an AppBundle:Question Entity. Change the details accordingly. The number of questions is taken from the edit form, make sure to check the variable for the form builder and use accordingly.
First get the MAX value from DB table & then use this as offset in PHP i.e
$offset = mt_rand(1, $maxId)
I know this is an old question. But I used the following solution to get the random row.
Using an EntityRepository method:
public function findOneRandom()
{
$id_limits = $this->createQueryBuilder('entity')
->select('MIN(entity.id)', 'MAX(entity.id)')
->getQuery()
->getOneOrNullResult();
$random_possible_id = rand($id_limits[1], $id_limits[2]);
return $this->createQueryBuilder('entity')
->where('entity.id >= :random_id')
->setParameter('random_id', $random_possible_id)
->setMaxResults(1)
->getQuery()
->getOneOrNullResult();
}
Probably the easiest (but not necessarily the smartest) way to get a single object result ASAP would be implementing this in your Repository class:
public function findOneRandom()
{
$className = $this->getClassMetadata()->getName();
$counter = (int) $this->getEntityManager()->createQuery("SELECT COUNT(c) FROM {$className} c")->getSingleScalarResult();
return $this->getEntityManager()
->createQuery("SELECT ent FROM {$className} ent ORDER BY ent.id ASC")
->setMaxResults(1)
->setFirstResult(mt_rand(0, $counter - 1))
->getSingleResult()
;
}
Just add the following:
->orderBy('RAND()')
I'm doing a join between two tables using the doctrine that comes bundled in the current symfony release. This is my controller code:
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Acme\GearDBBundle\Entity\TbGear;
use Acme\GearDBBundle\Entity\TbDships;
class DefaultController extends Controller
{
public function indexAction()
{
$repository = $this->getDoctrine()
->getRepository('AcmeGearDBBundle:TbGear');
$query = $repository->createQueryBuilder('p')
->select('p', 'q')
->innerJoin('p.fkShip', 'q', 'WITH', 'p.fkShip = q.id')
->getQuery();
$result = $query->getResult();
foreach ( $result as $p ) {
$gear[] = array('shortname' => $p->getGearShortName(), 'name' => $p->getGearName(), 'shipname' => $p->getShipName /* Does not work, since the getter is in a different entity */);
}
return $this->render('AcmeGearDBBundle::index.html.twig', array('gear' => $gear));
}
}
The query generated by this is correct and delivers the expected fields if I execute it in phpmyadmin.
SELECT t0_.GEAR_NAME AS GEAR_NAME0, t0_.GEAR_SHORT_NAME AS GEAR_SHORT_NAME1, t0_.STATUS AS STATUS2, t0_.ID AS ID3, t1_.SHIP_NAME AS SHIP_NAME4, t1_.CONTACT_NAME AS CONTACT_NAME5, t1_.CONTACT_EMAIL AS CONTACT_EMAIL6, t1_.ID AS ID7, t0_.FK_SHIP_ID AS FK_SHIP_ID8, t0_.FK_TYPE AS FK_TYPE9
FROM tb_gear t0_
INNER JOIN tb_dships t1_ ON t0_.FK_SHIP_ID = t1_.ID
AND (t0_.FK_SHIP_ID = t1_.ID)
However, I have no clue how do access those fields in the returned result set. The way I expected it to work ( by accessing the getter of the joined table entity ) does not work. The error message reads: FatalErrorException: Error: Call to undefined method Acme\GearDBBundle\Entity\TbGear::getShipName() in /var/www/symfony/src/Acme/GearDBBundle/Controller/DefaultController.php line 24
which makes sense since the TbGear entity doesn't have a getter method called getShipName() , since that's a method from the joined entity. But how do I access those values? This probably is a stupid question, but I just can't figure it out. Any help is appreciated.
$p->getFkShip()->getShipName() maybe?
This should work since it will retrieve only TbGear that satisfies you relationship. So you could be able to access to all FkShip (I suppose that is a many-to-one relation) that should be only one, and then .... you got it!
EDIT
Of course I suppose that you have correctly designed your class so that you have a getter from TbGear to access the relation with FkShip
Can you add that custom getter: getShipName()?
public function getShipName(){
if ( $this->ship != null ){
return $this->ship->getName();
}
return null; // or an empty string
}
I have a query which is returning a sum, so naturally it returns one row.
I need to count the number of records in the DB which made that sum.
Here's a sample of the type of query I am talking about (MySQL):
SELECT
i.id,
i.vendor_quote_id,
i.product_id_requested,
SUM(i.quantity_on_hand) AS qty,
COUNT(i.quantity_on_hand) AS count
FROM vendor_quote_item AS i
JOIN vendor_quote_container AS c
ON i.vendor_quote_id = c.id
LEFT JOIN company_types ON company_types.company_id = c.company_id
WHERE company_types.company_type = 'f'
AND i.product_id_requested = 12345678
I have found and am now using the select_min(), select_max(), and select_sum() functions, but my COUNT() is still hard-coded in.
The main problem is that I am having to specify the table name in a tightly coupled manner with something like $this->$db->select( 'COUNT(myDbPrefix_vendor_quote_item.quantity_on_hand) AS count' ) which kills portability and makes switching environments a PIA.
How can/should I get my the count values I am after with CI in an uncoupled way??
If you want a completely decoupled way of dealing with this, just run the query to get all the rows you'd add with SUM() and then add them together in PHP.
$sum = 0;
foreach($query->result() as $row)
{
$sum += $row->quantity_on_hand;
}
Or something like that.
What about defining your table in a var or const and then doing the query like so:
define('VENDOR_QUOTE_ITEM', 'vendor_quote_item');
$this->$db->select( 'COUNT(' . VENDOR_QUOTE_ITEM . '.quantity_on_hand) AS count' );
This should be faster than $query->num_rows() as that would retrieve results and have PHP count them. The above code cuts to the chase and just asks the DB for the count without returning anything else (because it uses mysql's COUNT())
As for why $query->num_rows(); isn't working.. Make sure that var you call num_rows on a CI query result object. You should have something like this:
$your_query = $this->db->query("YOUR QUERY");
$your_query->num_rows()
if you would like to use any MySQL function inside $this->db->select() function pass the second parameter as FALSE.
So it should be $this->$db->select( 'COUNT(myDbPrefix_vendor_quote_item.quantity_on_hand) AS count' , FALSE)
Well ... while it's a different direction than I initially envisioned, I ended up simply extending CI via the directions found HERE.
I added a select_count() method to match the existing select_min(), select_max(), and select_sum() methods.
This addition only applies to MySQL at this time, but it's a solid solution.
In case someone encounters a similar problem in the future, here's what I did:
I dropped Simons "MY_Loader" directly into my "application/core"
directory (didn't need to change a thing).
Then I created a "MY_DB_mysql_driver" in the "application/core" directory,
as per his instructions ... and made it looke like this: (sans comments for brevity)
.
<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');
class My_DB_mysql_driver extends CI_DB_mysql_driver {
final public function __construct($params) {
parent::__construct($params);
log_message('debug', 'Extended DB driver class instantiated!');
} /* method: __CONSTRUCT */
final public function select_count($select = '', $alias = ''){
if ( !is_string($select) OR $select == ''){
$this->display_error('db_invalid_query');
}
if ($alias == ''){
$alias = $this->_create_alias_from_table(trim($select));
}
$sql = 'COUNT('.$this->_protect_identifiers(trim($select)).') AS '.$alias;
$this->ar_select[] = $sql;
if ($this->ar_caching === TRUE){
$this->ar_cache_select[] = $sql;
$this->ar_cache_exists[] = 'select';
}
return $this;
} /* method: SELECT_COUNT */
}
Hope it helps.
I have the following:
public function getAll($limit = 100)
{
//if ($thread != 0) { $threadq = "WHERE threadId=$threadId"; }
$query = <<<EOF
SELECT
x.*
FROM x
ORDER BY dater DESC
LIMIT ?
EOF;
return self::$db->fetchAll($query, $limit);
}
It seems it turns the LIMIT x into LIMIT 'x' and so the MySQL query parser goes wrong and complains.
However doing LIMIT $limit works fine
Why does this not work? and is there another method of doing this?
The replacement parameters must be inside array even if only single:
return self::$db->fetchAll($query, array($limit));
And for limit part of query you don't need to use ? replacement wildcard!
I'm going to go crazy and suggest you keep it easy and use Zend_Db_Select. This appears to be a simple query.
This demonstration uses a default ZF DbTable as the the adapter (I could just as easily use Zend_Db_Table::getDefaultAdapter();), however it can be adapted to almost any type of query:
<?php
class Application_Model_DbTable_User extends Zend_Db_Table_Abstract
{
protected $_name = 'user';
protected $_primary = 'id';
public function getAll($limit = 100)
{
$select = $this->select();//This is the adapter for this table and implies SELECT * FROM user
$select->order('name', 'DESC') //can also pass an array of columns
->limit($limit);//limit has a second arg for offset
return $this->fetchAll($select);//returns a rowset object, if an array is required append ->toArray to the fetchall().
}
}