I built a loop in PHP that makes 500 SQL queries but I would like to merge the 500 SQL queries into one and get the same return (the companies and the count of the users on each company)
Example of PHP code
$companies = array();
foreach ($fortune500Service->listAll() as $c ){
$count = $entityManager
->createQueryBuilder()
->select("count(u)")
->from("AppBundle\Entity\User","u")
->where("u.email LIKE :d")
->setParameter("d", "%#" . $c["Domain"])
->getQuery()->getSingleScalarResult();
if ($count == 0) {
continue;
}
$companies[] = array(
"Domain" => $c["Domain"],
"Company" => "{$c["Company"]} ({$count})",
);
}
return $companies;
Example of 2 SQL queries that I want to merge
Query 1
SELECT
count(u0_.id)
FROM
user u0_
WHERE
u0_.email LIKE '%#company1.com'
Query 2
SELECT
count(u0_.id)
FROM
user u0_
WHERE
u0_.email LIKE '%#company2.com'
I prefer a solution using createQueryBuilder http://symfony.com/doc/current/book/doctrine.html#querying-for-objects-using-doctrine-s-query-builder but I am happy also with an SQL native query.
Use conditional aggregation:
SELECT COUNT(CASE WHEN u0_.email LIKE '%#company1.com' THEN 1 END) as First_cnt,
COUNT(CASE WHEN u0_.email LIKE '%#company2.com' THEN 1 END) as First_cnt
FROM user u0_
Related
I have Service entity with title, tags and description fields. When searching service with QueryBuilder how can I get results sorted by field priority. For example, when I search term php I want get services with php in their title at the top of the list, then services with php in their tags and services with search term in their description as last.
This is part of my Querybuilder:
$qb = $this->createQueryBuilder('service');
$qb->leftJoin('service.tags', 'tag');
$conditions = array($conditions[] = $qb->expr()->eq('service.enabled', true));
$conditions[] = $qb->expr()->eq('service.category', $categoryId);
$term = '%' . $term . '%';
$conditions[] = $qb->expr()->orX(
$qb->expr()->like('service.title', "'$term'"),
$qb->expr()->like('service.description', "'$term'"),
$qb->expr()->like('tag.name', "'$term'")
);
$conditions = call_user_func_array(array($qb->expr(), 'andX'), $conditions);
$qb->where($conditions);
The best way to do this would be to perform a series of UNION statements and then weed out the duplicates at the same time giving weight.
(Unchecked pseudo-SQL to give you the idea):
SELECT id,title,tag,SUM(weight) score
FROM (
SELECT id,title,tag, 100 as weight FROM service WHERE title LIKE '%foo%'
UNION ALL
SELECT id,title,tag, 10 as weight FROM service WHERE tags LIKE '%foo%'
UNION ALL
SELECT id,title,tag, 1 as weight FROM service WHERE description LIKE '%foo%'
) t
GROUP BY id
ORDER BY score DESC /* This sort by probably won't work; might need to do it a different way, but you get the idea */
You can use native query for this. Ex.:
$em = $this->get('doctrine')->getManager();
$sql = "
select *
from service s
where
s.title like '%xxx%'
or s.tags like '%xxx%'
or s.description like '%xxx%'
order by
s.title like '%xxx%' desc,
s.tags like '%xxx%' desc,
s.description like '%xxx%' desc
";
$rsm = new \Doctrine\ORM\Query\ResultSetMappingBuilder($em);
$rsm->addRootEntityFromClassMetadata('\You\Entity\Service\Class', 's');
$query = $em->createNativeQuery($sql, $rsm);
$data = $query->getResult();
dump($data);
http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/native-sql.html how to use sql insted of dql in orm
I would not try get the desired results in only one query. It will be too complicated and you'll spent to much time trying get what you want.
First, I assume you'd like to get a sorted Array of Service entities like this:
/** #var Service[] $results **/
$results = $this->getDoctrine()->getManager()->getRepository('YourBundle:YourServiceRepo')->findSerivesByGivenSerchCriteria($yourConditions);
so you can simply iterate (foreach) through it
If it's the case, concider following:
create three separate methods whitin your service-repository to get
services with search-criteria in their title - first priority
services with search-criteria in their tags - second priority
services with serach-criteria in their desc - third priority
and in your findSerivesByGivenSearchCriteria()method you call all three of them and could combine the found results in any order i'd like
for Example:
public function findSerivesByGivenSerchCriteria( $searchTirm ) {
$foundServicesByTitle = $this->findServicesBySearchCriteriaInTitle( $searachTirm );
$foundServicesByTag = $this->findServicesBySearchCriteriaInTags( $searchTirm );
$foundServicesByDesc = $this->findServicesBySearchCriteriaInDesc( $searchTirm );
// Hier you can combine found results in any order
$results = [];
if( false === empty($foundServicesByTitle ) )
{
// for example with array_merge ...
// NOTE: If you choose array_merge() you have to make sure your $foundServicesByTitle, $foundServicesByTag, $foundServicesByDesc have unique array-indexes
// And you get Results like
// array( 'unique_id_1' => ServiceEntity, 'unique_id_2' => ServiceEntity ... )
$results = array_merge($results, $foundServicesByTitle);
}
// .. do the same with other two
return $results;
}
To get the "uniqness" try to combine Doctrine's INDEX BY and HIDDEN
INDEX BY -> http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/dql-doctrine-query-language.html#using-index-by
INDEX BY in QueryBuilder -> https://stackoverflow.com/a/15120793/348193
HIDDEN -> https://michelsalib.com/2012/03/04/doctrine2-ninja-with-the-hidden-keyword/
Why I wouldn't choose the UNION way?
Since you work with symfony and doctrine UNION isn't the best way. See How to write UNION in Doctrine 2.0
As you can see, UNION isn't supported by Doctrine and you have to use NativeQuery (which could be frustrating with ResultSetMappingBuilder and corect mapping and turning raw-results in to desired entities)
Doing my firsts steps in eloquent, I'm trying to achieve a query that retrieve some data and count some conditions at the same time.
This raw SQL sentence works as expected
SELECT *
FROM photos,
(SELECT COUNT(*) FROM photos where type = 1 and published =1 group by type) as qA,
(SELECT COUNT(*) FROM photos where type = 2 and published =1 group by type) as qP
where product_id = 7
and published = 1
But using Photo model I can't get the same results
Photo::select(*,
Photo::raw('count(*) where type = 1 and published =1 group by type as qA),
Photo::raw('count(*) where type = 2 and published =1 group by type as qP))
->where ('product_id',7)
->where('published', 1)
->get()
Stripping out "raw" sentences, this query works, but I need to count those "type" ocurrences.
Thanks in advance for any guide over it.
Do you really need to get all in one query? Does it make sense for your application? I mean, I guess those types count are not really information about the record(s?) you are fetching (product_id = 7), rather they are general information about the state of the table as a whole.
If this makes sense, you could try
$photo = Photo::where ('product_id',7)
->where('published', 1)
->get();
$types_count = Photo::where('published', 1)
->selectRaw('type, COUNT(*) AS count')
->groupBy('type')
->get()
->keyBy('type');
$qA = $types_count[1]->count;
$qP = $types_count[2]->count;
My task is:
"To take transactions table, grouped row by transaction date and calculate statuses. This manipulations will be formed statistics, wich will be rendered on the page".
This is my method of this statistics generation
public static function getStatistics(Website $website = null)
{
if($website == null) return [];
$query = \DB::table('transactions')->where("website_id", $website->id)->orderBy("dt", "desc")->get();
$transitions = collect(static::convertDate($query))->groupBy("dt");
$statistics = collect();
dd($transitions);
foreach ($transitions as $date => $trans) {
$subscriptions = $trans->where("status", 'subscribe')->count();
$unsubscriptions = $trans->where("status", 'unsubscribe')->count();
$prolongations = $trans->where("status", 'rebilling')->count();
$redirections = $trans->where("status", 'redirect_to_lp')->count();
$conversion = $redirections == 0 ? 0 : ((float) ($subscriptions / $redirections));
$earnings = $trans->sum("pay");
$statistics->push((object)[
"date" => $date,
"subscriptions" => $subscriptions,
'unsubscriptions' => $unsubscriptions,
'prolongations' => $prolongations,
'redirections' => $redirections,
'conversion' => round($conversion, 2),
'earnings' => $earnings,
]);
}
return $statistics;
}
if count of transaction rows below 100,000 - it's all wright. But, if count is above 150-200k - nginx throw 502 bad gateway. What can you advise to me? I'm don't have any expierince in bigdata handling. May be, my impiments has fundamental error?
Big data is never easy, but I would suggest using the Laravel chunk instead of get.
https://laravel.com/docs/5.1/eloquent (ctrl+f "::chunk")
What ::chunk does is select n rows at a time, and allow you to process them bit by bit. This is convenient in that it allows you to stream updates to the browser, but at the ~150k result range, I would suggest looking up how to push this work into a background process instead of handling it on request.
After several days of researching information on this question, I found the right answer:
NOT to use PHP for handling raw data. It's better to use SQL!
In my case, we are using PostgreSQL.
Below, i'll write sql-query which worked for me, maybe it will help someone else.
WITH
cte_range(dt) AS
(
SELECT
generate_series('2016-04-01 00:00:00'::timestamp with time zone, '{$date} 00:00:00'::timestamp with time zone, INTERVAL '1 day')
),
cte_data AS
(
SELECT
date_trunc('day', dt) AS dt,
COUNT(*) FILTER (WHERE status = 'subscribe') AS count_subscribes,
COUNT(*) FILTER (WHERE status = 'unsubscribe') AS count_unsubscribes,
COUNT(*) FILTER (WHERE status = 'rebilling') AS count_rebillings,
COUNT(*) FILTER (WHERE status = 'redirect_to_lp') AS count_redirects_to_lp,
SUM(pay) AS earnings,
CASE
WHEN COUNT(*) FILTER (WHERE status = 'redirect_to_lp') > 0 THEN 100.0 * COUNT(*) FILTER (WHERE status = 'subscribe')::float / COUNT(*) FILTER (WHERE status = 'redirect_to_lp')::float
ELSE 0
END
AS conversion_percent
FROM
transactions
WHERE
website_id = {$website->id}
GROUP BY
date_trunc('day', dt)
)
SELECT
to_char(cte_range.dt, 'YYYY-MM-DD') AS day,
COALESCE(cte_data.count_subscribes, 0) AS count_subscribe,
COALESCE(cte_data.count_unsubscribes, 0) AS count_unsubscribes,
COALESCE(cte_data.count_rebillings, 0) AS count_rebillings,
COALESCE(cte_data.count_redirects_to_lp, 0) AS count_redirects_to_lp,
COALESCE(cte_data.conversion_percent, 0) AS conversion_percent,
COALESCE(cte_data.earnings, 0) AS earnings
FROM
cte_range
LEFT JOIN
cte_data
ON cte_data.dt = cte_range.dt
ORDER BY
cte_range.dt DESC
I need to add WHERE, SELECT, ORDER BY etc...like a chain SQL by means of PHP. I use https://github.com/greenlion/PHP-SQL-Parser to spilt up the SQL in Array and then to form the SQL
i need a intermediate code. i can to add SELECT,WHERE, ORDER BY, ETC.. and i can test if exist in the table, fields, in array of PHPSQLParser.
Example:
$sSql = 'Select a, b, c from table1 inner join table2 on (table1.a = table2.a) where b = 900';
# Return array SQL
$aParser = new PHPSQLParser( $sSql );
$aParser = $aParser->parsed;
// I need to intermediate code add fields, conditions, etc. //
// Create the SQL through the array
$creator = new PHPSQLCreator( $this->aParser );
// Show SQL
echo $creator->created;
The package you are using is missing methods you need to work with the parsed query. If you write your own "query builder" package, you can use this package as a starting point so you don't have to write the code that parses the sql. Or better yet, you can use a different package altogether that has both query parsing and building included.
Here's one: https://github.com/jasny/dbquery-mysql
It can be used in the way you are asking:
<?php
use Jasny\DB\MySQL\Query;
$query = new Query("SELECT * FROM foo LEFT JOIN bar ON foo.bar_id = bar.id WHERE active = 1 LIMIT 25");
if (isset($_GET['page'])) $query->page(3);
$filter = isset($_POST['filter']) ? $_POST['filter'] : array(); // array('type' => 'bike', 'price between ? and ?' => array(10, 20))
foreach ($filter as $field => $value) {
$query->where($field, $value);
}
$result = $mysqli->query($query); // SELECT * FROM foo LEFT JOIN bar ON foo.bar_id = bar.id WHERE (active = 1) AND (`type` = "bike") AND (`price` between 10 and 20) LIMIT 25 OFFSET 50
I need to get the percentage of each possible values in the field column, over the total value of my table.
I found two way to get my result in SQL:
SELECT m.field, sum(m.value) * 100 / t.total
FROM my_table AS m
CROSS JOIN (
SELECT SUM(value) AS total FROM
WHERE year = 2000) t
WHERE m.year = 2000
GROUP BY m.field, t.total
And
SELECT m.field, sum(m.value) * 100 / (SELECT SUM(value) AS total FROM WHERE year = 2000)
FROM my_table AS m
WHERE m.year = 2000
GROUP BY m.field
But both are nested queries, and I don't know how to prepare statments with the Doctrine's QueryBuilder into a nested queries.
Is there a way to do it?
I have been trying to do so using querybuilder and DQL with no success. As it seems, DQL doesn't allow operations with subqueries in SELECT. What I've achieved so far:
$subQuery = $em->createQueryBuilder('m')
->select("SUM(m.value)")
->where("m.year = 2000")
->getDQL();
The following query works though doesn't calculate the percentage:
$query = $em->createQueryBuilder('f')
->select("f.field")
->addSelect(sprintf('(%s) AS total', $subQuery))
->addSelect('(SUM(f.value)*100) AS percentage')
->where("f.year = 2000")
->groupBy("f.field")
->getQuery()
->getResult();
However, if you try to add the division in the select in order to get the percentage and you use the subquery, it simply doesn't work. Looks like the construction it's not allowed in DQL. I've tried with an alias and with the subquery directly and neither of them worked.
Doesn't work:
$query = $em->createQueryBuilder('f')
->select("f.field")
->addSelect(sprintf('(%s) AS total', $subQuery))
->addSelect('(SUM(f.value)*100)/total AS percentage')
->where("f.year = 2000")
->groupBy("f.field")
->getQuery()
->getResult();
Doesn't work either:
$query = $em->createQueryBuilder('f')
->select("f.field")
->addSelect(sprintf('(SUM(f.value)*100)/(%s) AS percentage', $subQuery))
->where("f.year = 2000")
->groupBy("f.field")
->getQuery()
->getResult();
I'd suggest using SQL directly (Doctrine allows it). Using native sql queries and mapping the results would do the trick. There is no disadvantage in doing so.
Documentation
If you find a way of doing it using queryBuilder or DQL, please let me know.
Hope it helps.
yeah! the solution is:
$qs = $this
->createQueryBuilder('h');
$d = $qs ->select($qs->expr()->count('h'));
$e = $d->getQuery()->getScalarResult();
$qs->addSelect('(COUNT(h.id)*100 / :t) AS percentage')->setParameter('t', $e);
$qs->addGroupBy(sprintf('h.%s', $type));
return $qs->getQuery()->getResult();