Hallo,
I have sort of daily random image system build with Codeigniter and Doctrine.
There is key function, which returns daily image or sets new one. I am using recursive calling, but sometimes are two images selected for one day.
class Daily extends Controller {
function find_daily(){
$connection = Doctrine_Manager::connection();
$connection->setCharset('utf8');
$q = Doctrine_Query::create()
->select('COUNT(id) as dailycount')
->from('Image')
->where('daily_date = ?',date("Y-m-d"));
$set = $q->execute();
if( $set[0]->dailycount != 0 ){ //theres one or more for today
$q = Doctrine_Query::create()
->select('id')
->from('Image')
->where('daily_date = ?',date("Y-m-d"))
->limit(1);
$set = $q->execute();
$i = Doctrine::getTable('Image')->findOneById($set[0]["id"]);
return $i;
}else { // none image selected with today date
$q = Doctrine_Query::create()
->select('id')
->from('Image')
->where('daily_date = ?', "0000-00-00")
->andWhere("dir =?","queue")
->orderBy("rand()")
->limit(1);
$set = $q->execute();
$i = Doctrine::getTable('Image')->findOneById($set[0]["id"]);
$i->daily_date = date("Y-m-d");
$i->dir = "archive";
$i->save();
$this -> find_daily();
};
}
...
I think theres some stupid mistake.. maybe calling something like "Doctrine::doAllAndClear" before $this -> find_daily(); or I forget something...
Or is better not to use recursive calls?
Thanks.
Well, I've not tested this at all, but I'd probably be more likely to write your code more like this; hopefully this will give you some ideas:
function find_daily() {
$connection = Doctrine_Manager::connection();
$connection->setCharset('utf8');
// Make sure "today" doesn't change while the script runs
$today = date("Y-m-d");
// Find an image with today's date.
$q = Doctrine_Query::create()
->from('Image')
->where('daily_date = ?', $today)
->limit(1);
$image = $q->fetchOne();
if (!$image) {
// There was no image with today's date, so pick one from the queue at random instead.
$q = Doctrine_Query::create()
->from('Image')
->where('daily_date = ?', "0000-00-00")
->andWhere("dir =?","queue")
->orderBy("rand()")
->limit(1);
$image = $q->fetchOne();
// ToDo: Cope with there being no image in the queue
$image->setDailyDate($today);
$image->setDir("archive");
$image->save();
}
return $image;
}
However, you need to think about what happens if two people hit your script at the same time. In that case, they may both fetch a new image for the day (because they could each run the first SELECT at the same time, and both get a result that says there's no daily image yet.)
To avoid that, I'd guess you'd want to wrap this in a transaction, and set a transaction isolation level such that the first caller of the function "wins" the race condition, and subsequent callers block until the update is safely saved. Not entirely sure how you'd do that with Doctrine, but it shouldn't be hard.
Checking the manual, a beginTransaction() at the start, a commit() at the end, and a setup before everything else involving $tx = $conn->transaction; $tx->setIsolation('SERIALIZABLE'); (otherwise two users could probably still run the SELECT at once) would probably be all you need, but you might want to wait for someone who really knows Doctrine to comment on that...
I also saved "today"'s date at the start of the script, otherwise there's a danger of it being different during the SELECT and the later UPDATE, if the script runs across midnight.
Related
I'm using the query builder in my Laravel 8 project to create a monthly sum of all of the deleted users in my application, I'm then outputting two items to use as part of a graph, total and date.
This works well, but, if a month didn't have any data then it would skip straight onto the next month, e.g:
2021-01
2021-04
2021-05
How can I modify the query to add all of the months, from a given start date, up until "now" and effectively add blank values for those months that don't have data?
My current query is:
$data = User::selectRaw('DATE_FORMAT(created_at, "%Y-%m") as date, COUNT(*) as total')
->groupByRaw('DATE_FORMAT(created_at, "%Y-%m")')
->withTrashed()
->whereNotNull('deleted_at')
->get();
And I'm thinking of calculating the start by doing something like this:
$user = User::orderBy('created_at', 'asc')->first();
$start = $user->created_at;
$data = User::selectRaw('DATE_FORMAT(created_at, "%Y-%m") as date, COUNT(*) as total')
->groupByRaw('DATE_FORMAT(created_at, "%Y-%m")')
->withTrashed()
->whereNotNull('deleted_at')
->get();
$end = Carbon::now()->endOfMonth();
Not sure how to get it into the query though
The problem here is that groups originate from the rows, not the other way around. A group will not exist unless a row exists to be included within the group. You only see "missing" months because, in your mind, there are months between January and April.
I'd recommend doing it in post-processing, because any clever attempts to create phantom rows so that groups appear will inevitably be more complicated and more frustrating to maintain.
It may feel clunky, but looping through months from Carbon and adding values to your query result will work fine. Plus, you don't need to rely on $start from your first user result, you can set it yourself.
$start = Carbon::today()->subYear(); // Use any start date, even include it from user input (like a datepicker).
$end = Carbon::today(); // Use any end date, though it won't be useful any later than today.
$loop = $start->copy();
while ($loop->lessThanOrEqualTo($end)) {
$exists = $data->first(function($item) use ($end) {
return $item->date == $end->format('Y-m');
});
if (!$exists) {
$row = new stdClass();
$row->date = $loop->copy()->format('Y-m');
$row->total = 0;
$data->push($row);
}
$loop->addMonth(); // This keeps the loop going.
}
This accomplishes what you want and doesn't get into any N+1 issues.
Edit: Added example below in re-usable function.
function fillEmptyMonths(Collection $data, Carbon $start, Carbon $end): Collection
{
$loop = $start->copy();
// Loop using diff in months rather than running comparison over and over.
for ($months = 0; $months <= $start->diffInMonths($end); $months++) {
if ($data->where('date', '=', $loop->format('Y-m'))->isEmpty()) {
$row = new stdClass();
$row->date = $loop->copy()->format('Y-m');
$row->total = 0;
$data->push($row);
}
$loop->addMonth();
}
return $data;
}
You could also expand this to take another parameter that defines the increment (and pass it "month", "day", "year", etc.). But if you are only using month, this should work.
I am currently doing a event scheduling module for my system. I want to count all the scheduled events for all months.. For example I have 10 events for this march, then 5 incoming events in April but I am encountering error "A Database Error Occurred"
CONTROLLER
$data['getAll'] = $this->CrudModel->count_all('events');
MODEL
public function count_all($table)
{
// $this->db->select('service');
// $this->db->from($table);
// $this->db->where('date LIKE','%'.$month.'%'); // 2017-03-04
// $num_results = $this->db->count_all_results();
// return $num_results->result();
$query = $this->db->query("SELECT date, service from $table");
foreach ($query->result() as $row)
{
# code...
$date = $row->date;
$service = $row->service;
$date_explode = explode('-', $date);
$year = $date_explode[0];
$month = $date_explode[1];
$day = $date_explode[2];
$service_explode = explode(',', $service);
echo "<pre>";
print_r($date_explode);
print_r($service_explode);
echo "</pre>";
$this->db->like('date',$month); // 2017-03-04
$num_results = $this->db->count_all_results();
}
// return $query->result();
}
Question: Is my query wrong? If yes what is it? Or any other suggestion how to count all the scheduled events?
NOTE: I only used one date.. The scheduled date(Eg. I scheduled the event in 2017-03-04), i dont have end date(cause I used the date input type in html)
you must specify table name
$this->db->count_all_results("table");
Active Record Documentation for count_all_results();
Looks to me like the query is missing a FROM clause and tablename. (Or an inline view query in place of an identifier.)
MySQL (or MariaDB) Server is reporting a syntax error, flagging the the WHERE keyword. MySQL is expecting the statement to have FROM clause, and is not finding it.
The error message reports that an invalid statement was issued, looks like this:
SELECT COUNT(*) AS `numrows` WHERE ...
What's missing is the FROM keyword and a table_name:
SELECT COUNT(*) AS `minimums` FROM some_row_source WHERE ...
^^^^^^^^^^^^^^^^^^^^
Maybe $table is being evaluated as an empty string?
For debugging this, I'd suggest first figuring out which line(s) of code is causing this SQL statement to be executed.
I have table with 100 000+ rows, and I want to select all of it in doctrine and to do some actions with each row, in symfony2 with doctrine I try to do with this query:
$query = $this->getDefaultEntityManager()
->getRepository('AppBundle:Contractor')
->createQueryBuilder('c')
->getQuery()->iterate();
foreach ($query as $contractor) {
// doing something
}
but then I get memory leak, because I think It wrote all data in memory.
I have more experience in ADOdb, in that library when I do so:
$result = $ADOdbObject->Execute('SELECT * FROM contractors');
while ($arrRow = $result->fetchRow()) {
// do some action
}
I do not get any memory leak.
So how to select all data from the table and do not get memory leak with doctrine in symfony2 ?
Question EDIT
When I try to delete foreach and just do iterate, I also get memory leak:
$query = $this->getDefaultEntityManager()
->getRepository('AppBundle:Contractor')
->createQueryBuilder('c')
->getQuery()->iterate();
The normal approach is to use iterate().
$q = $this->getDefaultEntityManager()->createQuery('select u from AppBundle:Contractor c');
$iterableResult = $q->iterate();
foreach ($iterableResult as $row) {
// do something
}
However, as the doctrine documentation says this can still result in errors.
Results may be fully buffered by the database client/ connection allocating additional memory not visible to the PHP process. For large sets this may easily kill the process for no apparant reason.
The easiest approach to this would be to simply create smaller queries with offsets and limits.
//get the count of the whole query first
$qb = $this->getDefaultEntityManager();
$qb->select('COUNT(u)')->from('AppBundle:Contractor', 'c');
$count = $qb->getQuery()->getSingleScalarResult();
//lets say we go in steps of 1000 to have no memory leak
$limit = 1000;
$offset = 0;
//loop every 1000 > create a query > loop the result > repeat
while ($offset < $count){
$qb->select('u')
->from('AppBundle:Contractor', 'c')
->setMaxResults($limit)
->setFirstResult($offset);
$result = $qb->getQuery()->getResult();
foreach ($result as $contractor) {
// do something
}
$offset += $limit;
}
With this heavy datasets this will most likely go over the maximum execution time, which is 30 seconds by default. So make sure to manually change set_time_limit in your php.ini. If you just want to update all datasets with a known pattern, you should consider writing one big update query instead of looping and editing the result in PHP.
Try using this approach:
foreach ($query as $contractor) {
// doing something
$this->getDefaultEntityManager()->detach($contractor);
$this->getDefaultEntityManager()->clear($contractor);
unset($contractor); // tell to the gc the object is not in use anymore
}
Hope this help
If you really need to get all the records, I'd suggest you to use database_connection directly. Look at its interface and choose method which won't load all the data into memory (and won't map the records to your entity).
You could use something like this (assuming this code is in controller):
$db = $this->get('database_connection');
$query = 'select * from <your_table>';
$sth = $db->prepare($query);
$sth->execute();
while($row = $sth->fetch()) {
// some stuff
}
Probably it's not what you need because you might want to have objects after handling all the collection. But maybe you don't need the objects. Anyway think about this.
I have the function getDistance(). The function findDistance() inside the while loop, calculates the distance between 2 users, by using coordinates (latitude-longitude), and returns to var $djson the distance in meters. $distance is a string committed by the user for first time and $user_old["distance"] is a string which is called from a database in $query. I wanted to be able in $matched_names, to save all the names of the users from my database, for who the condition inside if() is true, regarding the sum of the distance of the new user who commits his data and the old ones inside the database. The problem is that $matched_names saves the first name which is called from the database and for as many times the loop goes on without even considering the if() restriction. For example if the first name called in $user is "Mike", and $user has 5 rows then the output will be: Mike,Mike,Mike,Mike,Mike.
I suppose that i have made some mistake in the way things work inside while..
<?php
public function getDistance($uuid, $name, $distance, $latstart, $lonstart, $latend, $lonend, $gcm_regId) {
$query = sprintf("SELECT uid, gcm_regid, name, distance,latstart, lonstart, latend, lonend FROM user_demand WHERE latstart='%s' AND lonstart='%s'",
mysql_real_escape_string($latstart),
mysql_real_escape_string($lonstart));
$user = mysql_query($query);
$no_of_rows = mysql_num_rows($user);
$user_old = mysql_fetch_assoc($user);
while( $user_old = mysql_fetch_assoc($user)) {
$djson = $this->findDistance($latend,$lonend,$user_old["latend"],$user_old["lonend"] );
if ($user_old["distance"] + $distance >= $djson) {
$match = $this ->df->addUserMatch($user_old['gcm_regid'],$user_old['name'],$gcm_regId,$name);
$matched_names = array_fill(0,$no_of_rows,$user_old['name']);
$matched_gcmz = array_fill(0,$no_of_rows,$user_old['gcm_regid']);
}
}
$registatoin_ids = array($gcm_regId);
$message = array("names" =>$matched_names,"gcm" => $matched_gcmz);
$result = $this ->gcm->send_notification($registatoin_ids, $message);
}
?>
What i usually do is, when I'm going to write something that is complicated, is to get it to working without being in a function then break it down into functions.
It is less confusing and easier to troubleshoot problems that way.
It is kind of hard to tell what it is doing since you didn't post your other function and the if statement relies on the output of that function.
Offhand, it could be this line where you are using the $name and $user_old['name']
$match = $this ->df->addUserMatch($user_old['gcm_regid'],$user_old['name'],$gcm_regId,$name);
I think you would want it to match each other. Like if $name = $user_old['name'] then add it, if not, do something else.
I have ran into a problem...
I have a bunch of where statments like so...
$this->db->where('Pool', "1");
$this->db->where('Bedrooms >=', "3");
Then a limit statement
$this->db->limit($limit, $offset);
And finally my get statement
$query = $this->db->get('table-name');
My problem is I need to count the results before my limit statement, to get the total rows without the limit.. So I tried this..
$this->db->where('Pool', "1");
$this->db->where('Bedrooms >=', "3");
$num_rows = $this->db->count_all_results();
$this->db->limit($limit, $offset);
$query = $this->db->get('table-name');
This counts my rows with the where statements fine.. However, the get statement now gets records without the previous where statements working.
It's not visible, but there is a large amount of code handling more where statements, and grabbing things in urls, So I'd prefer not to perform the retrieval of data twice in order to fix this...
Cheers!
I know this is an old question, but I just ran into this problem and came up with a different solution.
The idea is to take a copy of the db class before the limit and offset.
$this->db->where('Pool', "1");
$this->db->where('Bedrooms >=', "3");
//here we use the clone command to create a shallow copy of the object
$tempdb = clone $this->db;
//now we run the count method on this copy
$num_rows = $tempdb->from('table-name')->count_all_results();
$this->db->limit($limit, $offset);
$query = $this->db->get('table-name');
I know that's an old question but I found a pretty simple solution.
//do your select, from and where
$this->db->select('your selects');
$this->db->from('your table');
$this->db->where('your where');
//get the filtered rows count
//the trick is the first empty parameter and second false parameter
$filtered_count = $this->db->count_all_results('', false);
//limit your results and get the rows
$this->db->limit($length, $start);
$results = $this->db->get()->result_array();
Hope it helps someone
$get_data = $this->your_model->get_data();
$data = $get_data['data'];
$count = $get_data['count'];
Model
function get_data($limit = 10, $offset= 0)
{
$table = 'table-name';
$where = array('Pool' => 1, 'Beedrooms >=' 3);
$return['data'] = $this->db->from($table)->where($where)->limit($limit, $offset)->get();
$return['count'] = $this->db->from($table)->where($where)->count_all_results();
return $return;
}
$this->db->select('*');
$this->db->from('users');
$this->db->where('active',$status);
//open1 here we copy $this->db in to tempdb and apply
//count_all_results() function on to this tempdb
$tempdb = clone $this->db;
$num_results= $tempdb->count_all_results();
// now applying limit and will get actual result
$this->db->limit(10);
$this->db->get();
$query = $this->db->last_query();
$res = $this->db->query($query);
$data_array = array('num_results' => $num_results, 'results' => $res->result() );
return $data_array;
It is quite evident that you would need to use two different queries. It would be optimum to do this as quickly as possible using a single query, but since you need to get all the records before the second query, we need to use two queries.
However, you can optimize the first query based on the engine you use with MySQL. If you use InnoDB then you should use SELECT COUNT(*) FROM <table-name> cause the total row size is cached in InnoDB.
I believe count_all_rows uses count(*) for performance and you should be sorted using this direcctly.
With MyISAM you can use COUNT(<column-name>).
So, you have a count function in your model class which returns the count for your table and then you can call the function to insert/update/get data from your database.
You can use SQL_CALC_FOUND_ROWS of mysql
SELECT SQL_CALC_FOUND_ROWS * FROM table1
WHERE
cond1, cond2, ..., condN
LIMIT 10
SELECT FOUND_ROWS();
This is using mysql, you may need to check how to use it codeignitor way.
Reference:
https://dev.mysql.com/doc/refman/5.7/en/information-functions.html#function_found-rows
I know this question is old but I had the same issue and got a simpler solution than the displayed here. No need to build the query twice and no need to clone the db class. Just count the result without resting the query builder after you add the where part and then you add the limit and execute your query.
$this->db->where('Pool', 1);
$this->db->where('Beedrooms >=' 3);
$count = $this->db->count_all_results('table-name', false); // No reset query builder
$this->db->limit($limit, $offset)
$result = $this->db->get();
You are going to have two variables:
$count with the number of results and $result with the result.