I'm writing a very big extbase extension and have to handle some complicated filter functions. It would be a one liner in pure sql, however, I have to use Extbase ORM for various reasons.
This is about performances of theater plays. Every play has multiple performances, the play is the aggregate root.
I have to find performances within a given time range and that's a no-brainer. However, the category is a child of the parent (play). I do not save a reference of the play in the performance, but in the play (that's because the TCA has to be configured as select-type).
I have to forward the result to the paginator widget, which requires an instance of the QueryResultInterface. When I use the plays as aggregate root and extract the performances from there, it returns ObjectStorage and breaks the widget.
Is it somehow possible to restrict the output if a child of a parent category matches or am I assume right: there's no reasonable chance?
public function findByTimeRangeInBeginTimeAndAttendCategoryAndStartOfBooking(Datetime $startRange, Datetime $endRange, $category = 'undefined') {
$query = $this->createQuery();
$constraints = array();
$constraints[] = $query->greaterThanOrEqual('timebegin', $startRange->getTimestamp());
$constraints[] = $query->lessThanOrEqual('timebegin', $endRange->getTimestamp());
if($category !== 'undefined') {
$constraints[] = $query->equals('play.category.name', $category);
}
$query->matching (
$query->logicalAnd($constraints)
);
$query->setOrderings (
Array('timebegin' => Tx_Extbase_Persistence_Query::ORDER_ASCENDING)
);
return $query->execute();
}
I don't think there is much chance without a reference to the play. I would even say your domain model cannot work out this way. Either you have play as aggregate root and then select the plays and go from there to the performances or the performance has a repository too and in that sense becomes an aggregate root itself, but then you would want to have a reference to the play...
I have to use Extbase ORM for various reasons.
What reasons? Are you sure you really have to?
It won't solve your problem, but you should read this:
http://lists.typo3.org/pipermail/typo3-dev/2011-May/043180.html
I really like the idea to bring MVC and ORM to TYPO3 and I am using Extbase myself, but I'd go with a "regular" TYPO3-Extension or another framework if I had to write complex database-driven stuff.
Related
First the actual state: There is a ZF2 application with a form. The form contains some autocomplete fields (implemented on the frontend side with jQuery Autocomplete).
The SQL statements behind it look like this:
SELECT name FROM countries WHERE countries.name LIKE %en%
-> should find "Arg[en]tina", "Arm[en]ia", "B[en]in", "Turkm[en]istan" etc.
or
SELECT name FROM countries WHERE countries.continent_id = 2
-> should find "Afghanistan", "Armenia", "Azerbaijan" etc. (2 = Asia)
or
SELECT name FROM countries WHERE countries.continent_id = 2 AND countries.name LIKE %en%
-> should find "Arm[en]ia", "Turkm[en]istan" etc. (2 = Asia)
Of course, it leads to the problem, that the database gets terrorized by a lot of small autocomplete requests. Caching should help -- and I've already started implementing a Zend\Cache\Storage\Adapter\Apcu-based caching mechanism. But then I saw the next trouble: Using a common cache like APCu I cannot filter the results dynamically. So such a cache seems not to work for a case with autocomplete.
I'm pretty sure, that it's a common issue and there is already a solution for this.
How to realize a caching mechanism in a ZF2 application for the autocomplete functionality?
There is nothing to do with the ZF2 here. This is all about a custom search service design and it's challenges.
Without a proper caching layer and/or full-text search engine, building an autocomplete implementation such this would be suicide for the application. You can easily achieve tens of thousands of unnecessarily repeated queries in a very short time.
In an "ideal" world, a good autocomplete implementation utilizes a full-text search engine under the hood such as Elasticsearch or Apache Solr. And uses their Completion Suggester and Suggester components respectively.
Anyway a simple autocompletion feature still achievable using only an object cache and database. Only you need is a helper method to create a proper "cache key" for the every letter combinations. For example:
function createKeyByQuery($str)
{
return 'autocomplete-prefix-'.(string) $str;
}
and in your suggest() method:
public function suggest($keyword)
{
$key = $this->createKeyByQuery($keyword);
if($this->cache->hasItem($key)) {
return $this->cache->getItem($key);
}
// fetch form the database here
$data = $this->db->query();
$this->cache->setItem($key, $data);
return $data;
}
If the count of your filters are not too much, just make them part of the key too. In this scenario, signature of the suggest method would be:
public function suggest($keyword, array $filters = []);
and key generator needs an update:
function createKeyByQuery($str, array $filters = [])
{
return 'autocomplete-prefix-' . (string) $str . md5(serialize($filters));
}
This solution may not appropriate for the complicated/domain related data because it has a pretty big invalidation challenge. Eg. how you would find the cache keys which holds the "Argentina" in the payload?
Since you're dealing only with the list of countries and continents as filter, it should solve the problem.
For the england keyword and two different filter with total of 10 filter options, there will be 10x2x7 = 140 distinct cache key(s). For a single filter with 5 option, 5x1x7 = 37 distinct keys.
APCu is a good choice for this implementation.
Hope it helps.
According to FOSElasticaBundle documentation it is possible to configure application to use custom query builder method like this:
user:
persistence:
elastica_to_model_transformer:
query_builder_method: createSearchQueryBuilder
But is it possible to choose QB method live, e.g. in controller action?
I'd like to be able to control what's being fetched from DB while transforming Elastica results to Doctrine entities. E.g. sometimes I'll want to do eager fetch on some relations, but can't do that by default.
Since FOSElasticaBundle documentation is not very precise, I went through its code and found it impossible to control what query builder is used on controller level.
It is possible to change whole elastica_to_model_transformer to a custom service, but still it's statically defined in configuration. Maybe with some dirty solution it would be possible going this way, but I don't think it's worth it.
I decided to just not using this feature of FOSElasticaBundle. The main problem I had was that when you use fos_elastica.index instead of fos_elastica.finder or elastica repository (in order to get plain not transformed results Elastica\Resultset), there's no findPaginated method with returns Pagerfanta paginator object, which is very helpful in my case.
Fortunately although it's not mentioned in documentation it's possible to create the Pagerfanta this way too, but a little bit more manually.
Here's a code snippet:
//generate ElaticaQuery somehow.
$browseQuery = $browseData->getBrowseQuery();
$search = $this->container->get('fos_elastica.index.indexName.typName');
//create pagerfanta's adapter manually
$adapter = new \Pagerfanta\Adapter\ElasticaAdapterElasticaAdapter($search, $browseQuery);
// now you can create the paginator too.
$pager = new Pagerfanta($adapter);
//do some paging work on it...
$pager->setMaxPerPage($browseData->getPerPage());
try {
$pager->setCurrentPage($browseData->getPage());
} catch(OutOfRangeCurrentPageException $e) {
$pager->setCurrentPage(1);
}
//and get current page results.
/** #var Result[] $elasticaResults */
$elasticaResults = $pager->getCurrentPageResults();
// we have to grab ids manyally, but it's done the same way inside FOSElasticaBundle with previous approach
$ids = array();
foreach($elasticaResults as $elasticaResult) {
$ids[] = $elasticaResult->getId();
}
//use regular Doctrine's repository to fetch Entities any way you want.
$entities = $this->getDoctrine()->getRepository(MyEntity::class)->findByIdentifiers($ids);
This actually has a few advantages. In general it gives you back control over your data and doesn't tie ElasticSearch with Doctrine. Therefore you can resign on fetching data from Doctrine if you have all needed data in ElasticSearch (if they are read only data of course). This lets you optimize your application performance but reducing amount of SQL queries.
The code above may be wrapped with some kind of service in order to prevent making mess in controllers.
I'm working on an application written in PHP. I decided to follow the MVC architecture.
However, as the code gets bigger and bigger, I realized that some code gets duplicated in some cases. Also, I'm still confused whether I should use static functions when quering the database or not.
Let's take an example on how I do it :
class User {
private id;
private name;
private age;
}
Now, inside this class I will write methods that operate on a single user instance (CRUD operations). On the other hand, I added general static functions to deal with multiple users like :
public static function getUsers()
The main problem that I'm facing is that I have to access fields through the results when I need to loop through users in my views. for example :
$users = User::getUsers();
// View
foreach($users as $user) {
echo $user['firstname'];
echo $user['lastname'];
}
I decided to do this because I didn't feel it's necessary to create a single user instance for all the users just to do some simple data processing like displaying their informations. But, what if I change the table fields names ? I have to go through all the code and change those fields, and this is what bothers me.
So my question is, how do you deal with database queries like that, and is it fine to use static functions when querying the database. And finally, where is it logical to store those "displaying" functions like the one I talked about ?
Your approach seems fine, howerver I would still use caching like memcached to cache values and then you can remove static.
public function getUsers() {
$users = $cacheObj->get('all_users');
if ($users === false) {
//use your query to grab users and set it to cache
$users = "FROM QUERY";
$cacheObj->set('all_users', $users);
}
return $users;
}
(M)odel (V)iew (C)ontroller is a great choice choice, but my advice is look at using a framework. The con is they can have a step learning curve, pro is it does a lot of heavy lifting. But if you want to proceed on your own fair play, it can be tough to do it yourself.
Location wise you have a choice because the model is not clearly define:
You'll hear the term "business logic" used, basically Model has everything baring views and the controllers. The controllers should be lean only moving data then returning it to the view.
You model houses DB interaction, data conversions, timezone changes, general day to day functions.
Moudle
/User
/Model
/DB or (Entities and Mapper)
/Utils
I use Zend and it uses table gateways for standard CRUD to avoid repetition.
Where you have the getUsers() method you just pass a array to it, and it becomes really reusable and you'd just have different arrays in various controller actions and it builds the queries for you from the array info.
Example:
$data = array ('id' => 26)
$userMapper->getUsers($data);
to get user 26
enter code here
$data = array ('active' => 1, 'name' => 'Darren')
$userMapper->getUsers($data);`
to get active users named Darren
I hope this help.
I'm querying big chunks of data with cachephp's find. I use recursive 2. (I really need that much recursion sadly.) I want to cache the result from associations, but I don't know where to return them. For example I have a Card table and card belongs to Artist. When I query something from Card, the find method runs in the Card table, but not in the Artist table, but I get the Artist value for the Card's artist_id field and I see a query in the query log like this:
`Artist`.`id`, `Artist`.`name` FROM `swords`.`artists` AS `Artist` WHERE `Artist`.`id` = 93
My question is how can I cache this type of queries?
Thanks!
1. Where does Cake "do" this?
CakePHP does this really cool but - as you have discovered yourself - sometimes expensive operation in its different DataSource::read() Method implementations. For example in the Dbo Datasources its here. As you can see, you have no direct 'hooks' (= callbacks) at the point where Cake determines the value of the $recursive option and may decides to query your associations. BUT we have before and after callbacks.
2. Where to Cache the associated Data?
Such an operation is in my opinion best suited in the beforeFind and afterFind callback method of your Model classes OR equivalent with Model.beforeFind and Model.afterFind event listeners attached to the models event manager.
The general idea is to check your Cache in the beforeFind method. If you have some data cached, change the $recursive option to a lower value (e.g. -1, 0 or 1) and do the normal query. In the afterFind method, you merge your cached data with the newly fetched data from your database.
Note that beforeFind is only called on the Model from which you are actually fetching the data, whereas afterFind is also called on every associated Model, thus the $primary parameter.
3. An Example?
// We are in a Model.
protected $cacheKey;
public function beforeFind($query) {
if (isset($query["recursive"]) && $query["recursive"] == 2) {
$this->cacheKey = genereate_my_unique_query_cache_key($query); // Todo
if (Cache::read($this->cacheKey) !== false) {
$query["recursive"] = 0; // 1, -1, ...
return $query;
}
}
return parent::beforeFind($query);
}
public function afterFind($results, $primary = false) {
if ($primary && $this->cacheKey) {
if (($cachedData = Cache::read($this->cacheKey)) !== false) {
$results = array_merge($results, $cachedData);
// Maybe use Hash::merge() instead of array_merge
// or something completely different.
} else {
$data = ...;
// Extract your data from $results here,
// Hash::extract() is your friend!
// But use debug($results) if you have no clue :)
Cache::write($this->cacheKey, $data);
}
$this->cacheKey = null;
}
return parent::afterFind($results, $primary);
}
4. What else?
If you are having trouble with deep / high values of $recursion, have a look into Cake's Containable Behavior. This allows you to filter even the deepest recursions.
As another tip: sometimes such deep recursions can be a sign of a general bad or suboptimal design (Database Schema, general Software Architecture, Process and Functional flow of the Appliaction, and so on). Maybe there is an easier way to achieve your desired result?
The easiest way to do this is to install the CakePHP Autocache Plugin.
I've been using this (with several custom modifications) for the last 6 months, and it works extremely well. It will not only cache the recursive data as you want, but also any other model query. It can bring the number of queries per request to zero, and still be able to invalidate its cache when the data changes. It's the holy grail of caching... ad-hoc solutions aren't anywhere near as good as this plugin.
Write query result like following
Cache::write("cache_name",$result);
When you want to retrieve data from cache then write like
$results = Cache::read("cache_name");
I'm trying to create my own xml sitemap. Everything is done except for the part that I thought was going to be the easiest. How do you get a list of all the pages on the site? I have a bunch of views in a /site folder and a few others. Is there a way to explicitly request their URLs or perhaps via the controllers?
I do not want to make use of an extension
You can use reflection to iterate through all methods of all your controllers:
Yii::import('application.controllers.*');
$urls = array();
$directory = Yii::getPathOfAlias('application.controllers');
$iterator = new DirectoryIterator($directory);
foreach ($iterator as $fileinfo)
{
if ($fileinfo->isFile() and $fileinfo->getExtension() == 'php')
{
$className = substr($fileinfo->getFilename(), 0, -4); //strip extension
$class = new ReflectionClass($className);
foreach ($class->getMethods(ReflectionMethod::IS_PUBLIC) as $method)
{
$methodName = $method->getName();
//only take methods that begin with 'action', but skip actions() method
if (strpos($methodName, 'action') === 0 and $methodName != 'actions')
{
$controller = lcfirst(substr($className, 0, strrpos($className, 'Controller')));
$action = lcfirst(substr($methodName, 6));
$urls[] = Yii::app()->createAbsoluteUrl("$controller/$action");
}
}
}
}
You need to know what content you want to include in your sitemap.xml, I don't really think you want to include ALL pages in your sitemap.xml, or do you really want to include something like site.com/article/edit/1 ?
That said, you may only want the result from the view action in your controllers. truth is, you need to know what you want to indexed.
Do not think in terms of controllers/actions/views, but rather think of the resources in your system that you want indexed, be them articles, or pages, they are all in your database or stored somehow, so you can list them, and they have a URI that identifies them, getting the URI is a matter of invoking a couple functions.
There are two possiblities -
Case 1:
You are running a static website then you can find all your HTML inside 1 folder - protected/views/site/pages
http://www.yiiframework.com/wiki/22/how-to-display-static-pages-in-yii/
Case 2:
Website is dynamic. Tasks such as generating and regenerating Sitemaps can be classified into background tasks.
Running background taks can be achieved by emulating the browser which is possible in linux using - WGET, GET or lynx commands
Or, You can create a CronController as a CConsoleCommand. How to use Commands in YII is shown in link below -
http://tariffstreet.com/yii/2012/04/implementing-cron-jobs-with-yii-and-cconsolecommand/
Sitemap is an XML which lists your site's URL. But it does more than that.
It helps you visualize the structure of a website , you may have
category
subcategories.
While making a useful extension, above points can be kept into consideration before design.
Frameworks like Wordpress provide way to generate categorical sitemap.
So the metadata for each page is stored from before and using that metadata it discovers and group pages.
Solution by Reflection suggested by #Pavle is good and should be the way to go.
Consider there may be partial views and you may or may not want to list them as separate links.
So how much effort you want to put into creating the extension is subject to some of these as well.
You may either ask user to list down all variables in config fie and go from there which is not bad or you have to group pages and list using some techniques like reflection and parsing pages and looking for regex.
For ex - Based on module names you can group them first and controllers inside a module can form sub-group.
One first approach could be to iterate over the view files, but then you have to take into account that in some cases, views are not page destinations, but page sections included in another pages by using CController::renderPartial() method. By exploring CController's Class Reference I came upon the CController::actions() method.
So, I have not found any Yii way to iterate over all the actions of a CController, but I used php to iterate over all the methods of a SiteController in one of my projects and filter them to these with the prefix 'action', which is my action prefix, here's the sample
class SiteController extends Controller{
public function actionTest(){
echo '<h1>Test Page!</h1></p>';
$methods = get_class_methods(get_class($this));
// The action prefix is strlen('action') = 6
$actionPrefix = 'action';
$reversedActionPrefix = strrev($actionPrefix);
$actionPrefixLength = strlen($actionPrefix);
foreach ($methods as $index=>$methodName){
//Always unset actions(), since it is not a controller action itself and it has the prefix 'action'
if ($methodName==='actions') {
unset($methods[$index]);
continue;
}
$reversedMethod = strrev($methodName);
/* if the last 6 characters of the reversed substring === 'noitca',
* it means that $method Name corresponds to a Controller Action,
* otherwise it is an inherited method and must be unset.
*/
if (substr($reversedMethod, -$actionPrefixLength)!==$reversedActionPrefix){
unset($methods[$index]);
} else $methods[$index] = strrev(str_replace($reversedActionPrefix, '', $reversedMethod,$replace=1));
}
echo 'Actions '.CHtml::listBox('methods', NULL, $methods);
}
...
}
And the output I got was..
I'm sure it can be furtherly refined, but this method should work for any of the controllers you have...
So what you have to do is:
For each Controller: Filter out all the not-action methods of the class, using the above method. You can build an associative array like
array(
'controllerName1'=>array(
'action1_1',
'action1_2'),
'controllerName2'=>array(
'action2_1',
'action2_2'),
);
I would add a static method getAllActions() in my SiteController for this.
get_class_methods, get_class, strrev and strlen are all PHP functions.
Based on your question:
1. How do you get a list of all the pages on the site?
Based on Yii's way of module/controller/action/action_params and your need to construct a sitemap for SEO.
It will be difficult to parse automatically to get all the urls as your action params varies indefinitely. Though you could simply get controller/action easily as constructed by
Pavle Predic. The complexity comes along when you have customized (SERF) URL rules meant for SEO.
The next best solution is to have a database of contents and you know how to get each content via url rules, then a cron console job to create all the urls to be saved as sitemap.xml.
Hope this helps!