The PHP IDS system expose uses Monolog to store logs into MongoDB. The following is how it stores a log:
{
"message": "Executing on data 4f2793132469524563fa9b46207b21ee",
"context": [
],
"level": NumberLong(200),
"level_name": "INFO",
"channel": "audit",
"datetime": "1441721696",
"extra": [
]
}
I want to use the auto-delete function in Mongo, and I need the datetime field to store in ISOdate format, like this:
"datetime":ISODate("2015-09-08T17:43:25.678Z")
I look at the class Mongo in \Expose\Log\Mongo(); and this is the part responsible for storing the datetime in seconds format
public function log($level, $message, array $context = array())
{
$logger = new \Monolog\Logger('audit');
try {
$handler = new \Monolog\Handler\MongoDBHandler(
new \MongoClient($this->getConnectString()),
$this->getDbName(),
$this->getDbCollection()
);
} catch (\MongoConnectionException $e) {
throw new \Exception('Cannot connect to Mongo - please check your server');
}
$logger->pushHandler($handler);
$logger->pushProcessor(function ($record) {
$record['datetime'] = $record['datetime']->format('U');
return $record;
});
return $logger->$level($message, $context);
}
I have changed the $record['datetime'] into this
//$record['datetime'] = $record['datetime']->format('U');
$record['datetime'] = new \MongoDate();;
but the time isn't store as ISOdate but this:
"datetime": "[object] (MongoDate: 0.84500000 1441721683)"
Can anyone tell me how to store the datetime in ISODate format?
You are correct to use MongoDate to generate the datetime object. eg:
$date = new MongoDate(strtotime("2015-11-23 00:00:00"));
If you echo this back you will get the format you describe, but internally Mongo should be storing it correctly. To check, you can test:
echo date(DATE_ISO8601, ($date->sec);
which should return it in a ISO readable format.
Then you can write your own PSR-3 compatible logger that stores data in that manner.
$filters = new \Expose\FilterCollection();
$filters->load();
$logger = new \YourCustom\PSR3Logger();
$manager = new \Expose\Manager($filters, $logger);
On your Mongo instance you will want to set the TTL on your field like such:
db.log.ensureIndex( { "datetime": 1 }, { expireAfterSeconds: 3600 } )
For more info on the TTL setting see the Mongo docs: Expire Data from Collections by Setting TTL
I had the same issue on a Symfony2 configuration. I solved by setting a custom formatter that basically does nothing. It worked pretty well in my case since I was storing just scalar stuff, the only exception was the MongoDate that didn't have to be formatted. So it could be that you may need to do some tuning on your side.
Here's the custom formatter:
<?php
namespace AppBundle\Service\Formatter;
use Monolog\Formatter\FormatterInterface;
/**
* Class MongoLoggerFormatter
*
* #package AppBundle\Service
* #author Francesco Casula <fra.casula#gmail.com>
*/
class MongoLoggerFormatter implements FormatterInterface
{
/**
* {#inheritdoc}
*/
public function format(array $record)
{
return $record;
}
/**
* {#inheritdoc}
*/
public function formatBatch(array $records)
{
return $records;
}
}
And here's an excerpt of my exception listener:
/**
* #return \Symfony\Bridge\Monolog\Logger
*/
public function getLogger()
{
return $this->logger;
}
/**
* #return \Monolog\Handler\HandlerInterface|null
*/
private function getMongoHandler()
{
foreach ($this->getLogger()->getHandlers() as $handler) {
if ($handler instanceof MongoDBHandler) {
return $handler;
}
}
return null;
}
/**
* #return \Monolog\Handler\HandlerInterface|null
*/
private function addDefaultMongoHandlerSettings()
{
$mongoHandler = $this->getMongoHandler();
if ($mongoHandler) {
$mongoHandler->setFormatter(new MongoLoggerFormatter());
$mongoHandler->pushProcessor(function (array $record) {
$record['created_at'] = new \MongoDate(time());
return $record;
});
}
return $mongoHandler;
}
Related
I am using a library, and it has the following process to attach many operations to one event:
$action = (new EventBuilder($target))->addOperation($Operation1)->addOperation($Operation2)->addOperation($Operation3)->compile();
I am not sure how to dynamically add operations depending on what I need done.
Something like this
$action = (new EventBuilder($target));
while (some event) {
$action = $action->addOperation($OperationX);
}
$action->compile();
I need to be able to dynamically add operations in while loop and when all have been added run it.
Your proposed solution will work. The EventBuilder provides what is known as a Fluent Interface, which means that there are methods that return an instance of the builder itself, allowing you to chain calls to addOperation as many times as you want, then call the compile method to yield a result. However you are free to ignore the return value of addOperation as long as you have a variable containing an instance of the builder that you can eventually call compile on.
Take a walk with me...
// Some boilerplate classes to work with
class Target
{
private ?string $name;
public function __construct(string $name)
{
$this->name = $name;
}
public function getName(): string
{
return $this->name;
}
}
class Operation
{
private ?string $verb;
public function __construct(string $verb)
{
$this->verb = $verb;
}
public function getVerb(): string
{
return $this->verb;
}
}
class Action
{
private ?Target $target;
private array $operations = [];
public function __construct(Target $target, array $operations)
{
$this->target = $target;
$this->operations = $operations;
}
/**
* Do the things
* #return array
*/
public function run(): array
{
$output = [];
foreach ($this->operations as $currOperation)
{
$output[] = $currOperation->getVerb() . ' the ' . $this->target->getName();
}
return $output;
}
}
Here is a basic explanation of what your EventBuilder is doing under the covers:
class EventBuilder
{
private ?Target $target;
private array $operations = [];
public function __construct(Target $target)
{
$this->target = $target;
}
/**
* #param Operation $operation
* #return $this
*/
public function addOperation(Operation $operation): EventBuilder
{
$this->operations[] = $operation;
// Fluent interface - return a reference to the instance
return $this;
}
public function compile(): Action
{
return new Action($this->target, $this->operations);
}
}
Let's try both techniques and prove they will produce the same result:
// Mock some operations
$myOperations = [
new Operation('Repair'),
new Operation('Clean'),
new Operation('Drive')
];
// Create a target
$target = new Target('Car');
/*
* Since the EventBuilder implements a fluent interface (returns an instance of itself from addOperation),
* we can chain the method calls together and just put a call to compile() at the end, which will return
* an Action instance
*/
$fluentAction = (new EventBuilder($target))
->addOperation($myOperations[0])
->addOperation($myOperations[1])
->addOperation($myOperations[2])
->compile();
// Run the action
$fluentResult = $fluentAction->run();
// Traditional approach, create an instance and call the addOperation method as needed
$builder = new EventBuilder($target);
// Pass our mocked operations
while (($currAction = array_shift($myOperations)))
{
/*
* We can ignore the result from addOperation here, just keep calling the method
* on the builder variable
*/
$builder->addOperation($currAction);
}
/*
* After we've added all of our operations, we can call compile on the builder instance to
* generate our Action.
*/
$traditionalAction = $builder->compile();
// Run the action
$traditionalResult = $traditionalAction->run();
// Verify that the results from both techniques are identical
assert($fluentResult == $traditionalResult, 'Results from both techniques should be identical');
// Enjoy the fruits of our labor
echo json_encode($traditionalResult, JSON_PRETTY_PRINT).PHP_EOL;
Output:
[
"Repair the Car",
"Clean the Car",
"Drive the Car"
]
Rob Ruchte thank you for detailed explanation, one thing I did not include was that each operation itself had ->build() call and I needed to move that to each $builder for it to work.
I then have a Base DTO class
class BaseDto
{
public function __construct(array $dtoValues)
{
$this->properties = array_map(static function (ReflectionProperty $q) {
return trim($q->name);
}, (new ReflectionClass(get_class($this)))->getProperties(ReflectionProperty::IS_PUBLIC));
foreach ($dtoValues as $propertyName => $value) {
$propertyName = Str::camel($propertyName);
if (in_array($propertyName, $this->properties, false)) {
$this->{$propertyName} = $value;
}
}
}
}
I also have an actual DTO class
class ModelDTO extends BaseDto
{
public int $id
public string $name;
}
I have the following Trait in PHP
trait ToDtoTrait
{
/**
* #param string $dtoClassName
* #return BaseDto
* #throws InvalidArgumentException
*/
public function toDto(string $dtoClassName): BaseDto;
{
$this->validateDtoClass($dtoClassName, BaseDto::class);
return new $dtoClassName($this->toArray());
}
/**
* #param string $dtoClassName
* #param string $baseClassName
* #return void
*/
private function validateDtoClass(string $dtoClassName, string $baseClassName)
{
if (!class_exists($dtoClassName)) {
throw new InvalidArgumentException("Trying to create a DTO for a class that doesn't exist: {$dtoClassName}");
}
if (!is_subclass_of($dtoClassName, $baseClassName)) {
throw new InvalidArgumentException("Can not convert current object to '{$dtoClassName}' as it is not a valid DTO class: " . self::class);
}
}
}
That trait is then used inside of my Model classes
class MyDbModel
{
use ToDtoTrait;
}
So this allows me to get an entry from the DB via the Model and then call toDto to receive an instance of the DTO class. Simple enough.
Now I have a service and this service will basically find the entry and return the DTO.
class MyService
{
public function find(int $id): ?ModelDTO
{
$model = MyModel::find($id);
if (empty($model)) {
return null;
}
return $model->toDto();
}
}
When I do it this way, I get a warning in the IDE:
Return value is expected to be '\App\Dtos\ModelDto|null', '\App\Dtos\BaseDto' returned
How do I declare this so that people can see that MyService::find() returns an instance of ModelDto so they will have autocomplete for the attributes of the DTO and any other base functions that come with it (not shown here).
The warning is raised because the return type of ToDtoTrait::toDto isBaseDto while the return type of MyService::find is ?ModelDTO, which are polymorphically incompatible (a BaseDto is not necessarily a ModelDTO).
An easy solution is to narrow down the DTO type using instanceof:
// MyService::find
$model = MyModel::find($id);
if (empty($model)) {
return null;
}
$dto = $model->toDto();
if (!$dto instanceof ModelDTO) {
return null;
}
return $dto;
Sidenote: Why is ToDtoTrait::toDto called without arguments in MyService (return $model->toDto();)? Looks like you want to pass ModelDTO::class to it: return $model->toDto(ModelDTO::class);.
I want to create GUI for failed_jobs and associate them with other tables record. So user could know which jobs property value failed during job handle and could be retried.
Laravel Job has a function failed(\Exception $exception) but it is called after the exception but before the record is saved in the failed_jobs table.
Also Laravel has Queue:failing(FailedJob $job) event but there I have only serialized job but not failed_jobs.
Did anyone run into similar problem? Is there any relation with processed job and failed one?
After much fussing, I accomplished this by embedding the related model within the Exception that is stored to the database. You could easily do similar but store only the id within the exception and use it to look up the model later. Up to you...
Wherever the exception that fails the job occurs:
try {
doSomethingThatFails();
} catch (\Exception $e) {
throw new MyException(OtherModel::find($id), 'Some string error message.');
}
app/Exceptions/MyException.php:
<?php
namespace App\Exceptions;
use App\Models\OtherModel;
use Exception;
class MyException extends Exception
{
/**
* #var OtherModel
*/
private $otherModel = null;
/**
* Construct
*/
public function __construct(OtherModel $otherModel, string $message)
{
$this->otherModel = $otherModel;
parent::__construct(json_encode((object)[
'message' => $message,
'other_model' => $otherModel,
]));
}
/**
* Get OtherModel
*/
public function getOtherModel(): ?object
{
return $this->otherModel;
}
}
That will store a string containing an object to the exception column on the failed_jobs table. Then you just need to decode that later on...
app/Models/FailedJob (or just app/FailedJob):
/**
* Return the human-readable message
*/
protected function getMessageAttribute(): string
{
if (!$payload = $this->getPayload()) {
return 'Unexpected error.';
}
$data = $this->decodeMyExceptionData();
return $data->message ?? '';
}
/**
* Return the related record
*/
protected function getRelatedRecordAttribute(): ?object
{
if (!$payload = $this->getPayload()) {
return null;
}
$data = $this->decodeMyExceptionData();
return $data->other_model ?? null;
}
/**
* Return the payload
*/
private function getPayload(): ?object
{
$payload = json_decode($this->payload);
return $payload ?? null;
}
/**
* Return the data encoded in a WashClubTransactionProcessingException
*/
private function decodeMyExceptionData(): ?object
{
$data = json_decode(preg_replace('/^App\\\Exceptions\\\WashClubTransactionProcessingException: ([^\n]*) in.*/s', '$1', $this->exception));
return $data ?? null;
}
anywhere:
$failedJob = FailedJob::find(1);
dd([
$failedJob->message,
$failedJob->related_record,
]);
Here I'm building Symfony SDK for REST API. Most of data are JSON objects with nested other JSON objects. Like here
{
"id": "eng_pl",
"name": "Premier League",
"_links": {
"self": {
"href": "/tournaments/eng_pl"
},
"seasons": {
"href": "/tournaments/eng_pl/seasons/"
}
},
"coverage": {
"id": "eng",
"name": "England",
"_links": {
"self": {
"href": "/territories/eng"
}
}
}
}
Deserialization must produce an object equal to object produced by the code listed below:
$tournament = new Tournament();
$tournament->setId('eng_pl');
$tournament->setName('Premier League');
$coverage = new Territory();
$coverage->setId('eng');
$coverage->setName('England');
$tournament->setCoverage($coverage);
I'm using my own custom Denormalizers, below the fragment of code of denormalizer for Tournament objects:
class TournamentDenormalizer implements DenormalizerInterface
{
/**
* #inheritdoc
*/
public function supportsDenormalization($object, $type, $format = null)
{
if ($type != Tournament::class) {
return false;
}
return true;
}
/**
* #inheritdoc
* #return Tournament
*/
public function denormalize($object, $class, $format = null, array $context = array())
{
$tournament = new Tournament();
$tournament->setId($object->id);
$tournament->setName($object->name);
if (isset($object->coverage)) {
/** #var Territory $coverage */
$coverage = ???; //HOWTO how to implement Territory denormalization here???
$tournament->setCoverage(
$coverage
);
}
return $tournament;
}
}
The question is how should I access TerritoryDenormalizer inside TournamentDenormalizer? I see two options:
First one (I'm using now) is to add implements DenormalizerAwareInterface to signature of denormalizer class and rely on Symfony\Component\Serializer\Serializer class:
$serializer = new Symfony\Component\Serializer\Serializer(
[
new TournamentDenormalizer(),
new TerritoryDenormalizer()
], [
new Symfony\Component\Serializer\Encoder\JsonDecode()
]
);
$serializer->deserialize($json, Tournament::class, 'json');
So in TournamentDeserializer it will be like here:
if (isset($object->coverage)) {
/** #var Territory $coverage */
$coverage = $this->denormalizer->denormalize(
$object->coverage,
Territory::class,
$format,
$context
);
$tournament->setCoverage(
$coverage
);
}
}
The second approach
Is to inject necessary denormalizers explicitly
$tournamentDenormalizer = new TournamentDenormalizer();
$tournamentDenormalizer->setTerritoryDenormalizer(new TerritoryDenormalizer());
So in TournamentDeserializer it will be like here:
if (isset($object->coverage)) {
/** #var Territory $coverage */
$coverage = $this->territoryDenormalizer->denormalize(
$object->coverage,
Territory::class,
$format,
$context
);
$tournament->setCoverage(
$coverage
);
}
}
Which of approaches is the best? What alternative approaches are possible?
Making your normalizer implement NormalizerAwareInterface (and eventually use NormalizerAwareTrait) is the way to go, this interface has been introduced for this specific use case.
I have created a base repository and am now extending it to add caching, but the problem I seem to be experiencing more than most is pagination
in my all() method, I do the following without caching:
public function all($pagination = 20)
{
try
{
$query = $this->model->newQuery();
return $this->handlePagination($query, $pagination);
}
catch (Exception $e)
{
throw $e;
}
}
protected function handlePagination($query, $pagination)
{
if (is_null($pagination))
{
return $query;
}
$collection = $query->paginate($pagination);
return $collection;
}
This is working well, but when I try to implement caching, I want to cache each model individually and store the keys for each collection, so, if I was paginating all of the entries, I would store each paginated collection in a cache:
cache set.1 [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
cache set.2 [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40]
etc...
the problem is that is seems impossible to use the actual pagination class to return the results since you are paginating only the id
I could return the data and the paginator separately but that seems very hacky.
Is there a way to repopulate the Paginator class with model data without over-writing the whole thing?
EDIT
I was thinking about something like this:
public function all($pagination = 20)
{
try
{
$cache_key = $this->cache_key . '.all';
$ids = Cache::get($cache_key);
if (! $ids)
{
$query = $this->model->newQuery();
$ids = $query->pluck('id');
Cache::put($cache_key, $ids, $this->cache_ttl);
}
$ids = $this->handlePagination($ids, $pagination);
$collection = new Collection();
foreach ($ids as $id)
{
if ($model = $this->find($id))
{
$collection->put($id, $model);
}
}
return $collection;
}
catch (Exception $e)
{
throw $e;
}
}
/**
* #var \Illuminate\Pagination\Paginator
*/
protected $paginator;
public function handlePagination($array, $pagination = 20)
{
if (!is_null($pagination))
{
$this->paginator = Paginator::make($array, count($array), $pagination);
return $this->paginator->getItems();
}
return $array;
}
public function getPaginator()
{
return $this->paginator;
}
I had to implement caching in a project where I work and I faced a similar issue but not with pagination. But the approach should be the same.
Laravel handles internally by default query caching if the model is told to do so.
What I did was creating a class that all the objects I want to cache should extend somehow. Then you can use pagination without even thinking about caching.
In the code below, pay special attention to the following method overrides:
newQuery
newBaseQueryBuilder
newFromBuilder
The CacheModel class looks like this:
<?php
class CacheModel extends Eloquent
{
/**
* Holds the query builder from which this model was fetched from.
*
* #var Illuminate\Database\Query\Builder
*/
protected $queryBuilder;
/**
* Overrides Illuminate\Database\Eloquent\Model's newQuery().
* We need to do this to store the query builder in our class for caching purpuses.
*
* #return \Illuminate\Database\Eloquent\Builder
*/
public function newQuery($excludeDeleted = true)
{
$eloquentBuilder = parent::newQuery($excludeDeleted);
return $eloquentBuilder->rememberForever();
}
/**
* Overrides Illuminate\Database\Eloquent\Model's newBaseQueryBuilder().
* We need to do this to store the query builder in our class for caching purpuses.
*
* #return \Illuminate\Database\Query\Builder
*/
protected function newBaseQueryBuilder()
{
$queryBuilder = parent::newBaseQueryBuilder();
$this->queryBuilder = $queryBuilder;
return $queryBuilder;
}
/**
* Overrides Illuminate\Database\Eloquent\Model's newFromBuilder().
* We need to do this to update the cache.
*
* #return an instance of the specified resource
*/
public function newFromBuilder($attributes = array())
{
$object = parent::newFromBuilder($attributes);
$that = $this;
$referencedCacheKeysFromObject = Cache::rememberForever($object->getCacheIdKey(), function() use ($that){
return array( $that->getQueryBuilder()->getCacheKey() => true );
});
if ( !isset($referencedCacheKeysFromObject[$this->getQueryBuilder()->getCacheKey()] ))
{
# Update the cache entries that hold the object
$referencedCacheKeysFromObject[$this->getQueryBuilder()->getCacheKey()] = true;
Cache::forget($object->getCacheIdKey());
Cache::forever($object->getCacheIdKey(), $referencedCacheKeysFromObject);
}
$referencedCacheKeysFromObjectTable = Cache::rememberForever($object->getCacheTableKey(), function() use ($that){
return array( $that->getQueryBuilder()->getCacheKey() => true );
});
if ( !isset( $referencedCacheKeysFromObjectTable[$this->getQueryBuilder()->getCacheKey()] ))
{
# Udate the cache entries that hold objects from the object's table.
$referencedCacheKeysFromObjectTable[$this->getQueryBuilder()->getCacheKey()] = true;
Cache::forget($object->getCacheTableKey());
Cache::forever($object->getCacheTableKey(), $referencedCacheKeysFromObjectTable);
}
return $object;
}
/**
* Overrides Illuminate\Database\Eloquent\Model's save().
* We need to do this to clean up the cache entries related to this object.
*
* #return \Illuminate\Database\Query\Builder
*/
public function save(array $attributes = array())
{
if (!$this->exists)
{
# If the object doesn't exists, it means that the object is gonna be created. So refresh all queries involving the object table.
# This is needed because the new created object might fell within one of the cache entries holding references to objects of this type.
$this->cleanUpCacheQueriesOfObjectTable();
}
$this->cleanUpCacheQueriesOfObject();
return parent::save();
}
/**
* Overrides Illuminate\Database\Eloquent\Model's delete().
* We need to do this to clean up the cache entries related to this object.
*
*/
public function delete()
{
$this->cleanUpCacheQueriesOfObject();
return parent::delete();
}
/**
* Overrides Illuminate\Database\Eloquent\Model's delete().
* We need to do this to clean up the cache entries related to this object.
*
*/
public static function destroy($id)
{
$this->find($id)->cleanUpCacheQueriesOfObject();
Cache::forget($this->getCacheIdKey($id));
return parent::destroy($id);
}
/**
* Returns the asociated query builder from which the model was created
*
* #return \Illuminate\Database\Query\Builder
*/
public function getQueryBuilder()
{
return $this->queryBuilder;
}
/**
* Cleans up all the cache queries that involve this object, if any.
*
*/
private function cleanUpCacheQueriesOfObject()
{
# Clean up the cache entries referencing the object as we need to re-fetch them.
if ( $referencedCacheKeys = Cache::get($this->getCacheIdKey()) )
{
foreach ($referencedCacheKeys as $cacheKey => $dummy)
{
Cache::forget($cacheKey);
}
}
}
/**
* Cleans up all the cache queries that involve this object table, if any.
* Needed when a a new object of this type is created.
* The cache needs to be refreshed just in case the object fells into
* one of the cache queries holding entries from the same type of the object.
*
*/
private function cleanUpCacheQueriesOfObjectTable()
{
# Clean up the cache entries referencing the object TABLE as we need to re-fetch them.
if ( $referencedCacheKeys = Cache::get($this->getCacheTableKey()) )
{
foreach ($referencedCacheKeys as $cacheKey => $dummy)
{
Cache::forget($cacheKey);
}
}
}
/**
* Returns a string containing a key for the table cache
*
*/
private function getCacheTableKey()
{
return '_' . $this->getTable();
}
/**
* Returns a string containing a key for the object cache
*
*/
private function getCacheIdKey($id = null)
{
if (!isset($id))
{
if (isset($this->id))
$id = $this->id;
else
$id = md5(serialize($this->getAttributes()));
}
return $this->getCacheTableKey() . '_' . $id;
}
}
Then you can do another class extending this CacheModel that could be called PaginateModel and you can do whatever pagination operations you'd like to do.
Of course the rest of the objects should extend this PaginateModel class.
EDIT: Added some if conditions in the method newFromBuilder as I just figuered out that there's a bug in APC since 3.1.3.
Pass the page number along as well for caching. Sonething like this
$currentPg = Input::get('page') ? Input::get('page') : '1';
$boards = Cache::remember('boards'.$currentPg, 60, function(){ return WhatEverModel::paginate(15); });
You need create the paginator manually as instructed in Laravel documentation. So basically you need to return paginator objects from your repository and those can be cached easily. Other option is to generate the paginator on the fly based on the cached/queried parameters.