Recently I am coding a small web page with CakePHP. The most of my models represent data that has to be collected through an external API with the HttpSocket component which provides CakePHP 2.x.
This is the main reason I include the following method in AppModel.php
public function get ($url, $options = array()) {
$options += $this->default;
$url .= 'api_key=' . $options['api_key'];
$hs = new HttpSocket();
return $hs->get($url);
}
I want to implement some measures to control the number of times CakePHP call the API, because it has limitations (100 request per minute and 1000 per hour, for example)
What is the best way to accomplish this?
I thought about store data related to the request in a new table but I am not sure if I can do queries from AppModel.php
Thank you
I would create a new table like api_call_log and increment the count. You can simply get a model instance in your AppModel:
$ApiCallLog = ClassRegistry::init('ApiCallLog')
$ApiCallLog->log();
Put your logic for the count into the model method and pass arguments as you need, not sure what else you might need.
Instead of putting it in AppModel a behavior might be a better place and you can implement additional methods there as well as your code grows and attach it only to models that need it.
An alternative that is very likely less resource hungry than using a SQL DB is to use a Redis cache. So use Cache::read() and Cache::write() to get the actual count and increment it. This is very likely the better alternative if you get A LOT requests to the API. You might get collisions using a SQL DB if you don't lock the tables during the counting and incrementing operation. I never had this case but I would not count on that it won't happen.
Additional note: get() is a very generic name, I would rename it to httpGet(). Names that are not specific can be pretty confusing, especially if there are methods of the same name in other classes around.
Related
Having a basic example of a REST endpoint which returns a User json object, how would we fill our Model having that we get the data from a Web Service instead of a DB?
My approach is to have a very very basic 'SDK' with independent Classes with my data representation and access data methods, and in every method I make a call to the API with Guzzle to actually perform the action.
The problem is that I'm losing all the power that Eloquent provides.
Is there any recommended approach to do this in Laravel? Non Laravel approachs will do too.
I'd recommend to use a combination of cache and collections.
First you'll collect the web service results to a collection that you will then cache.
Something like this :
$users = Cache::remember('users', $minutes, function() use ($guzzle) {
$apiUsers = $guzzle->get('api/users'); // Get the API response as json
return collect(json_decode($apiUsers));
});
Using collections you'll have access to almost all of the Eloquent features (because it returns Collection as well. So you might do things like $user = $users->where('email', 'you#email.com')->first();
Then if you want to make it more like an "SDK", you can create a package that has classes and methods that makes it easier to get and use your API.
The problem is that I'm losing all the power that Eloquent provides.
That's not the only problem here. Your approach will be quite ineffective if you are planing to call remote API on each page display and the overall performance will greatly suffer from such approach. What you can do instead is to have your API data cached in local DB, so your models can use Eloquent to get them from with all the benefits.
I am starting developing in a new architectural paradigm over some legacy code we have. Let's call the different paradigms v1 and v2.
On v1 we had models for different domains (e.g games) that nested all the information of the entity and also of the attributes - many times mixing what a entity was and what an attribute was. So you could get a response that nested everything related to the domain when you hit a endpoint. E.g/ v1/games would bring you the game + the players + the teams + the arena.
On v2 we're trying to be more resource oriented so you actually get the resource you're querying for (e.g) - You have a endpoint for v2/games, and then you'll have an endpoint v2/games/{id}/players. With the former you'll get games objects and in the second you get players objects. I think it's self documentary by the path - You know what you're going to get by looking at the request you're making.
On v1 we had this huge models where everything was included - whatever we saw as an "attribute" of the domain. So our actual model would include the object to the related "attributes" - E.g/ Games objects would contain a players objects array.
Whenever you were getting or posting you would actually user the same model, you would just get more information (nested objects) on the GET and some fields weren't required for the POST (e.g/ you wouldn't need to send the players with your post - we had a separate endpoint for that).
class Game {
public $id;
public $datePlayed;
public $players = [];
public $homeTeam = [];
public $awayTeam = [];
}
On v2 models, we do an actual representation of the database table (like the active record approach of modeling were the model is based on the persistence layer). So the game model would not include players (since that is a different object altogether).
class Game {
public $id;
public $datePlayed;
public $homeTeamId;
public $awayTeamId;
}
So far I think the end result is very good because we're actually getting the resource we need and not a nested version with a lot of verbosity and unnecessary data.
However I have a new challenge now. There are endpoints that need to return responses with more information which will not be the actual database representation. Since we're not on a perfect world and I don't want developers that use our API to actually make tons of request for all the attributes I need to come up with a solution that helps me deliver them some basic information of the related resources.
So for example returning to our Game endpoint -> for creating I do want them to send me a payload that is the exact database object
{
"datePlayed": "2016-04-29T17:20:08+00:00",
"homeTeamId": 5,
"awayTeamId": 15
}
But for the GET I would like to decorate the response with basic data (not the whole object but just enough so that the developer get the basic information so that they only make a new request to get the complete related resource but not the basic information).
{
"datePlayed": "2016-04-29T17:20:08+00:00",
"homeTeam": {
"id": 5,
"name": "Killer clan"
},
"awayTeam": {
"id": 15,
"name": "Retry team"
}
}
At first I delegated this responsibility to the controller->repository who would create a new object upon request and return it on the GET method but would validate the POST and PUT requests to the actual model. However this doesn't seem too maintainable on the long run since you have to dig into the code to understand what you're sending.
So I gave it a long thought and I have an idea to create a new set of models with the convention Stub (the part of a check, receipt, ticket, or other document torn off and kept as a record) that will declare the structure of the response. Eg:/ I would create a GameStub model class like this
class GameStub {
public $id;
public $datePlayed;
public $homeTeam = [];
public $awayTeam = [];
}
I think placeholder or metadata also are good name conventions, but the gist is the same.
What do you guys think of this solution? Does it make sense?
Makes definitively sense because it´s a good approach to decouple your application from the api.
Resources in API != application models.
I use the following structure in my applications:
Model
- Game (This is your model)
Param
- GameParam (Model / Param / Resource which you want to provide over the api)
Converter
- GameConverter (Converts the Model into a Param and Param into a Model)
Part of it makes a great deal of sense but I did not understand your stub metaphor and how it relates to the problem.
So GET /games/gameId would normally just return team ids. To get more team information (such as team name) the user would then use GET /teams/teamId. You want to avoid the additional queries by sending some (but not all) team information as part of the game query.
So how would you know what additional information a particular request needs? What if (for example) some request needs the coach's name? Seems like you would be trying to guess what was needed and eventually end up sending everything again.
Have you looked at Facebook's graph query language? http://graphql.org/
The basic idea if to allow the request to specify what information they really need. So you might have GET/games/gameId?want=teamName,coachName
Then it is up to the server to figure how to return the necessary data. I'm not suggesting using graphql in your application. It's almost certainly overkill.
But maybe giving the consumer some ability to specify what they need might help.
And by all means, take advantage of a pattern called Command Query Responsibility Segregation (http://martinfowler.com/bliki/CQRS.html). Anything to do with creating or updating objects should be in it's own space. Treat querying independently.
I have a project that is a results database for several sport event series.
As you an imagine most the content remains more or less the same.
I would like to cache some of the content to save database queries.
The project is build using PHP and is using a custom MVC.
Where would you add the caching logic?
Use Memcached. Something like this, where you cache the results of the query, and try retrieving from cache before DB.
Edit: Added json_encoding the data... json_encode is slightly faster than PHP's default serialization for arrays, and can be used by other applications accessing the data
Memcached by default uses FastLZ compression.
// The ID of the item you're looking for (from $_GET or whatever)
$id = 1;
// Create a memcached instance
$m = new Memcached();
$m->addServer('localhost', 11211);
// Try retrieving the event from memcached
$data = $m->get('event_' . $id);
if ($data) {
// If you found data in memcached, decode it for use in code
$data = json_decode($data);
} else {
// If you didn't find the data in memcached, try retrieving from the DB
$result = mysqli_query('SELECT * FROM events WHERE id = ' . $id);
// If you found results in the DB...
if ($data = mysqli_fetch_assoc($result)) {
// Save them to memcached for future calls
// Note: here, I'm using the optional param to expire it after 1 day
$m->set('event_' . $id, json_encode($data), 86400);
}
}
// Now you can use your data
var_dump($data);
Edited to add comments in code
There are two points where you can use caching in the MVC and MVC-inspired design patterns: the output and data retrieval:
Cache the output:
This would have to be implemented in the views, as part of UI logic (views should be instances, not just dumb templates). Somewhere is same area where view is combining the templates in the response. Only in this case, some of the "templates" would actually be completely without variables.
This way you can cache fragments of HTML in context of MVC .. but this would no apply to your particular use-case. It's better suited for situations, when interaction with data source take too much time. Like when storage is actually REST API.
Cache the data:
To enable this functionality it would have to be performed within model layer. To be precise: in services (classes/instances, that mostly contain interaction between domain and storage logic).
If you have a fully implemented model layer, where you separate application, domain and storage logic, then implementation of caching would simply look as different form of storage.
$user = new User;
$cache = new UserCacheMapper;
$user->setId( 42 );
if ( ! $cache->fetch( $user ) )
{
$storage = new UserSQLMapper( $pdo );
$storage->fetch( $user );
$cache->store( $user );
}
// the $user object has been initialized
The initialization of objects actually should be performed by factories within the service, but this is a simplified example
This way you can create application without caching, and only later add it by modifying the services(that are responsible for application logic). Neither the domain objects (domain logic) nor the data mappers (storage logic) would have to change.
Objects can be cached using Memcache(d), which is most useful for multi server setups. For a single server, you can store things in an APC/Xcache etc which offer caching facilities.
I would cache the data in the dataaccess method.
I would used memcached as its really easy to build a cacheing solution using it http://memcached.org/
I normally put the caching logic in the controller although the model may normally be a more correct place for it, reason being that the context of model use can affect caching strategies
Putting it in the model makes the caching strategy sit with the rest of the data access functionality which is correct, it means that you can swap models and the caching code moves with it. Where this can fall down is where you have complex invalidation that cannot be put in the model.
“There are only two hard things in Computer Science: cache invalidation and naming things” Phil Karlton
You say you're using your own MVC pattern framework so it's hard to give a specific recommendation.
There are a few places you can cache.
If you use HTTP properly (idempotency, etags, cache-control headers, etc), you might benefit from placing the caching layer outside the application and using a forward cache like varnish to cache whole pages. However, it's very easy to get HTTP wrong, so this may not be the way to go. For example, if you have user accounts and the same url produces different server output depending on what user is logged in, you can't use HTTP caching because your resource depends on cookie state. (What you should do is put the user into the url, E.g. /mysite/{userid}/profile instead of /mysite/profile.) You may still be able to use varnish via its more advanced features, but it will be more difficult.
You say you are just interested in caching database query results, so I'll address that.
If you have a few well-defined queries where you want to cache a database response, you can put the logic in the controller: before you pass a model on to the view, cache it explicitly. This solution produces a messy codebase if you have multiple places to cache. Also, if you are retrieving the same data in different controllers, you will need to carefully and manually ensure that all controller code that retrieves the same models also consults the cache and uses the same cache key. Obviously, it also becomes difficult to turn caching on and off.
A cleaner solution is to wrap your models in a way that caches results transparently. This means you can add caching declaratively without making substantial changes to your controllers or models. However, this requires having a very well designed model api, which you may not have.
I see two general ways of handling instances of classes in PHP.
Take, for example, that we have users in the system.
Method 1
$user_id = 5;
$u = new User($user_id); // fills in any and all info from the database
echo $u->getName();
echo $u->getEmail();
Method 2
$user_id = 5;
echo Users::getName($user_id); // makes a call to the database
echo Users::getEmail($user_id); // makes a call to the database
Implied in this approach is that I have a class Users that has the business logic related to users. It has direct access to a User class that can do CRUD operations by calling MySQL.
Method 3
$user = UserDao::findById($id);
$user->getName();
I like this approach because you should separate the model from the factory that creates the model.
Executing a query to get each property is expensive if you are going to get every property anyway. So your method 1 or #Amir's neater approach would be a better choice in this case.
It's a trade off, get "everything" with one db access in case you need it versus storing lots of stuff in memory that you might not need.
Your two approaches are get everything at once
or get each as you need it.
There are in between options basically
get it if you haven't got it yet,
get "common" stuff always and other bits as and when you need them.
or a mixture always get this, and then the other bits if you haven't got them yet.
There's no hard and fast rule. you have to look at what operations are carried out on the object, and how concurrent they are. ie can session 2 change person1's attributes while session1 is looking at it. Will this result in error, confusion, or not matter.
I'm trying to replace a site written procedurally with a nice set of classes as a learning exercise.
So far, I've created a record class that basically holds one line in the database's main table.
I also created a loader class which can:
loadAllFromUser($username)
loadAllFromDate($date)
loadAllFromGame($game)
These methods grab all the valid rows from the database, pack each row into a record, and stick all the records into an array.
But what if I want to just work with one record? I took a stab at that and ended up with code that was nearly identical to my procedural original.
I also wasn't sure where that one record would go. Does my loader class have a protected record property?
I'm somewhat confused.
EDIT - also, where would I put something like the HTML template for outputting a record to the site? does that go in the record class, in the loader, or in a 3rd class?
I recommend looking into using something like Doctrine for abstracting your db-to-object stuff, other than for learning purposes.
That said, there are many ways to model this type of thing, but in general it seems like the libraries (home-grown or not) that handle it tend to move towards having, at a high level:
A class that represents an object that is mapped to the db
A class that represents the way in which that object is mapped to the db
A class that represents methods for retrieving objects from the db
Think about the different tasks that need done, and try to encapsulate them cleanly. The Law of Demeter is useful to keep in mind, but don't get too bogged down with trying to grok everything in object-oriented design theory right this moment -- it can be much more useful to think, design, code, and see where weaknesses in your designs lie yourself.
For your "work with one record, but without duplicating a bunch of code" problem, perhaps something like having your loadAllFromUser methods actually be methods that call a private method that takes (for instance) a parameter that is the number of records to be retrieved, where if that parameter is null it retrieves all the records.
You can take that a step further, and implement __call on your loader class. Assuming it can know or find out about the fields that you want to load by, you can construct the parameters to a function that does the loading programatically -- look at the common parts of your functions, see what differs, and see if you can find a way to make those different parts into function parameters, or something else that allows you to avoid repetition.
MVC is worth reading up on wrt your second question. At the least, I would probably want to have that in a separate class that expects to be passed a record to render. The record probably shouldn't care about how it's represented in html, the thing that makes markup for a record shouldn't care about how the record is gotten. In general, you probably want to try to make things as standalone as possible.
It's not an easy thing to get used to, and most of "getting good" at this sort of design is a matter of practice. For actual functionality, tests can help a lot -- say you're writing your loader class, and you know that if you call loadAllFromUser($me) that you should get an array of three specific records with your dataset (even if it's a dataset used for testing only), if you have something you can run which would call that on your loader and check for the right results, it can help you know that your code is at least right from the standpoint of behavior, if not from design -- and when you change the design you can ensure that it still behaves correctly. PHPUnit seems to be the most popular tool for this in php-land.
Hopefully this points you in a useful group of directions instead of just being confusing :) Good luck, and godspeed.
You can encapsulate the unique parts of loadAllFrom... and loadOneFrom... within utility methods:
private function loadAll($tableName) {
// fetch all records from tableName
}
private function loadOne($tableName) {
// fetch one record from tableName
}
and then you won't see so much duplication:
public function loadAllFromUser() {
return $this->loadAll("user");
}
public function loadOneFromUser() {
return $this->loadOne("user");
}
If you like, you can break it down further like so:
private function load($tableName, $all = true) {
// return all or one record from tableName
// default is all
}
you can then replace all of those methods with calls such as:
$allUsers = $loader->load("users");
$date = $loader->load("date", false);
You could check the arguments coming into your method and decide from there.
$args = func_get_args();
if(count($args) > 1)
{
//do something
}
else // do something else
Something simple liek this could work. Or you could make two seperate methods inside your class for handling each type of request much like #karim's example. Whichever works best for what you would like to do.
Hopefully I understand what you are asking though.
To answer your edit:
Typically you will want to create a view class. This will be responsible for handling the HTML output of the data. It is good practice to keep these separate. The best way to do this is by injecting your 'data class' object directly into the view class like such:
class HTMLview
{
private $data;
public function __construct(Loader $_data)
{
$this->data = $_data;
}
}
And then continue with the output now that this class holds your processed database information.
It's entirely possible and plausible that your record class can have a utility method attached to itself that knows how to load a single record, given that you provide it a piece of identifying information (such as its ID, for example).
The pattern I have been using is that an object can know how to load itself, and also provides static methods to perform "loadAll" actions, returning an array of those objects to the calling code.
So, I'm going through a lot of this myself with a small open source web app I develop as well, I wrote most of it in a crunch procedurally because it's how I knew to make a working (heh, yeah) application in the shortest amount of time - and now I'm going back through and implementing heavy OOP and MVC architecture.