I want to be able to call the CakeS3 plugin from the Cake Shell. However, as I understand it components cannot be loaded from the shell. I have read this post outlining strategies for overcoming it: using components in Cakephp 2+ Shell - however, I have had no success. The CakeS3 code here is similar to perfectly functioning cake S3 code in the rest of my app.
<?php
App::uses('Folder','Utility');
App::uses('File','Utility');
App::uses('CakeS3.CakeS3','Controller/Component');
class S3Shell extends AppShell {
public $uses = array('Upload', 'User', 'Comment');
public function main() {
$this->CakeS3 = new CakeS3.CakeS3(
array(
's3Key' => 'key',
's3Secret' => 'key',
'bucket' => 'bucket')
);
$this->out('Hello world.');
$this->CakeS3->permission('private');
$response = $this->CakeS3->putObject(WWW_ROOT . '/file.type' , 'file.type', $this->CakeS3->permission('private'));
if ($response == false){
echo "it failed";
} else {
echo "it worked";
}
}
This returns an error of "Fatal error: Class 'CakeS3' not found in /home/app/Console/Command/S3Shell.php. The main reason I am trying to get this to work is so I can automate some uploads with a cron. Of course, if there is a better way, I am all ears.
Forgive me this "advertising"... ;) but my plugin is probably better written and has a better architecture than this CakeS3 plugin if it is using a component which should be a model or behaviour task. Also it was made for exactly the use case you have. Plus it supports a few more storage systems than only S3.
You could do that for example in your shell:
StorageManager::adapter('S3')->write($key, StorageManager::adapter('Local')->read($key));
A file should be handled as an entity on its own that is associated to whatever it needs to be associated to. Every uploaded file (if you use or extend the models that come with the plugin, if not you have to take care of that) is stored as a single database entry that contains the name of the config that was used and some meta data for that file. If you do the line of code above in your shell you will have to keep record in the table if you want to access it this way later. Just check the examples in the readme.md out. You don't have to use the database table as a reference to your files but I really recommend the system the plugin implements.
Also, you might not be aware that WWW_ROOT is public accessible, so in the case you store sensitive data there it can be accessed publicly.
And finally in a shell you should not use echo but $this->out() for proper shell output.
I think the App:uses should look like:
App::uses('CakeS3', 'CakeS3.Controller/Component');
I'm the author of CakeS3, and no I'm afraid there is no "supported" way to do this as when we built this plugin, we didn't need to run uploads from shell and just needed a simple interface to S3 from our controllers. We then open sourced the plugin as a simple S3 connector.
If you'd like to have a go at modifying it to support shell access, I'd welcome a PR.
I don't have a particular road map for the plugin, so I've tagged your issue on github as an enhancement and will certainly consider it in future development, but I can't guarantee that it would fit your time requirements so that's why I mention you doing a PR.
Related
I've managed to get a stable load balanced front end servers that can scale horizontally quite well however the next bottle neck would be the db. There was a blog post discussing scaling dbs horizontally however very little detail on it. I'm currently using PostgreSQL and so the only plugin I've found wouldn't work.
Are my only options creating my own HAProxy or rewriting the PostgreSQL plugin to allow connections with read replicas?
I'm using AWS for all my hosting
Firstly - I'd love to be corrected on this!
Having only had a quick look through some of the ORM classes in a SilverStripe 3.5 site, it looks like while the ORM does support multiple database connections (see DB::get_conn with argument for name) it is designed for specific use cases in mind. That is to say, you may have a module that needs to write to a specific database, so this would allow it to.
What you want is native and automatic support for this within the framework, so that all reads go to your slave(s) and writes go to your master. Unfortunately, it doesn't look like this comes out of the box. You might be able to achieve it by overloading a couple of the core SQL classes using the injector.
If you were to try it, this answer outlines how you could separate select statements out from the rest and run them through a different database connector.
As a quick example of how you might go at achieving this with SQLSelect, you will notice that it is injectable, which means you can easily overload it.
File: mysite/_config/injector.yml
Injector:
SQLSelect:
class: ReadOnlySQLSelect
You need to register a new database connection with the DB class:
File: mysite/_config.php
$readDatabaseConfig = array(/** define your DB credentials here, as with the default $databaseConfig **/);
if (!DB::connect($readDatabaseConfig, 'default_read')) {
user_error('Failed to connect to read replica DB!', E_USER_ERROR);
}
Now, overload the SQLSelect class and replace the parts of it that call the DB class methods. This class inherits from SQLExpression which is the class the contains the methods you actually care about in this instance:
File: mysite/code/ReadOnlySQLSelect.php
class ReadOnlySQLSelect extends SQLSelect
{
public function sql(&$parameters = array())
{
// Changed from SQLExpression: third parameter passed as connection name
$sql = DB::build_sql($this, $parameters, 'default_read');
if (empty($sql)) {
return null;
}
if ($this->replacementsOld) {
$sql = str_replace($this->replacementsOld, $this->replacementsNew, $sql);
}
return $sql;
}
public function execute()
{
$sql = $this->sql($parameters);
// Changed from SQLExpression: skip DB::prepared_query since it doesn't allow
// you to provide the connection name - replace it with its contents instead.
$conn = DB::get_conn('default_read');
return $conn->preparedQuery($sql, $parameters);
}
}
Note: SQLSelect::unlimitedRowCount should technically be replaced where it calls DB::prepared_query, since the prepared query method calls DB::get_conn with no arguments, so will always return the default connection. You could replace the DB::prepared_query line the same as used above:
$conn = DB::get_conn('default_read');
$result = $conn->preparedQuery($sql, $innerParameters);
If you implement the above method, also change new SQLSelect() to SQLSelect::create(), otherwise you'll end up with some queries that still hit the master server because it'll bypass your class by not using the injector.
There's also an instance in SQLConditionalExpression that you should replace too (::toSelect) but that is likely to affect query transformations from other child implementations of that class, and you won't be able to do much about it without either (A) PRing a fix to the framework or (B) overloading all the other SQL* classes.
At this point you should have everything you need to route select queries to your default_read connection.
Infrastructure
On the infrastructure side, you should be able to set up read replicas through the RDS console. When you do so it will provide you with a DNS endpoint for your replica node(s), which you can use in your _config.php to configure the connection to the read replica database.
If this works for you, you should create a module for it and put it up on GitHub - this would definitely be useful for others in future!
You may also consider making pull requests to the framework to add additional arguments to methods like DB::prepared_query to accept a connection name.
Also worth noting is that if you're using the mysqlnd database adapter you may be able to take advantage of read/write splitting, implemented with some sort of injector overloading but all handled at a lower level than the application layer.
I have a PHP daemon script running on the command line that can be connected to via telnet etc and be fed commands.
What it does with the command is based on what modules are loaded, which is currently done at the start. (psuedocode below for brevity)
$modules = LoadModules();
StartConnection();
while(true){
ListenForCommands();
}
function LoadModules(){
$modules = Array();
$dir = scandir("modules");
foreach($dir as $folder){
include("modules/".$folder."/".$folder.".php");
$modules[$folder] = new $folder;
}
}
function ListenForCommands(){
if(($command = GetData())!==false){
if(isset($modules[$command])){
$modules[$command]->run();
}
}
}
So, an example module called "bustimes" would be a class called bustimes, living in /modules/bustimes/bustimes.php
This works fine. However, I'd like to make it so modules can be updated on the fly, so as part of ListenForCommands it looks at the filemtime of the module, works out if it's changed, and if so, effectively reloads the class.
This is where the problem comes in, obviously if I include the class file again, it'll error as the class already exists.
All of the ideas I have of how to get around this problem so far are pretty sick and I'd like to avoid doing.
I have a few potential solutions so far, but I'm happy with none of them.
when a module updates, make it in a new namespace and point the reference there
I don't like this option, nor am I sure it can be done (as if I'm right, namespaces have to be defined at the top of the file? That's definitely workaroundable with a file_get_contents(), but I'd prefer to avoid it)
Parsing the PHP file then using runkit-method-redefine to redefine all of the methods.
Anything that involves that kind of parsing is a bad plan.
Instead of including the file, make a copy of the file with everything the same but str_replacing the class name to something with a rand() on the end or similar to make it unique.
Does anyone have any better ideas about how to either a) get around this problem or b) restructure the module system so this problem doesn't occur?
Any advice/ideas/constructive criticism would be extremely welcome!
You should probably load the files on demand in a forked process.
You receive a request
=> fork the main process, include the module and run it.
This will also allow you to run several commands at once, instead of having to wait for each one to run before launching the next.
Fork in php :
http://php.net/manual/en/function.pcntl-fork.php
Tricks with namespaces will fail if module uses external classes (with relative paths in namespace).
Trick with parsing is very dangerous - what if module should keep state? What if not only methods changed, but, for example, name of implemented interface? How it will affect other objects if they have link to instance of reloaded class?
I think #Kethryweryn is something you can try.
Here's the situation:
I have a catch-all on my domain email (so *#domain.com) redirecting to a piping script located at /home/domain/scripts/piper.php. This piper script is not within the Kohana ORM, but all of my other files are. I want to try to use Kohana inside this piper.php file.
I have tried (unsuccessfully) all of the following:
Including Kohana
I couldn't figure out what needed to be included, and more importantly how to override the url variable that Kohana uses to determine the right controller. Also, this is a catch-all piper, so it isn't using HTTP (to my knowledge), so much as executing a command.
Piping
I tried piping to the following:
/home/domain/public_html/index.php --uri="piper"
But cPanel makes this impossible, as you can only specify the destination script, and not the proper flags and such (unless I am missing something).
PHP exec()
I tried using the following line:
exec("php /home/domain/public_html/index.php --uri=\"/piper\"")
I was hoping that the stdin data would be maintained across the exec() command, but I could never get it to recognize the uri command, though I can run this on my localhost and it works just fine.
I was using http://www.coderelic.com/2011/10/creating-cron-jobs-in-kohana-3-x-is-a-piece-of-cake/ as a reference, but can't get anything to work.
I'm happy with either one of these solutions such that I can see an incoming email, parse it, then send emails based on the parameters.
Let me know if you need more information! I'm le stumped.
/home/domain/public_html/index.php --uri="piper" would be a valid way to do it. If your host sucks and doesn't let you specify that, put it into a bash script instead and reference that.
If you are on any recent version of kohana (3.2 or 3.3), a better way to do this would be to use Minion to run the command line task. This is what Minion was designed for.
All you need to do is to:
modify your piper.php script to be a valid PHP class;
place it in /application/classes/ folder;
Kohana will automatically load your class file (like include) during initialization.
Then you can use your piper class as usual class by $piper = new Piper; ....
UPD
You have to serve your emails trough Kohana.
Create controller, for example pipe (route it with /pipe URL):
public function action_pipe() {
$pipe = new Pipe; // This creates new Pipe object (your emails serving class)
$pipe->serve(); // Sserve emails within `serve()` method of Pipe class
}
Although admittedly, I'm not sure if these other answers are correct because I can't figure out how to reproduce the results.
What ended up working for my situation was to create a Controller_Piper class that is called in the /home/domain/scripts/piper.php. What I did was to copy the code from /home/domain/public_html/index.php and changed the following:
echo Request::factory("/piper")
->execute()
->send_headers(TRUE)
->body();
This loads the piper controller and executes everything very nicely. Not sure if it's the cleanest, but it does work.
I need to be able to "effectivly" redeclare my class, so that during runtime, whilst my PHP IRC bot is running I can reload modules as the code base changes, which requires getting the class, but PHP won't allow it to be redeclared, nor is there a way to undeclare a class.
Is there a way I can acheive this? Perhaps some other method? I've tried "runkit", but that failed.
My current code is here:
http://pastie.org/private/ptj7c0t0teh3nnzn7ehcg
So to clarify, I effecivly need to be able to reload my class, instatiate it and put into a property in my main bot class once code has changed in said module, whilst the bot is running (run-time).
A brief look at your application leads me to believe your best course of action is to use a base class as suggested in the comment above.
Something to the extent of:
class IRCModule {
private static $module_dir = ''; //put your module directory here w/o trailing slash
public static function getModule( $module ) {
//if the module directory doesn't exist, don't do anything
if( !is_dir( self::$module_dir.DIRECTORY_SEPARATOR.$module ) ) return false;
//load the module file
$fname = scandir(self::$module_dir.DIRECTORY_SEPARATOR.$module);
$fname = $fname[2]; //first 2 elements will be . and ..
require_once( self::$module_dir.DIRECTORY_SEPARATOR.$module.DIRECTORY_SEPARATOR.$fname );
$className = str_replace('.class.php',NULL,$fname);
return new $className();
}
}
You would then extend that using your modules. That would allow you to overwrite a module by simply removing it's old file /my/module/dir/moduleName/moduleNameV1.0.class.php and replacing it with a new version /my/module/dir/moduleName/moduleNameV1.1.class.php
As mentioned in the comments, this will eventually fill the memory on the server, so you should schedule a reboot of the service each time you make substantial changes, but it also allows you to load new versions on demand without stopping the service.
A more stable approach would be to take advantage of process control and spin off daemons for each connection from your parent script, or implement a cache system that stores all data on the disk/database so that you can detect a change in module version and instantly reboot the server. But the solution above should work for you for the time being :)
I am trying to write a cronjob controller, so I can call one website and have all modules cronjob.php executed. Now my problem is how do I do that?
Would curl be an option, so I also can count the errors and successes?
[Update]
I guess I have not explained it enough.
What I want to do is have one file which I can call like from http://server/cronjob and then make it execute every /application/modules/*/controller/CronjobController.php or have another way of doing it so all the cronjobs aren't at one place but at the same place the module is located. This would offer me the advantage, that if a module does not exist it does not try to run its cronjob.
Now my question is how would you execute all the modules CronjobController or would you do it a completly different way so it still stays modular?
And I want to be able to giveout how many cronjobs ran successfully and how many didn't
After some research and a lot procrastination I came to the simple conclusion that a ZF-ized cron script should contain all the functionality of you zend framework app - without all the view stuff. I accomplished this by creating a new cronjobfoo.php file in my application directory. Then I took the bare minimum from:
-my front controller (index.php)
-my bootstrap.php
I took out all the view stuff and focused on keeping the environment setup, db setup, autoloader, & registry setup. I had to take a little time to correct the document root variable and remove some of the OO functionality copied from my bootstrap.
After that I just coded away.. in my case it was compiling and emailing out nightly reports. It was great to use Zend_Mail. When I was confident that my script was working the way I wanted, I just added it my crontab.
good luck!
For Zend Framework I am currently using the code outlined bellow. The script only includes the portal file index.php, where all the paths, environment and other Zendy code is bootstrapped. By defining a constant in the cron script we cancel the final step , where the application is run.
This means the application is only setup, not even bootstrapped. At this point we start bootstraping the resources we need and that is that
//public/index.php
if(!defined('DONT_RUN_APP') || DONT_RUN_APP == false) {
$application->bootstrap()->run();
}
// application/../cron/cronjob.php
define("DONT_RUN_APP",true);
require(realpath('/srv/www/project/public/index.php'));
$application->bootstrap('config');
$application->bootstrap('db');
//cron code follows
I would caution putting your cronjobs accessible to the public because they could be triggered outside their normal times and, depending on what they do, cause problems (I know that is not what you intend, but by putting them into an actual controller it becomes reachable from the browser). For example, I have one cron that sends e-mails. I would be spammed constantly if someone found the cron URL and just began hitting it.
What I did was make a cron folder and in there created a heartbeat.php which bootstraps Zend Framework (minus MVC) for me. It checks a database which has a list of all the installed cron jobs and, if it is time for them to run, generates an instances of the cron job's class and runs it.
The cron jobs are just child classes from an abstract cron class that has methods like install(), run(), deactivate(), etc.
To fire off my job I just have a simple crontab entry that runs every 5 minutes that hits heartbeat.php. So far it's worked wonderful on two different sites.
Someone mentioned this blog entry a couple days ago on fw-general (a mailinglist which I recommend reading when you use the Zend Framework).
There is also a proposal for Zend_Controller_Request_Cli, which should address this sooner or later.
I have access to a dedicated server and I initially had a different bootstrap for the cron jobs. I eventually hated the idea, just wishing I could do this within the existing MVC setup and not have to bother about moving things around.
I created a file cron.sh, saved is within my site root (not public) and in it I put a series of commands I would like to run. As I wanted to run many commands at once I wrote the PHP within my controllers as usual and added curl calls to those urls within cron.sh. for example curl http://www.mysite.com/cron_controller/action Then on the cron interface I ran bash /path/to/cron.sh.
As pointed out by others your crons can be fired by anyone who guesses the url so there's always that caveat. You can find a solution to that in many different ways.
Take a look at zf-cli:
scripts at master from padraic/ZFPlanet - GitHub
This handles well all cron jobs.
Why not just create a crontab.php, including, or requiring the index.php bootstrap file?
Considering that the bootstrap is executing Zend_Loader::registerAutoload(), you can start working directly with the modules, for instance, myModules_MyClass::doSomething();
That way you are skipping the controllers. The Controller job is to control the access via http. In this case, you don't need the controller approach because you are accessing locally.
Do you have filesystem access to the modules' directories? You could iterate over the directories and determine where a CronjobController.php is available. Then you could either use Zend_Http_Client to access the controller via HTTP or use an approach like Zend_Test_PHPUnit: simulate the actual dispatch process locally.
You could set up a database table to hold references to the cronjob scripts (in your modules), then use a exec command with a return value on pass/fail.
I extended gregor answer with this post. This is what came out:
//public/index.php
// Run application, only if not started from command line (cli)
if (php_sapi_name() != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
$application->run();
}
Thanks gregor!
My solution:
curl /cron
Global cron method will include_once all controllers
Check whether each of the controllors has ->cron method
If they have, run those.
Public cron url (for curl) is not a problem, there are many ways to avoid abuse. As said, checking remote IP is the easiest.
This is my way to run Cron Jobs with Zend Framework
In Bootstrap I will keep environment setup as it is minus MVC:
public static function setupEnvironment()
{
...
self::setupFrontController();
self::setupDatabase();
self::setupRoutes();
...
if (PHP_SAPI !== 'cli') {
self::setupView();
self::setupDbCaches();
}
...
}
Also in Bootstrap, I will modify setupRoutes and add a custom route:
public function setupRoutes()
{
...
if (PHP_SAPI == 'cli') {
self::$frontController->setRouter(new App_Router_Cli());
self::$frontController->setRequest(new Zend_Controller_Request_Http());
}
}
App_Router_Cli is a new router type which determines the controller, action, and optional parameters based on this type of request: script.php controller=mail action=send. I found this new router here: Setting up Cron with Zend Framework 1.11
:
class App_Router_Cli extends Zend_Controller_Router_Abstract
{
public function route (Zend_Controller_Request_Abstract $dispatcher)
{
$getopt = new Zend_Console_Getopt (array());
$arguments = $getopt->getRemainingArgs();
$controller = "";
$action = "";
$params = array();
if ($arguments) {
foreach($arguments as $index => $command) {
$details = explode("=", $command);
if($details[0] == "controller") {
$controller = $details[1];
} else if($details[0] == "action") {
$action = $details[1];
} else {
$params[$details[0]] = $details[1];
}
}
if($action == "" || $controller == "") {
die("Missing Controller and Action Arguments == You should have:
php script.php controller=[controllername] action=[action]");
}
$dispatcher->setControllerName($controller);
$dispatcher->setActionName($action);
$dispatcher->setParams($params);
return $dispatcher;
}
echo "Invalid command.\n", exit;
echo "No command given.\n", exit;
}
public function assemble ($userParams, $name = null, $reset = false, $encode = true)
{
throw new Exception("Assemble isnt implemented ", print_r($userParams, true));
}
}
In CronController I do a simple check:
public function sendEmailCliAction()
{
if (PHP_SAPI != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
echo "Program cannot be run manually\n";
exit(1);
}
// Each email sent has its status set to 0;
Crontab runs a command of this kind:
* * * * * php /var/www/projectname/public/index.php controller=name action=send-email-cli >> /var/www/projectname/application/data/logs/cron.log
It doesn't make sense to run the bootstrap in the same directory or in cron job folder. I've created a better and easy way to implement the cron job work. Please follow the below things to make your work easy and smart:
Create a cron job folder such as "cron" or "crobjob" etc. whatever you want.
Sometimes we need the cron job to run on a server with different interval like for 1 hr interval or 1-day interval that we can setup on the server.
Create a file in cron job folder like I created an "init.php", Now let's say you want to send a newsletter to users in once per day. You don't need to do the zend code in init.php.
So just set up the curl function in init.php and add the URL of your controller action in that curl function. Because our main purpose is that an action should be called on every day. for example, the URL should be like this:
https://www.example.com/cron/newsletters
So set up this URL in curl function and call this function in init.php in the same file.
In the above link, you can see "cron" is the controller and newsletters is the action where you can do your work, in the same way, don't need to run the bootstrap file etc.