I'm having an issue in a Laravel Zero project I'm working on. I'm working on a command that handles direct file transfers between 2 disks—1 SFTP and another local.
I have configured both correctly and tested that I'm able to transfer files between them using the Storage code below. My issue pops up when I try to do this using the spatie/async package to create a pool of concurrent transfers (or maybe just the way I'm trying to do it).
$pool = Pool::create()
->concurrency($limit);
$progress = $this->output->createProgressBar($file_list);
if(!Storage::disk('local')->exists($local_folder_path)) {
Storage::disk('local')->makeDirectory($local_folder_path);
}
foreach($file_list as $filename => $remote_path) {
$pool->add(function() use ($remote_path, $filename, $local_folder_path) {
Storage::disk('local')
->writeStream(
"{$local_folder_path}/{$filename}",
Storage::disk('remote')->readStream($remote_path)
);
return $filename;
})->then(function($filename) use (&$progress) {
$this->info("{$filename} downloaded");
$progress->advance();
})->catch(function($exception) {
$this->error($exception);
});
}
$pool->wait();
$progress->finish();
By the way, the error, RuntimeException: a facade root has not been set, is being printed to my console via the catch() handler for the first item in the pool. I did discover that much.
I've searched for answers to this issue, but all of the articles and other SO/Stack Exchange postings I've come across didn't seem even similar to whatever's causing my issue.
Thanks in advance for any help.
The problem is that your callback (child process) is running without any setup.
A Task is useful in situations where you need more setup work in the child process. Because a child process is always bootstrapped from nothing, chances are you'll want to initialise eg. the dependency container before executing the task.
The facades are setup by the kernel that runs the \LaravelZero\Framework\Bootstrap\RegisterFacades::class.
You can create an instance of the kernel and run the bootstrap method to have your facades setup properly.
$pool->add(function() use ($remote_path, $filename, $local_folder_path) {
$app = require __DIR__.'/../../bootstrap/app.php';
$kernel = $app->make(\Illuminate\Contracts\Console\Kernel::class);
$kernel->bootstrap();
Storage::disk('local')
->writeStream(
"{$local_folder_path}/{$filename}",
Storage::disk('remote')->readStream($remote_path)
);
return $filename;
})
Related
There are two noisy console commands in my Laravel 5.3 app that I want to keep logs for but would prefer to have them write to a different log file from the rest of the system.
Currently my app writes logs to a file configured in bootstrap/app.php using $app->configureMonologUsing(function($monolog) { ...
Second prize is writing all console commands to another log file, but ideally just these two.
I tried following these instructions (https://blog.muya.co.ke/configure-custom-logging-in-laravel-5/ and https://laracasts.com/discuss/channels/general-discussion/advance-logging-with-laravel-and-monolog) to reroute all console logs to another file but it did not work and just caused weird issues in the rest of the code.
If this is still the preferred method in 5.3 then I will keep trying, but was wondering if there was newer method or a method to only change the file for those two console commands.
They are two approaches you could take
First, you could use Log::useFiles or Log::useDailyFiles like suggests here.
Log::useDailyFiles(storage_path().'/logs/name-of-log.log');
Log::info([info to log]);
The downside of this approach is that everything will still be log in your default log file because the default Monolog is executed before your code.
Second, to avoid to have everything in your default log, you could overwrite the default logging class. An exemple of this is given here. You could have a specific log file for let's say Log::info() and all the others logs could be written in your default file. The obvious downside of this approach is that it requires more work and code maintenance.
This is possible but first you need to remove existing handlers.
Monolog already has had some logging handlers set, so you need to get rid of those with $monolog->popHandler();. Then using Wistar's suggestion a simple way of adding a new log is with $log->useFiles('/var/log/nginx/ds.console.log', $level='info');.
public function fire (Writer $log)
{
$monolog = $log->getMonolog();
$monolog->popHandler();
$log->useFiles('/var/log/nginx/ds.console.log', $level='info');
$log->useFiles('/var/log/nginx/ds.console.log', $level='error');
...
For multiple handlers
If you have more than one log handler set (if for example you are using Sentry) you may need to pop more than one before the handlers are clear. If you want to keep a handler, you need to loop through all of them and then readd the ones you wanted to keep.
$monolog->popHandler() will throw an exception if you try to pop a non-existant handler so you have to jump through hoops to get it working.
public function fire (Writer $log)
{
$monolog = $log->getMonolog();
$handlers = $monolog->getHandlers();
$numberOfHandlers = count($handlers);
$saveHandlers = [];
for ($idx=0; $idx<$numberOfHandlers; $idx++)
{
$handler = $monolog->popHandler();
if (get_class($handler) !== 'Monolog\Handler\StreamHandler')
{
$saveHandlers[] = $handler;
}
}
foreach ($saveHandlers as $handler)
{
$monolog->pushHandler($handler);
}
$log->useFiles('/var/log/nginx/ds.console.log', $level='info');
$log->useFiles('/var/log/nginx/ds.console.log', $level='error');
...
For more control over the log file, instead of $log->useFiles() you can use something like this:
$logStreamHandler = new \Monolog\Handler\StreamHandler('/var/log/nginx/ds.console.log');
$pid = getmypid();
$logFormat = "%datetime% $pid [%level_name%]: %message%\n";
$formatter = new \Monolog\Formatter\LineFormatter($logFormat, null, true);
$logStreamHandler->setFormatter($formatter);
$monolog->pushHandler($logStreamHandler);
I have a PHP daemon script running on the command line that can be connected to via telnet etc and be fed commands.
What it does with the command is based on what modules are loaded, which is currently done at the start. (psuedocode below for brevity)
$modules = LoadModules();
StartConnection();
while(true){
ListenForCommands();
}
function LoadModules(){
$modules = Array();
$dir = scandir("modules");
foreach($dir as $folder){
include("modules/".$folder."/".$folder.".php");
$modules[$folder] = new $folder;
}
}
function ListenForCommands(){
if(($command = GetData())!==false){
if(isset($modules[$command])){
$modules[$command]->run();
}
}
}
So, an example module called "bustimes" would be a class called bustimes, living in /modules/bustimes/bustimes.php
This works fine. However, I'd like to make it so modules can be updated on the fly, so as part of ListenForCommands it looks at the filemtime of the module, works out if it's changed, and if so, effectively reloads the class.
This is where the problem comes in, obviously if I include the class file again, it'll error as the class already exists.
All of the ideas I have of how to get around this problem so far are pretty sick and I'd like to avoid doing.
I have a few potential solutions so far, but I'm happy with none of them.
when a module updates, make it in a new namespace and point the reference there
I don't like this option, nor am I sure it can be done (as if I'm right, namespaces have to be defined at the top of the file? That's definitely workaroundable with a file_get_contents(), but I'd prefer to avoid it)
Parsing the PHP file then using runkit-method-redefine to redefine all of the methods.
Anything that involves that kind of parsing is a bad plan.
Instead of including the file, make a copy of the file with everything the same but str_replacing the class name to something with a rand() on the end or similar to make it unique.
Does anyone have any better ideas about how to either a) get around this problem or b) restructure the module system so this problem doesn't occur?
Any advice/ideas/constructive criticism would be extremely welcome!
You should probably load the files on demand in a forked process.
You receive a request
=> fork the main process, include the module and run it.
This will also allow you to run several commands at once, instead of having to wait for each one to run before launching the next.
Fork in php :
http://php.net/manual/en/function.pcntl-fork.php
Tricks with namespaces will fail if module uses external classes (with relative paths in namespace).
Trick with parsing is very dangerous - what if module should keep state? What if not only methods changed, but, for example, name of implemented interface? How it will affect other objects if they have link to instance of reloaded class?
I think #Kethryweryn is something you can try.
I want to be able to call the CakeS3 plugin from the Cake Shell. However, as I understand it components cannot be loaded from the shell. I have read this post outlining strategies for overcoming it: using components in Cakephp 2+ Shell - however, I have had no success. The CakeS3 code here is similar to perfectly functioning cake S3 code in the rest of my app.
<?php
App::uses('Folder','Utility');
App::uses('File','Utility');
App::uses('CakeS3.CakeS3','Controller/Component');
class S3Shell extends AppShell {
public $uses = array('Upload', 'User', 'Comment');
public function main() {
$this->CakeS3 = new CakeS3.CakeS3(
array(
's3Key' => 'key',
's3Secret' => 'key',
'bucket' => 'bucket')
);
$this->out('Hello world.');
$this->CakeS3->permission('private');
$response = $this->CakeS3->putObject(WWW_ROOT . '/file.type' , 'file.type', $this->CakeS3->permission('private'));
if ($response == false){
echo "it failed";
} else {
echo "it worked";
}
}
This returns an error of "Fatal error: Class 'CakeS3' not found in /home/app/Console/Command/S3Shell.php. The main reason I am trying to get this to work is so I can automate some uploads with a cron. Of course, if there is a better way, I am all ears.
Forgive me this "advertising"... ;) but my plugin is probably better written and has a better architecture than this CakeS3 plugin if it is using a component which should be a model or behaviour task. Also it was made for exactly the use case you have. Plus it supports a few more storage systems than only S3.
You could do that for example in your shell:
StorageManager::adapter('S3')->write($key, StorageManager::adapter('Local')->read($key));
A file should be handled as an entity on its own that is associated to whatever it needs to be associated to. Every uploaded file (if you use or extend the models that come with the plugin, if not you have to take care of that) is stored as a single database entry that contains the name of the config that was used and some meta data for that file. If you do the line of code above in your shell you will have to keep record in the table if you want to access it this way later. Just check the examples in the readme.md out. You don't have to use the database table as a reference to your files but I really recommend the system the plugin implements.
Also, you might not be aware that WWW_ROOT is public accessible, so in the case you store sensitive data there it can be accessed publicly.
And finally in a shell you should not use echo but $this->out() for proper shell output.
I think the App:uses should look like:
App::uses('CakeS3', 'CakeS3.Controller/Component');
I'm the author of CakeS3, and no I'm afraid there is no "supported" way to do this as when we built this plugin, we didn't need to run uploads from shell and just needed a simple interface to S3 from our controllers. We then open sourced the plugin as a simple S3 connector.
If you'd like to have a go at modifying it to support shell access, I'd welcome a PR.
I don't have a particular road map for the plugin, so I've tagged your issue on github as an enhancement and will certainly consider it in future development, but I can't guarantee that it would fit your time requirements so that's why I mention you doing a PR.
I need to be able to "effectivly" redeclare my class, so that during runtime, whilst my PHP IRC bot is running I can reload modules as the code base changes, which requires getting the class, but PHP won't allow it to be redeclared, nor is there a way to undeclare a class.
Is there a way I can acheive this? Perhaps some other method? I've tried "runkit", but that failed.
My current code is here:
http://pastie.org/private/ptj7c0t0teh3nnzn7ehcg
So to clarify, I effecivly need to be able to reload my class, instatiate it and put into a property in my main bot class once code has changed in said module, whilst the bot is running (run-time).
A brief look at your application leads me to believe your best course of action is to use a base class as suggested in the comment above.
Something to the extent of:
class IRCModule {
private static $module_dir = ''; //put your module directory here w/o trailing slash
public static function getModule( $module ) {
//if the module directory doesn't exist, don't do anything
if( !is_dir( self::$module_dir.DIRECTORY_SEPARATOR.$module ) ) return false;
//load the module file
$fname = scandir(self::$module_dir.DIRECTORY_SEPARATOR.$module);
$fname = $fname[2]; //first 2 elements will be . and ..
require_once( self::$module_dir.DIRECTORY_SEPARATOR.$module.DIRECTORY_SEPARATOR.$fname );
$className = str_replace('.class.php',NULL,$fname);
return new $className();
}
}
You would then extend that using your modules. That would allow you to overwrite a module by simply removing it's old file /my/module/dir/moduleName/moduleNameV1.0.class.php and replacing it with a new version /my/module/dir/moduleName/moduleNameV1.1.class.php
As mentioned in the comments, this will eventually fill the memory on the server, so you should schedule a reboot of the service each time you make substantial changes, but it also allows you to load new versions on demand without stopping the service.
A more stable approach would be to take advantage of process control and spin off daemons for each connection from your parent script, or implement a cache system that stores all data on the disk/database so that you can detect a change in module version and instantly reboot the server. But the solution above should work for you for the time being :)
I am trying to write a cronjob controller, so I can call one website and have all modules cronjob.php executed. Now my problem is how do I do that?
Would curl be an option, so I also can count the errors and successes?
[Update]
I guess I have not explained it enough.
What I want to do is have one file which I can call like from http://server/cronjob and then make it execute every /application/modules/*/controller/CronjobController.php or have another way of doing it so all the cronjobs aren't at one place but at the same place the module is located. This would offer me the advantage, that if a module does not exist it does not try to run its cronjob.
Now my question is how would you execute all the modules CronjobController or would you do it a completly different way so it still stays modular?
And I want to be able to giveout how many cronjobs ran successfully and how many didn't
After some research and a lot procrastination I came to the simple conclusion that a ZF-ized cron script should contain all the functionality of you zend framework app - without all the view stuff. I accomplished this by creating a new cronjobfoo.php file in my application directory. Then I took the bare minimum from:
-my front controller (index.php)
-my bootstrap.php
I took out all the view stuff and focused on keeping the environment setup, db setup, autoloader, & registry setup. I had to take a little time to correct the document root variable and remove some of the OO functionality copied from my bootstrap.
After that I just coded away.. in my case it was compiling and emailing out nightly reports. It was great to use Zend_Mail. When I was confident that my script was working the way I wanted, I just added it my crontab.
good luck!
For Zend Framework I am currently using the code outlined bellow. The script only includes the portal file index.php, where all the paths, environment and other Zendy code is bootstrapped. By defining a constant in the cron script we cancel the final step , where the application is run.
This means the application is only setup, not even bootstrapped. At this point we start bootstraping the resources we need and that is that
//public/index.php
if(!defined('DONT_RUN_APP') || DONT_RUN_APP == false) {
$application->bootstrap()->run();
}
// application/../cron/cronjob.php
define("DONT_RUN_APP",true);
require(realpath('/srv/www/project/public/index.php'));
$application->bootstrap('config');
$application->bootstrap('db');
//cron code follows
I would caution putting your cronjobs accessible to the public because they could be triggered outside their normal times and, depending on what they do, cause problems (I know that is not what you intend, but by putting them into an actual controller it becomes reachable from the browser). For example, I have one cron that sends e-mails. I would be spammed constantly if someone found the cron URL and just began hitting it.
What I did was make a cron folder and in there created a heartbeat.php which bootstraps Zend Framework (minus MVC) for me. It checks a database which has a list of all the installed cron jobs and, if it is time for them to run, generates an instances of the cron job's class and runs it.
The cron jobs are just child classes from an abstract cron class that has methods like install(), run(), deactivate(), etc.
To fire off my job I just have a simple crontab entry that runs every 5 minutes that hits heartbeat.php. So far it's worked wonderful on two different sites.
Someone mentioned this blog entry a couple days ago on fw-general (a mailinglist which I recommend reading when you use the Zend Framework).
There is also a proposal for Zend_Controller_Request_Cli, which should address this sooner or later.
I have access to a dedicated server and I initially had a different bootstrap for the cron jobs. I eventually hated the idea, just wishing I could do this within the existing MVC setup and not have to bother about moving things around.
I created a file cron.sh, saved is within my site root (not public) and in it I put a series of commands I would like to run. As I wanted to run many commands at once I wrote the PHP within my controllers as usual and added curl calls to those urls within cron.sh. for example curl http://www.mysite.com/cron_controller/action Then on the cron interface I ran bash /path/to/cron.sh.
As pointed out by others your crons can be fired by anyone who guesses the url so there's always that caveat. You can find a solution to that in many different ways.
Take a look at zf-cli:
scripts at master from padraic/ZFPlanet - GitHub
This handles well all cron jobs.
Why not just create a crontab.php, including, or requiring the index.php bootstrap file?
Considering that the bootstrap is executing Zend_Loader::registerAutoload(), you can start working directly with the modules, for instance, myModules_MyClass::doSomething();
That way you are skipping the controllers. The Controller job is to control the access via http. In this case, you don't need the controller approach because you are accessing locally.
Do you have filesystem access to the modules' directories? You could iterate over the directories and determine where a CronjobController.php is available. Then you could either use Zend_Http_Client to access the controller via HTTP or use an approach like Zend_Test_PHPUnit: simulate the actual dispatch process locally.
You could set up a database table to hold references to the cronjob scripts (in your modules), then use a exec command with a return value on pass/fail.
I extended gregor answer with this post. This is what came out:
//public/index.php
// Run application, only if not started from command line (cli)
if (php_sapi_name() != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
$application->run();
}
Thanks gregor!
My solution:
curl /cron
Global cron method will include_once all controllers
Check whether each of the controllors has ->cron method
If they have, run those.
Public cron url (for curl) is not a problem, there are many ways to avoid abuse. As said, checking remote IP is the easiest.
This is my way to run Cron Jobs with Zend Framework
In Bootstrap I will keep environment setup as it is minus MVC:
public static function setupEnvironment()
{
...
self::setupFrontController();
self::setupDatabase();
self::setupRoutes();
...
if (PHP_SAPI !== 'cli') {
self::setupView();
self::setupDbCaches();
}
...
}
Also in Bootstrap, I will modify setupRoutes and add a custom route:
public function setupRoutes()
{
...
if (PHP_SAPI == 'cli') {
self::$frontController->setRouter(new App_Router_Cli());
self::$frontController->setRequest(new Zend_Controller_Request_Http());
}
}
App_Router_Cli is a new router type which determines the controller, action, and optional parameters based on this type of request: script.php controller=mail action=send. I found this new router here: Setting up Cron with Zend Framework 1.11
:
class App_Router_Cli extends Zend_Controller_Router_Abstract
{
public function route (Zend_Controller_Request_Abstract $dispatcher)
{
$getopt = new Zend_Console_Getopt (array());
$arguments = $getopt->getRemainingArgs();
$controller = "";
$action = "";
$params = array();
if ($arguments) {
foreach($arguments as $index => $command) {
$details = explode("=", $command);
if($details[0] == "controller") {
$controller = $details[1];
} else if($details[0] == "action") {
$action = $details[1];
} else {
$params[$details[0]] = $details[1];
}
}
if($action == "" || $controller == "") {
die("Missing Controller and Action Arguments == You should have:
php script.php controller=[controllername] action=[action]");
}
$dispatcher->setControllerName($controller);
$dispatcher->setActionName($action);
$dispatcher->setParams($params);
return $dispatcher;
}
echo "Invalid command.\n", exit;
echo "No command given.\n", exit;
}
public function assemble ($userParams, $name = null, $reset = false, $encode = true)
{
throw new Exception("Assemble isnt implemented ", print_r($userParams, true));
}
}
In CronController I do a simple check:
public function sendEmailCliAction()
{
if (PHP_SAPI != 'cli' || !empty($_SERVER['REMOTE_ADDR'])) {
echo "Program cannot be run manually\n";
exit(1);
}
// Each email sent has its status set to 0;
Crontab runs a command of this kind:
* * * * * php /var/www/projectname/public/index.php controller=name action=send-email-cli >> /var/www/projectname/application/data/logs/cron.log
It doesn't make sense to run the bootstrap in the same directory or in cron job folder. I've created a better and easy way to implement the cron job work. Please follow the below things to make your work easy and smart:
Create a cron job folder such as "cron" or "crobjob" etc. whatever you want.
Sometimes we need the cron job to run on a server with different interval like for 1 hr interval or 1-day interval that we can setup on the server.
Create a file in cron job folder like I created an "init.php", Now let's say you want to send a newsletter to users in once per day. You don't need to do the zend code in init.php.
So just set up the curl function in init.php and add the URL of your controller action in that curl function. Because our main purpose is that an action should be called on every day. for example, the URL should be like this:
https://www.example.com/cron/newsletters
So set up this URL in curl function and call this function in init.php in the same file.
In the above link, you can see "cron" is the controller and newsletters is the action where you can do your work, in the same way, don't need to run the bootstrap file etc.