I'm looking for a sustainable solution to fetch data every x seconds (let's say 20) and store it in a relational database using PHP. After doing some research I found a few options:
1) Cronjobs (with shell scripts)
See https://askubuntu.com/questions/800/how-to-run-scripts-every-5-seconds for more information. This basically comes down to run a shell script (looping/sleeping)
This doesn't feel right as I could not catch exceptions and/or race conditions might occur. Also, cronjobs itself are not made for this kind of tasks.
2) Web-worker (with queued jobs)
Laravel provides a queue worker that can process new jobs (asynchronously) as they are pushed onto the queue. I could push multiple (say a lot) of jobs to the queue at once which should processed every x seconds consecutively.
This sounds like a more robust solution as I could catch exceptions and make sure the worker is running (using observers). The downside; it's slower and it might be overengineered.
3) Web socket
I could use node.js to run a websocket client like socket.io and implement some kind of timing mechanism to store the data every x seconds.
This solution feels odd as I was taught that sockets are used to push data to clients (realtime), but I have never seen that they were used to insert data.
All help is appreciated.
What you are looking for are artisan commands.
You would start by creating a command:
php artisan make:command FetchData
This creates a FetchData class. In this class you can edit the handle function.
public function handle()
{
//fetch your data here
}
You also need to edit the $signature variable.
protected $signature = 'fetch:data';
The next step is to register the command in the Kernel.php in the Console namespace.
You need to add your newly created FetchData class to the $commands array.
protected $commands = [
Commands\FetchData::class
];
You could now call this command from the console like php artisan fetch:data
After you registered your command in the Kernel.php you can schedule this command.
You start by adding following line to your crontab on your server (type crontab -e)
* * * * * php /path-to-your-project/artisan schedule:run >> /dev/null 2>&1
You can now add following command to the schedule function in the Kernel.php:
$schedule->command('fetch:data')->everyThirtyMinutes();
There is no option for a job to run every twenty minutes so in this example I chose thirty minutes. You can check the available options here.
Related
Laravel has Task Scheduling where events, job and commands can be scheduled. I just can't figure out where does it store them? It doesn't look like it stores it in database
Your batch jobs can be stored in a database table, then use the scheduler to execute it.
The function below can be put in app/console/Kernel.php
$schedule->call(function () {
//put your logic here e.g. send verification code to newly
//registered users at 10 pm
})->dailyAt('22:00');
To execute the scheduler, you must specify the name of the command:
protected $commands = [
send:email
];
Then you would call it in terminal as php artisan send:email and keep the terminal process running.
Or you can setup a cron job
My question is more of a general wondering. I have two commands created using Laravel, let's call them A and B.
Each one of these commands are scheduled with the ->dailyAt($par) method. But the $par parameter comes from a query.
I mean something like this:
protected function schedule(Schedule $schedule)
{
$schedulerTime_commandA = App\Model\CommandsTime::where('id', 1)->first()->time;
$schedulerTime_commandB = App\Model\CommandsTime::where('id', 2)->first()->time;
$schedule->command('A')
->dailyAt($schedulerTime_commandA);
$schedule->command('B')
->dailyAt($schedulerTime_commandB);
}
This is because the superuser wants to schedule the time when those commands will run. My question here is: How Laravel knows this schedule method within the App\Console\Kernel.php file has been changed?
NOTE: I have the following cron entry as Laravel talks about it on the docs.
* * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1
The way Laravel's scheduler system works is it uses a cron job that runs once every minute (that's the * * * * * part in the cron entry: match every minute of every hour of every day of every month of every year).
So every minute, php /path/to/artisan schedule:run >> /dev/null 2>&1 is being run by cron.
When that command runs, it should check the schedule as defined by the schedule() method in the Kernel class when it is run.
So technically, Laravel doesn't know that the schedule has changed, per se. Every minute it should run, hit the schedule() method, which will grab the latest values from the database, and then return the schedule as it is set at that particular minute in time.
Each individual run of the cron knows nothing about the ones that came before it, or the ones that will come after it.
At least that's my understand of the scheduler. I've only spent a little time in the core Laravel Kernel code, but I believe that to what is going on from my own experience.
Hope that helps!
Edit 1
Just confirmed my logic in Laravel's code. Every time the cron script runs, the schedule is rebuilt, so the changes to the schedule in the database will be used the next time the cron entry runs. Laravel doesn't really know that it changed - it just checks every time it runs and uses what it finds.
Specifically, here's the chain through code if it's of interest:
Calling php artisan on the command line will run the artisan file in the project root;
In the artisan file the Application is bootstrapped and a Kernel object is initialized;
When the Kernel class is initialized, the constructor for the Kernel class (Laravel\Lumen\Console\Kernel) calls the defineConsoleSchedule() method on itself;
The defineConsoleSchedule() method initializes a blank Schedule object (Illuminate\Console\Scheduling\Schedule);
The blank Schedule object will be passed to the schedule() method on the Kernel class; and finally
In the schedule() method, which is where you defined your command schedule, your two DB queries will be run, and your two schedule entries will be defined with the values that the DB returns at that moment.
The above actions happen every time the console application bootstraps, which means on every call to php artisan, regardless of what command you wish to run.
When running the schedule:run command specifically, here's what happens next:
After the application bootstraps, the ScheduleRunCommand object (Illuminate\Console\Scheduling\ScheduleRuneCommand) is initialized; and
The fire() method is called on the ScheduleRunCommand object, which runs through all the commands defined on the Schedule object when the application bootstrapped; and
Checks each command to see if that command must be run at that time by calling the isDue() method on the Event (Illuminate\Console\Scheduling\Event) that represents the command; and finally
Each command that needs to be run at that minute, by returning true from isDue() will be run.
I've set up the following Laravel commands on the App\Console\Kernel:
protected function schedule(Schedule $schedule) {
$schedule->command('command:daily-reset')->daily();
$schedule->command('command:monthly-reset')->monthly();
}
Then, on my server, I've set up a cron job to run once per day (at 00:00).
0 0 * * * php /home/privates/public_html/staging/current/artisan schedule:run
My cron job is running successfully each night, but the logs simply say: "No scheduled commands are ready to run."
What am I doing wrong? I would expect my daily command to run each night.
Thanks!
When you run
php artisan schedule:run
in the server, where your project is stored, you could see all of your commands running with output, looking like this:
"Running scheduled command: '/usr/local/bin/php' 'artisan' cache:update > '/dev/null' 2>&1 &"
but only if the current time is the exact one, for which the command is scheduled. Otherwise you are going to see this output:
"No scheduled commands are ready to run."
For example, if you schedule the command for every five minutes and run the command in 09:07 o'clock you will see that there are no scheduled commands, but if you run it in 09:10 you will see your command running.
In this way you can just schedule your command to run every 5 min just for debugging purposes:
$schedule->command('command:daily-reset')->everyFiveMinutes();
then observe if there is any error while running and eventually fix it. By me the problem was that I haven't installed GuzzleHttp (shame), so the fix was just running this in the terminal:
composer require guzzlehttp/guzzle
I realized that the problem for me was the below chained method:
->withoutOverlapping()
Once I removed that method, my commands started running and being found by the daemon process.
I think there might be a bug with the method, but my project for now can take a bit overlapping so it's cool.
Did you try running command manually?
Run php artisan and see if your commands have registered.
If you have registered your commands you should see command:daily-reset and command:monthly-reset under the list of available artisan commands.
If you don't see them there go ahead and register your commands by adding it to commands property available in app/Console/Kernel.php.
protected $commands = [
'App\Console\Commands\YourFirstCommand',
'App\Console\Commands\YourSecondCommand'
];
Change crontab entry to
* * * * * php /home/privates/public_html/staging/current/artisan schedule:run
The Laravel scheduled commands are based in the timezone that you have configured in your app/config/app.php file (laravel 5.1):
/*
|--------------------------------------------------------------------------
| Application Timezone
|--------------------------------------------------------------------------
|
| Here you may specify the default timezone for your application, which
| will be used by the PHP date and date-time functions. We have gone
| ahead and set this to a sensible default for you out of the box.
|
*/
'timezone' => 'America/Bogota',
So if you create a command and register it to run as a scheduled task with:
$schedule->command('command:daily-reset')->daily();
it will run every day at 00:00 OF THE TIMEZONE SPECIFIED (in this case America/Bogota)
The same thing applies if you specify a time to run the task:
$schedule->command('command:daily-reset')->daily()->at('02:30');
This will run at 02:30 am in America/Bogota local time.
NB: This is not answer for this question, but a clue for anyone debugging with php artisan schedule:run manually. Hope it saves someone a few minutes of headache.
Check if the scheduled task can run immediately. You can use the exec method for that.
<?php
//...
protected function schedule (Schedule $schedule) {
$schedule->exec("php artisan your:command");
}
The reason for this is that, you might be scheduling the task to run at a certain time and if that time isn't due yet, it will output:
No scheduled commands are ready to run.
The full answer to this question is not listed above as far as I can see. Let's assume that our schedule is as follows:
protected function schedule(Schedule $schedule)
{
$schedule
-> command('cbh:dummyCommand')
-> everyFiveMinutes()
-> appendOutputTo ('/my/logs/laravel_output.log');
}
What I've discovered is that this code doesn't set your job to run every 5 minutes. Nor does it prevent the command running again if it was run less than 5-minutes ago.
A better way to think about it is that this code sets the named command "to be runnable every time the minute-figure of the current time is 0 or 5". In other words, if I run the command-line argument: php artisan schedule:run at 11:04, then the response is:
# No scheduled commands are ready to run.
But if I run the same command at 11:00 or 11:05, then we get:
# Running scheduled command: php artisan cbh:dummyCommand >> /my/logs/laravel_output.log 2>&1
And I end up with output in my log-file.
I discovered the above when my everyFiveMinutes() schedule was creating a log in my file every 10 minutes based on the fact that my task-scheduler was running every 2 minutes.
However, this doesn't quite address your issue, given that the daily() schedule (0 0 * * *) aligns with your cron-job schedule. The only thing I can imagine is that there is some kind of misalignment with your time-zones as suggested by #Octavio Herrera. But that's difficult to say without knowing a bit more about your environment.
I had the same problem. Every command was correctly registered but I always received the “No scheduled commands are ready to run.” message. The problem was that the website was in "maintenance mode" (php artisan down command) while we were doing updates and tests.
I think that my blog will help you answer your question. Please see the below or link: Laravel Crontab
In many projects, you need use crontab (cron jobs) to execute some tasks as sending email or delete waste record in DB. With Laravel Project, you can do this easier.
Create a command in Laravel 4:
<?php
use Illuminate\Console\Command;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Input\InputArgument;
class FirstCommand extends Command {
/**
* The console command name.
*
* #var string
*/
protected $name = 'user:active';
/**
* The console command description.
*
* #var string
*/
protected $description = 'Command description.';
/**
* Create a new command instance.
*
* #return void
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* #return mixed
*/
public function fire()
{
echo "User Actived";
}
/**
* Get the console command arguments.
*
* #return array
*/
protected function getArguments()
{
return array(
);
}
/**
* Get the console command options.
*
* #return array
*/
protected function getOptions()
{
return array(
array('example', null, InputOption::VALUE_OPTIONAL, 'An example option.', null),
);
}
}
Next step, you need to register the command with Laravel CLI. So easy, you open app/start/artisan.php file, and add one line as below:
Artisan::add(new FirstCommand);
You are done creating Laravel Command. To test, you could use command below:
$ php artisan user:active
User Active
The output above mean you successfully register a command.
Finally, put your command into the crontab:
crontab -e
Add line (run command every 2 minutes):
*/2 * * * * php path_to_laravel_project/artisan user:active
That’s all. Thank you for talking time to read this.
On Windows, I fixed this issue by setting the Scheduled Task to run every minute (even though I only wanted to trigger a command once per day), otherwise I always got the No scheduled commands are ready to run. message.
Since I still ran into this issue 4 years later (2019) and a different workaround worked for me, I think it is worth hinting the simple step that solved for me, which is: Use a shorter interval first.
That seems to just wake it up to handle longer intervals in some ways. I had everyFiveMinutes() and for almost 2 hours it was getting the No scheduled commands are ready to run response. I simply changed it to everyMinute() and it started running correctly. I watched it consistently for like 10 minutes or so, then changed it back to everyFiveMinutes() and it all went smoothly.
I've stuck with this problem No scheduled commands are ready to run. for an hours, but solve it easly:
Problem was with rights to folder storage.
So, i've set chmod -R 777 storage/* (i'm not sure is this is elegant way).
After that cron starts working properly.
To run the Cron Commands on the local server, follow these steps:
I know you have already mentioned the command in app/console/Kernel.php
Now open the command line, enter "crontab -e"
Edit that file and mention the below code(without quote) to keep running PHP artisan schedule:run in the background
"* * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1"
Enter "crontab -l" in the command line, it will list running crons
Done !!
Now, wait for cron to process your command. Cheers!!
Reference- https://laravel.com/docs/7.x/scheduling#introduction
For whatever reason cron does not recognize the named version of your task.
So in your schedule instead of writing
$schedule->command('command:task');
you should use the path of the class, such as
$schedule->command(\App\Console\Commands\TASK::class)
...the same goes for the scheduler on Laravel Forge!
I have tried everything but finally I found a solution for this problem. Add the timestamp in the command. Below is the example for this.
$schedule->call(function(){
print("HELLO");
})->dailyAt('21:51')->timezone('Asia/Kolkata');
or
$schedule->command('tenam:before')
->dailyAt('22:28')->timezone('Asia/Kolkata');
Try
php artisan cache:clear
and then run scheduler again with
php artisan schedule:run
I was also facing same issue and it resolved my problem.
I have an event that is fired when I receive certain notifications. I want to Queue the event so that they aren't all fired at the same time but are Queued up as I receive them and then fired after the previous event completes. I want to know the best way to do this.
Edit: Just for anyone in the future, setting up the database Queue driver is very straightforward and simple. You run the php artisan queue:table and change the driver to 'database'. My problem was that my app wasn't recognizing my QUEUE_DRIVER setting in my .env file for some reason.
Laravel 5 has it's own way of dealing with queued jobs, but you can still use the options that were available in Laravel 4. I've personally been curious as to how it all works and just threw together a blank project and ran a couple of queued jobs with a little help from the docs so this may not be a full answer but I hope this helps you on your way.
First you will want to set your config to use the database queue driver, this can be done in config/queue.php or for me it was a matter of going to the .env file and doing this: QUEUE_DRIVER=database.
Then you want to set up the database table to hold the queued jobs, you can do this by running an artisan command: php artisan queue:table this will create the migration so then you need to create the table by running php artisan migrate and then you'll have your jobs table in your DB.
Following that, you'll want to set up a queued job which come in the form of Commands. For example I'll set up a job that writes some text to the log file. You can create jobs or commands using an artisan command, here's what I did to create a command: php artisan make:command WriteToLog --queued. And here's what my command class looks like after adding a little code to get it to write to the log file...
app/Commands/WriteToLog.php
use App\Commands\Command;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Bus\SelfHandling;
use Illuminate\Contracts\Queue\ShouldBeQueued;
class WriteToLog extends Command implements SelfHandling, ShouldBeQueued {
use InteractsWithQueue, SerializesModels;
protected $secs;
/**
* Create a new command instance.
*
* #return void
*/
public function __construct($secs)
{
$this->secs = $secs;
}
/**
* Execute the command.
*
* #return void
*/
public function handle()
{
\Log::info('Writing to the log in ' . $this->secs);
}
}
After creating a command, to test it out I wrote a route in my routes file ...
app/Http/routes.php
Route::get('/', function(){
// some time to delay the job
$fiveSecs = \Carbon\Carbon::now()->addSeconds(5);
$tenSecs = \Carbon\Carbon::now()->addSeconds(10);
// adds job to queue
Queue::later($fiveSecs, new App\Commands\WriteToLog('5 secs'));
Queue::later($tenSecs, new App\Commands\WriteToLog('10 secs'));
return 'All done';
});
Before we hit the route we want to listen for any jobs in order to process them, just run php artisan queue:listen then you can go to your browser to the route, after hitting the route in my browser the console shows
$ php artisan queue:listen
Processed: Illuminate\Queue\CallQueuedHandler#call
Processed: Illuminate\Queue\CallQueuedHandler#call
And if I check my log file I see the following:
[2015-05-19 19:25:08] local.INFO: Writing to the log in 5 secs
[2015-05-19 19:25:10] local.INFO: Writing to the log in 10 secs
Not exactly 5 and 10 seconds apart but hopefully you get the idea!
For me this is really just the tip of the iceberg and queued jobs are something very powerful in laravel, I highly recommend checking out the docs here: http://laravel.com/docs/5.0/queues and here: http://laravel.com/docs/5.0/bus
You can also fire events from your queued jobs or queue an event handler, see here for more details: http://laravel.com/docs/5.0/events#queued-event-handlers
Laravel makes queues pretty straightforward, but a bit long to explain fully here. Check out these guides:
If you are using forge, it is really painless:
https://mattstauffer.co/blog/laravel-forge-adding-a-queue-worker-with-beanstalkd
If you aren't using forge, it is still pretty ok: http://fideloper.com/ubuntu-beanstalkd-and-laravel4
I have added some jobs to a queue in Laravel. However, I forgot to put $job->delete() in the function and there is an error in my function. This means the job is never ending. It keeps going being replaced onto the queue and keeps erroring in my log file. How can I delete it from the command line?
I am using beanstalkd for my queuing.
I am using Redis instead of Beanstalkd but this should be the same in both. Restarting Redis doesn't solve the problem. I looked at RedisQueues in the Laravel 4.2 API Docs and found:
public Job|null pop(string $queue = null)
//Pop the next job off of the queue.
This is the same if you look at BeanstalkedQueue.
I threw it in app/routes.php inside dd*, loaded that page and voila.
Route::get('/', function() {
dd(Queue::pop());
#return View::make('hello');
});
NOTE: Reload the page once per queue.
The queue was pulled off the stack. I would like to see a cleaner solution but this worked for me more than once.
*dd($var) = Laravel's die and dump function = die(var_dump($var))
Edit 1: For Redis
The above obviously isn't the best solution so here is a better way. Be careful!
FLUSHDB - Delete all the keys of the currently selected DB. This command never fails.
For Redis use FLUSHDB. This will flush the Redis database not Laravel's database. In the terminal:
$ redis-cli
127.0.0.1:6379> FLUSHDB
OK
127.0.0.1:6379> exit
Restart Beanstalk. On Ubuntu:
sudo service beanstalkd restart
I made an artisan command which will clear all the jobs in your queue. You can optionally specify the connection and/or the pipe.
https://github.com/morrislaptop/laravel-queue-clear
Important note: This solution works only for beanstalk
There are two solutions:
1- From Your PHP Code
To delete jobs programatically, you can do this:
//Que the job first. (YourJobProcessor is a class that has a method called fire like `fire($job,$data)`
$res = Queue::later(5, YourJobProcessor::class, $data, 'queue_name');
//get the job from the que that you just pushed it to
$job = Queue::getPheanstalk()->useTube("queue_name")->peek($res);
//get the job from the que that you just pushed it to
$res = Queue::getPheanstalk()->useTube("queue_name")->delete($job);
If everything went good, the job will not execute, else the job will execute after 5 seconds
2- From Command Line (Linux and Mac only)
From command line (In linux and mac) you can use beanstool.
For example, if you want to delete 100 ready jobs from the queue_name tube you can do the following:
for i in {1..100}; do beanstool delete -t queue_name --state=ready; done
For Redis users, instead of flushing, using redis-cli I ran this command:
KEYS *queue*
on the Redis instance holding queued jobs,
then deleted whatever keys in the response
DEL queues:default queues:default:reserved
Only way I could do it was to restart my computer. Could not find a way to delete a job.
I've used this php-based web admin console in the past.
Otherwise, I believe you'll find yourself using Terminal + telnet, altho I can't find any documentation for deleting via telnet (Just viewing a list of jobs in queue).
It seems that most articles tell you to use your code+library of choice and loop around queues jobs to delete them in this situation.
Here is Laravel 5.1 compatible command, which allows you to clear Beanstalkd queue. The command takes queue name as argument ('default' by default). Do not forget to register it in app/Console/Kernel.php