I want PHP layer to execute a method (or call a web service) every month. My PHP sits in AWS elastic beanstalk environment.
My requirement is to run a task every month on a certain time.
The
triggering of event execute a PHP code that purges data that sits in
the RDS as per the defined business rules.
Cron job or MySQL triggers would not help me. Is this possible at all?
Related
I am working on the system which developed in php without framework.
It has the function is automatically run some jobs via third party api every night. It loops all the jobs in table and call api using curl.
// run cron job to loop this table
ID JOB
1 updateUser
2 getLatestInfo/topicA
……
//code
// if UpdateUser
loop user table and call api to get latest info…
Also other curl task will do here like send email / notification …
It works perfectly before. But recently we have many new users. It will call 50-100 API at the same time.
Each api call will take 10-20 seconds to respond, and we will retry the api if it is timeout.
I checked the log it totally take 3-4 hours for only first job (with many errors)
Although I can make the cron job for queueing the curl, like get first 5 curl and run them each 1 minutes. But if we keep increasing the users or task, and the third party api keep slow. It may take more hours to finish the task.
Is there any solution can make it keep listening to the job table, and run the curl one by one?
I want it can be auto Triggered if new row is added to the table. (Like websocket?) and not single php to run and infinite loop ( to prevent some error occurred and need to rerun the php task manually )
(The API keys is in the php project, so I hope that I can do this in same project)
PHP scripts need to be triggered in order to do something, they can't really "run in background" (I mean, they can, technically, but PHP isn't supposed to be used that way).
Instead, one of three options is usually used to do job management:
call jobs on every call from web, along with the actual code to generate output
use external web cron service to query specific URLs tied to job execution
use local cron job on your system to call the php executable and have it execute jobs periodically
If you want an event based system, PHP is likely the wrong option. Depending on your DB system though you might be able to create a small wrapper code that subscribes to DB changes and is triggered on inserts, that then calls PHP again - but it's definitely a cleaner solution to use a more suitable programming language / environment.
I am building a Multi-Tenant web application using Laravel/PHP that will be hosted on AWS as SaaS at the end. I have around 15-20 different background jobs that need scheduling for each tenant. The jobs need to be fired every 5 minutes as well. Thus the number of jobs which need to be fired for 100 tenants would be around 2000. I am left with 2 challenges in achieving this
Is there a cloud solution that distributes and manages the load of the scheduled jobs automatically?
If one is out there, how can we create those 15+ scheduled jobs on the fly? Is there an API available?
Looking for your assistance
Finally, I have found a solution to my problem.
We cannot scale the background jobs in the way I want. It required me to look into the solution from a completely different angle.
The ideal solution to my problem is that I should generate SQS messages (with a payload describing the tenant id, the job needs to be executed and any additional parameters) corresponding to the number of tenants on a set interval and queue it.
For example, if I have 100 tenants and I want to run "Job 1" every our, the main application will generate 100 SQS messages and queue it in a particular SQS Queue every hour. It will do the same for all 15 different jobs I have per tenant.
On the other end, a scalable AWS Lambda function listening to the SQS queue will pick up the payload and execute the intended task based on the data being carried by the payload.
But unfortunately, my expertise lies in PHP/Laravel technology which is still not in the AWS Lambda stack. Hence I figured out a workaround as follows.
I built a Docker image with my PHP/Laravel application and placed it in Amazon ECS (EC2 container service). Still, I have the AWS Lambda function in place but this time it acts as a trigger to my docker containers. The Lambda picks an SQS Message, processes the payload and spawns a Docker container on ECS based on my Docker image. I got some of the ideas from the following article to arrive at this solution.
https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/
Laravel has option to schedule Task/Jobs:
Refer: https://laravel.com/docs/6.x/scheduling
so you can keep jobs of your client in your database and than do it some like below:
Scheduling Queued Jobs
The job method may be used to schedule a queued job. This method provides a convenient way to schedule jobs without using the call method to manually create Closures to queue the job:
$schedule->job(new ClientJob)->everyFiveMinutes();
// Dispatch the job to the "clientjob" queue...
$schedule->job(new ClientJob, 'clientjob')->everyFiveMinutes();
or
Scheduling Shell Commands
The exec method may be used to issue a command to the operating system:
$schedule->exec('node /home/forge/script.js')->everyFiveMinutes();
my current script using CRON to handle the checking to DB and do the requests
so every mint the CRON will be called and check which action should be done as per the schedule table entry so if now the time to send email/publish post etc...
and this entry getting more and more with time and with many users now my CRON take around 20 to 50 mint to be done
so if I have to send email on 10 AM it sends between 10:20 AM to 10:50 AM
after searching I found RabbitMQ and Redis and other systems and I choose RabbitMQ
so what is next, what I need to do next, as for my experience I never work with a system like Redis etc.. so its something totally new, so if someone has and resources to check read or watch and help me with migrating the whole system from CRON to RabbitMQ.
small note, the current script is built on top of custom PHP framework only for this script and don't have API.
Write a php shell script for create linux pid in a infinite loop and call a method by cron.
Every job push to rabbitMq basic_publish with data set.
this method create a basic_consume with rabbitMq for performing queue with queue data set.
Hoping I can get some help (again). I'm working on a multi-tenant PHP application. Each tenant will have their own database (mysql). Ultimately, my plan is to stand the service up on AWS using ELB, EC2, and DynamoDB.
However, the app will need to run certain scheduled tasks (it needs to pull open invoices from a PSA for certain customers, and then charge the customer using Authroize.net CIM and mark it paid in the PSA).
For a regular application, I would simply create a cron script that runs daily to create/process the payment batches. I'm just not sure what the appropriate approach would be to run the cron across each tenant (for each database). Maybe one master cron job that runs across each tenant, or do I write a script to create / maintain cron jobs for each tenant using SWF?
Thanks for your input.
I've had reasonable success with doing batch processing via cron in the past. It might be helpful to record tenant creation in a table that you can query as a source for which databases to run against within the cron job.
We have a website running on multiple Azure instances – typically between 2 and 5.
There is a PHP script I would like to schedule to run every few minutes on each instance. (It just makes a local copy of data from a system that couldn't handle the load from all our users hitting it in real-time.)
If it were just one instance, that would be easy - I'd use Azure Scheduler to call www.example.com/my-scheduled-task.php every 5 minutes.
But the script needs to run on each instance, so that every instance has a reasonably up-to-date copy of the data. How would you achieve this? I can't work out if it's something in Azure Scheduler, or if I should be looking at some sort of startup script?
You can use a continuous webjob for that.
Just tweak your php script to have a loop and add a sleep of a few minutes between runs of your code.
The continuous webjob will run on all of your instances and even if somethings fails it will be brought back up.
Per my experience, a PHP webjob running on your each webapp instance is the good solution as #AmitApple said. However, I think you can try to use a scheduled webjob with a CRON expression for ensuring a start time, not a continuous one with a sleep time. And please make sure the script can be completed in the interval time.
You can refer to the section Create a scheduled WebJob using a CRON expression of the doc Run Background tasks with WebJobs to know how to get start.
Please see the note of the section Create a continuously running WebJob https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/#CreateContinuous.
Note:
If your web app runs on more than one instance, a continuously running WebJob will run on all of your instances. On-demand and scheduled WebJobs run on a single instance selected for load balancing by Microsoft Azure.
For Continuous WebJobs to run reliably and on all instances, enable the Always On* configuration setting for the web app otherwise they can stop running when the SCM host site has been idle for too long.