Hoping I can get some help (again). I'm working on a multi-tenant PHP application. Each tenant will have their own database (mysql). Ultimately, my plan is to stand the service up on AWS using ELB, EC2, and DynamoDB.
However, the app will need to run certain scheduled tasks (it needs to pull open invoices from a PSA for certain customers, and then charge the customer using Authroize.net CIM and mark it paid in the PSA).
For a regular application, I would simply create a cron script that runs daily to create/process the payment batches. I'm just not sure what the appropriate approach would be to run the cron across each tenant (for each database). Maybe one master cron job that runs across each tenant, or do I write a script to create / maintain cron jobs for each tenant using SWF?
Thanks for your input.
I've had reasonable success with doing batch processing via cron in the past. It might be helpful to record tenant creation in a table that you can query as a source for which databases to run against within the cron job.
Related
I am building a Multi-Tenant web application using Laravel/PHP that will be hosted on AWS as SaaS at the end. I have around 15-20 different background jobs that need scheduling for each tenant. The jobs need to be fired every 5 minutes as well. Thus the number of jobs which need to be fired for 100 tenants would be around 2000. I am left with 2 challenges in achieving this
Is there a cloud solution that distributes and manages the load of the scheduled jobs automatically?
If one is out there, how can we create those 15+ scheduled jobs on the fly? Is there an API available?
Looking for your assistance
Finally, I have found a solution to my problem.
We cannot scale the background jobs in the way I want. It required me to look into the solution from a completely different angle.
The ideal solution to my problem is that I should generate SQS messages (with a payload describing the tenant id, the job needs to be executed and any additional parameters) corresponding to the number of tenants on a set interval and queue it.
For example, if I have 100 tenants and I want to run "Job 1" every our, the main application will generate 100 SQS messages and queue it in a particular SQS Queue every hour. It will do the same for all 15 different jobs I have per tenant.
On the other end, a scalable AWS Lambda function listening to the SQS queue will pick up the payload and execute the intended task based on the data being carried by the payload.
But unfortunately, my expertise lies in PHP/Laravel technology which is still not in the AWS Lambda stack. Hence I figured out a workaround as follows.
I built a Docker image with my PHP/Laravel application and placed it in Amazon ECS (EC2 container service). Still, I have the AWS Lambda function in place but this time it acts as a trigger to my docker containers. The Lambda picks an SQS Message, processes the payload and spawns a Docker container on ECS based on my Docker image. I got some of the ideas from the following article to arrive at this solution.
https://aws.amazon.com/blogs/compute/better-together-amazon-ecs-and-aws-lambda/
Laravel has option to schedule Task/Jobs:
Refer: https://laravel.com/docs/6.x/scheduling
so you can keep jobs of your client in your database and than do it some like below:
Scheduling Queued Jobs
The job method may be used to schedule a queued job. This method provides a convenient way to schedule jobs without using the call method to manually create Closures to queue the job:
$schedule->job(new ClientJob)->everyFiveMinutes();
// Dispatch the job to the "clientjob" queue...
$schedule->job(new ClientJob, 'clientjob')->everyFiveMinutes();
or
Scheduling Shell Commands
The exec method may be used to issue a command to the operating system:
$schedule->exec('node /home/forge/script.js')->everyFiveMinutes();
my current script using CRON to handle the checking to DB and do the requests
so every mint the CRON will be called and check which action should be done as per the schedule table entry so if now the time to send email/publish post etc...
and this entry getting more and more with time and with many users now my CRON take around 20 to 50 mint to be done
so if I have to send email on 10 AM it sends between 10:20 AM to 10:50 AM
after searching I found RabbitMQ and Redis and other systems and I choose RabbitMQ
so what is next, what I need to do next, as for my experience I never work with a system like Redis etc.. so its something totally new, so if someone has and resources to check read or watch and help me with migrating the whole system from CRON to RabbitMQ.
small note, the current script is built on top of custom PHP framework only for this script and don't have API.
Write a php shell script for create linux pid in a infinite loop and call a method by cron.
Every job push to rabbitMq basic_publish with data set.
this method create a basic_consume with rabbitMq for performing queue with queue data set.
I'm writing a web app in PHP + Laravel + MySQL.
In the system, a user can schedule emails (and other API calls) at arbitrary times (much like how you schedule posts in WordPress). I can use CRON to inspect the database every 5min or so to find emails that should be sent, send them, and update their status.
However, this is a SaaS app. So the amount of emails to be sent at a particular time can grow rapidly. I can create a "lock file" every time the CRON script runs so that only one instance of it is running at a time. The lock file will be deleted after a script finishes execution.
But with potentially large data, I would want a way to process multiple messages simultaneously, potentially using multiple "workers." Is there any existing solution manage such a queue?
Yes! Task/Message/Job queues are what you are looking for! They allow you to put various tasks in queues from which you can retrieve them and process them, this process can scale horizontally as each worker can pull a task once its finished with the previous one.
You should have the cron maybe every minute/two minutes that just uploads the task and what needs to be done. This will make sure the cron is very quick.
Take a look at Iron.io Here is an extract from the website which gives a nice overview of these kinds of systems:
An easy-to-use scalable task queue that gives cloud developers a
simple way to offload front-end tasks, run scheduled jobs, and process
tasks in the background and at scale.
Gearman is also a great solution that you can use yourself and is very simple. You can send the message in many different languages and use a different langauge to process it. Say PHP -> C etc...
The Wikipedia link will tell you everything you need to know, here is a quick excerpt:
Message queues provide an asynchronous communications protocol,
meaning that the sender and receiver of the message do not need to
interact with the message queue at the same time. Messages placed onto
the queue are stored until the recipient retrieves them.
i need to update/ empty my databases on a specific date. i've made it manually to be done by the admin. but if due to some reason, admin fails, then database won't be updated.
for doing this automatically, i've read about cron jobs or windows task scheduling.
but my project is under development now and i can't upload it to any site so that i can ping the specific page from other sites. my project is in USB, and i need to take to work, home, to friends or to wherever i can have access to a system. so creating a cron job or task schedule will not help as OS/ system is not constant.
how can i create a portable task scheduling?
or is there any other process to run the specific script and re-run after intervals whenever the server starts running?
i'm using xampp 1.8.2 in a USB under windows.
I want to be able to do few things using PHP: to setup CRON to run specific file at a specific moment (easy) and at the same time to get some sort of ID of that job, which would allow me to track/or cancel the job before it was even started. I would keep track of those IDs in my Database.
Messing directly with system files likes cron via PHP isn't really a good idea.
I would set up a master cron job file first which set up to run at every 5 mins and later will control all of your tasks
Build a solid database backend where you can store your tasks (script, start time, frequency etc.)
Build a panel where you can manage your tasks, start them, hold them, etc.
Check with your web-host. Besides running cron from the command-line or a script, some web management panels (like CPanel) has a graphical interface for setting up cron jobs.