I'm surprised I haven't been able to find something on this here - so if I've just completely missed it, please direct me to the proper thread.
Before I dive into any code, I'm trying to gather some good ideas for handling this situation.
We're developing a website with a list of tasks the user can select for the server to execute on their behalf. Automated emails, text messages, calendar reminders, etc.
I first went down the road of thinking about using cron, but as that the times and tasks for each user will likely change every day throughout each day - for this to be feasibly salable, I figured involving cron directly for each task could get pretty messy and buggy.
My next thought was to run a cron script every night at midnight and generate a task-list for the next day - but I'd still need cron or some sort of cron-like timing daemon to check the list against the time every minute.
I've run through several ideas, but they all seem fairly active or processor heavy. I'd like to find a good light-weight solution that can handle up to several thousand user defined tasks per day.
I'm working with your basic LAMP7 stack. If anybody has dealt with a similar task, I'm just looking for some good ideas to consider.
Thanks in advance.
You can use ReactPHP application run in background in your machine.
Then you can create a simple http server on your ReactPHP application for recieving the user data from your webserver such as you specified LAMP7. And once you recieved that you can trigger those events by setting asyncronous timer on the event-loop.
Related
Question
Using PHP & Jquery how would you execute code after a given amount of time, say 1 month (even after the user has closed the browser etc)
Scenario
I've wanted to build an application that does something in an amount of time specified by the user, "sort of like hootsuite". But i cant get my head around how it would work.
I know you can use node.js (I struggle to understand and implement this in any of my laravel projects...) but even then wouldnt the server be filled with stress if say 1000 people had something waiting to be executed on the server for a whole month or even a year while still handling other user requests?
I've looked around a bit and CRON jobs came up but this doesnt sound like what i was looking for! Im not sure, ill be grateful if anyone can explain to me how they think i could go about it
Essentially what you're looking for is a scheduling system. The reason why the UNIX cron tool has come up in your searches is because it is a scheduling tool; it allows UNIX users to schedule tasks to happen at certain times. Other operating systems also have task schedulers.
Schedulers
The principal implementation strategy for a scheduler is some kind of polling mechanism, i.e., a software component which periodically checks to see if there are any scheduled tasks which are now due to be executed and, if so, executes them.
Implementation strategies
In order to implement something like this you would need a way to store information about scheduled tasks (e.g. when they're supposed to happen, who they belong to, what they're supposed to do). For example, you might use a database management system, or a file on disk.
You would also need a component to do the polling. This could be a daemon process (i.e. a process which is always running in the background) which includes a sleep (or wait or timeout) call which allows it to check at intervals for scheduled tasks, rather than constantly checking (and thereby most likely consuming all the CPU cycles!). Or it could be a program (in PHP if you like) which is itself run by cron on the host system, say, every five minutes which checks for scheduled tasks and then executes in, perhaps in separate processes. If you were to use cron, there are numerous PHP wrappers to help such as https://packagist.org/packages/peppeocchi/php-cron-scheduler.
Services
However, instead of implementing all this yourself, you may consider making use of an existing service. There seem to be several options, including at least one free (within limits) service: https://atrigger.com/.
I am fairly certain this is an extremely easy thing to do for more of the experienced users, and probably shouldn't take more than a minute to do.
Unfortunately I have no experience with Cron Jobs, and despite reading documentation I can't seem to find any info on how to accomplish this.
I need to set up a Cron job to run every day at a certain time (This much I already know the syntax for), and what it needs to run is an "Export" link on my website that I use for product feeds.
The link is:
http://www.heroesofgc.com/index.php?route=feed/bidorbuystoreintegrator/export&t=73f6176e946fe85c5fb230b6040466d0
What command would I need to use to get it to work?
-I've read about using 'Wget', but I don't believe I have any such thing on my server at the moment (or don't know where to find its path at least).
Here shown, how you can set cronjob timing
http://www.adminschoice.com/crontab-quick-reference
http://kvz.io/blog/2007/07/29/schedule-tasks-on-linux-using-crontab
I'm trying to implement an iCal synchronization service for my project. There are nearly 10.000+ listings (more to come) in the database and almost every listing has a Google Calendar iCal URL to be synchronized every 12 hours. Synchronizing a single iCal URL takes about 0.5-1sec, so it will take serious time to process 10000+ items.
I'm not sure how to handle the synchronization process. I'm thinking to use Gearman but not sure if it's the best way. If I use Gearman, what would be the PROS and CONS of Gearman? How would I implement Gearman to handle iCal synchronization?
I also found BraincraftedBackgroundProcess written in PHP. I'm not sure if pure PHP can handle such a busy process. I might consider using it too but I'm still trying to figure out what the best way is.
I'm sure you've solved this by now, but Gearman would be possible solution to handle it. The beauty of Gearman is that you write rather simple workers that could have only one function, and that's how to perform the synchronization of one particular iCal URL.
You would then start as many workers as you'd be comfortable with making requests at the same time to Google's Calendar API, so you can easily scale back or up your performance by starting or killing of worker processes.
Updating calendars on regular intervals could be done by having a cron script fetch all calendars that hasn't been updated the last 12 hours, and then adding them to gearmand as tasks to be performed. You can also throttle the amount of requests if you do this every 15 minutes and have a hard limit of fetching only 250 calendars (or something like that) at a time, allowing updates to be bit more fluent that just every 12 hours.
If you need more specific advice on how to implement it using gearmand you'd have to have a more specific question and example code you're using, but there's nothing stopping you from using Gearman to do that task.
I'm putting together my first commercial PHP application, it's nothing really huge as I'm still eagerly learning PHP :)
Right now I'm still in the conceptual stage of planning my application but I run into one problem all the time, the application is supposed to be self-hosted by my customers, on their own servers and will include some very long running scripts, depending on how much data every customer enters in his application.
Now I think I have two options, either use cronjobs, like for example let one or multiple cronjobs run at a time that every customer can set himself, OR make the whole processing of data as daemons that run in the background...
My question is, since it's a self-hosted application (and every server is different)... is it even recommended to try to write php that starts background processes on a customers server, or is this more something that you can do reliably only on your own server...?
Or should I use cronjobs for these long running processes?
(depending on the amount of data my customers will enter in the application, a process could run 3+ hours)
Is that even a problem that can be solved, reliably, with PHP...? Excuse me if this should be a weird question, I'm really not experienced with PHP daemons and/or long running cronjobs created by php.
So to recap everything:
Commercial self-hosted application, including long running processes, cronjobs or daemons? And is either or maybe both also a reliable solution for a paid application that you can give to your customers with a clear conscience because you know it will work reliable on all kinds of different servers...?
EDIT*
PS: Sorry, I forgot to mention that the application targets only Linux servers, so everything like Debian, Ubuntu etc etc.
Short answer, no, don't go for background process if this will be a client hosted solution. If you go towards the ASP concept (Application Service Provider... not Active Server Pages ;)) then you can do some wacky stuff with background processes and external apps connecting to your sql servers and processing stuff for you.
What i suggest is to create a strong task management backbone and link that to a solid task processing infrastructure. I'll recommend you read an old post i did quite some time ago regarding background processes and a strategy i had adopted to fix long running processes:
Start & Stop PHP Script from Backend Administrative Webpage
Happy reading...
UPDATE
I realize that my old post is far from easy to understand so here goes:
You need 2 models: Job and JobQueue, 2 controller: JobProcessor, XYZProcessor
JobProcessor is called either by a user when a page triggers or using a cronjob as you wish. JobProcessor::process() is the key that starts the whole processing or continues it. It loads the JobQueues and asks the job queues if there is work to do. If there is work to do, it asks the jobqueue to start/continue it's job.
JobQueue Model: Used to queue several JOBS one behind each other and controls what job is currently current by keep some kind of ID and STATE about which job is running.
Job Model: Represents exactly what needs to be done, it contains for example the name of the controller that will process the data, the function to call to process the data and a serialized configuration property that describe what must be done.
XYZController: Is the one that contains the processing method. When the processing method is called, the controller must load everything it needs to memory and then process each individual unit of work as fast as possible.
Example:
Call of index.php
Index.php creates a jobprocessor controller
Index.php calls the jobprocessor's process()
JobProcessor::Process() loads all the queues and processes them
For each JobQueue::Process(), the job queue loads it's possible Jobs and detects if one is currently running or not. If none is running, it starts the next one by calling Job::Process();
Job::Process() creates the XYZController that will work the task at hand. For example, my old system had an InvoicingController and a MassmailingController that worked hand in hand.
Job::Process() calls XYZController::Prepare() so that it loads it's information to process. (For example, load a batch of emails to process, load a batch of invoices to create)
Job::Process() calls XYZController::RunWorkUnit() so that it processes a single unit of work (For example, create one invoice, send one email)
Job::Process() asks JobProcessingController::DoIStillHaveTimeToProcess() and if so, continues processing the next element.
Job::Process() runs out of time and calls XYZController::Cleanup() so that all resources are released
JobQueue::Process() ends and returns to JobController
JobController::Process() is about to end? Open a socket, call myself back so i can start another round of processing until i don't have anything to do anymore
Handle the request from the user that start in position #1.
Ultimately, you can instead open a socket each time and ask the processor to do something, or you can queue a CronJob to call your processor. This way your users won't get stuck waiting for the 3/4 work units to complete each time.
Its worth noting that, in addition to running daemons or cron jobs, you can kick off long running processes from a web request (but note that it must run outside of the webserver process group) and of course asynchronous message processing (which is essentially a variant on the batch approach).
All four of these approaches are very different in terms of how they behave, how concurrency and timing are managed. The factors which make them all different are the same ones you omitted from your question - so it's not really possible to answer.
Unfortunately all rely on facilities which are very different between MSWindows and POSIX systems - so although PHP will run on both, if you want to sell your app on both platforms it's going to need 2 versions.
Maybe you should talk to your potential customer base and ask them what they want?
I am creating a scheduler using PHP/MySQL where I have to allow the use to select date and time for publishing the content. The requirement is, the content should get posted at the exact scheduled time. If I create cron job, the notifications won't go out at exact time.
Running a cron job every minute is not really feasible in my case since I have to publish using an API and that itself is time consuming.
Is there any other way I can implement to make sure that the exact time provided in scheduler is followed. One of the best example that does this is Google Calendar which sends the reminders at the time you ask for.
Aditya
You should consider creating a daemon. The PEAR System_Daemon would be a great starting point. The daemon should essentially be a loop that queries the database, posts the content if necessary and sleeps.
Some sections of their documentation you might be interested in:
What is Daemon
Daemons vs Cronjobs
Installation
Example
YOU SHOULD CREATE CRONE JOB TO ACCOMPLISH THIS.