I'm building a website in PHP and need an API to be checked on a regular basis for EACH USER individually.
In a nutshell it's a SaaS to steer a user account on another website with additional/automated options. A web based bot if you want.
So basically, I need to dynamically create a new cron job with its individual interval for each user, and these should be executed in parallel (it would take too long to put all queries into one cron job if theres a lot of users).
Also, it might become necessary that each request is done with a different IP. Reason is that it is possible that the API provider is annoyed by us and wants to block us. Since the API key is public they will most likely do it by simply blocking our IP. When we change that frequently, that should help a lot.
Is something like that possible? What would this require? Any option that doesn't get too expensive?
I thought of RabbitMQ for example, but that wouldn't quite tackle all issues and I'm wondering if there's some better/smarter solution.
Thanks!
Look at temporal.io open source project. It supports practically an unlimited number of such periodic jobs. The PHP SDK can be found here.
Related
I am currently working on a fairly large purely dynamic site that contains various information that needs to be expired. Previously i did not have to worry about expiration because it was handled on user login ( various checks would run to expire the logged in user data if needed ) but with our increase member base and inactivity of users the data within the db is getting old. Normally this would not be a problem but the old data affects the rest of the sites features/functionality ( point based system features implemented, team building features, etc. ) All data stored in the database has an expiration timer so all i have to do is soft-delete the data using a php script but i don't want to trigger this on page load ( i want to avoid slowing down the user page load )
What alternatives are available aside from cronjobs. I want to be able to setup and manage the background services through php so i don't have to edit/create crons every time i need something new added, etc.
Ideally i am looking for or trying to implement a system that will allow me to insert a db row with specific instructions ( queue a specific update ) and it will be handled on the backend. I want/need to have the data updated as soon as possible to avoid the issues we are running into now. This background processor will eventually handle larger more complex tasks like auto scheduling an on site event ( tournaments ), or auto generating brackets for these tournaments. All help is appreciated!
You could try the MySQL event scheduler, so you could initiate a SQL script which would re-run every x days:
CREATE EVENT `delete_old_data`
ON SCHEDULE schedule
ON COMPLETION NOT PRESERVE
ENABLE
DO BEGIN
-- event body, something like delete all rows older than 5 days
END;
To be honest to do more complex events like generating 'site events' it looks like you should be using a cronjob. Is there any reason you cannot use this? I am sure if you explain to your hosting provider that you need a cronjob and show them the code that you will be using it for they will enable this option on your account. (most hosts should have this enabled already)
There are several approaches, but none of them is perfect.
CRONJOBS
You've pointed that you don't want to use them. But if your main concern about cronjobs is crontab management, you may just setup one every-minute PHP cron which then may trigger different tasks with different resolutions. You can fully manage background services via PHP that way. The main disadvantage is that you cannot perform your background tasks parallely.
GEARMAN WORKERS
You may also use 3rd party software, like Gearman. Gearman allow you to write so called workers (your background services), in any language you like (PHP is supported, as well as c++, java and many others). These workers behave like ansychronous functions which may be called from anywhere of your code using gearman api. You don't have to carry about the result, simply call the function and forget. It will be done in background. Also task scheduling is a built-in gearman feature.
Another software you may use is Rabbit MQ or any other message bus system.
NATIVE PHP ASYNC FUNCTIONS
New PHP7 will bring native php asynchronous programming. You will be able to separate your PHP script and http request handling, a bit like in node.js. Your script will be able to work all the time, doing some background stuff in any way you like and handling http requests like events in another process. This is probably the best option for you, if you may wait till the release date.
My answer covers only solutions I was using personally. Possibly there are also another ways to reach your goal, so keep searching!
I am a programmer at an internet marketing company that primaraly makes tools. These tools have certian requirements:
They run in a browser and must work in all of them.
The user either uploads something (.csv) to process or they provide a URL and API calls are made to retrieve information about it.
They are moving around THOUSANDS of lines of data (think large databases). These tools literally run for hours, usually over night.
The user must be able to watch live as their information is processed and is presented to them.
Currently we are writing in PHP, MySQL and Ajax.
My question is how do I process LARGE quantities of data and provide a user experience as the tool is running. Currently I use a custom queue system that sends ajax calls and inserts rows into tables or data into divs.
This method is a huge pain in the ass and couldnt possibly be the correct method. Should I be using a templating system or is there a better way to refresh chunks of the page with A LOT of data. And I really mean a lot of data because we come close to maxing out PHP memory and is something we are always on the look for.
Also I would love to make it so these tools could run on the server by themselves. I mean upload a .csv and close the browser window and then have an email sent to the user when the tool is done.
Does anyone have any methods (programming standards) for me that are better than using .ajax calls? Thank you.
I wanted to update with some notes incase anyone has the same question. I am looking into the following to see which is the best solution:
SlickGrid / DataTables
GearMan
Web Socket
Ratchet
Node.js
These are in no particular order and the one I choose will be based on what works for my issue and what can be used by the rest of my department. I will update when I pick the golden framework.
First of all, you cannot handle big data via Ajax. To make users able to watch the processes live you can do this using web sockets. As you are experienced in PHP, I can suggest you Ratchet which is quite new.
On the other hand, to make calculations and store big data I would use NoSQL instead of MySQL
Since you're kind of pinched for time already, migrating to Node.js may not be time sensitive. It'll also help with the question of notifying users of when the results are ready as it can do browser notification push without polling. As it makes use of Javascript you might find some of your client-side code is reusable.
I think you can run what you need in the background with some kind of Queue manager. I use something similar with CakePHP and it lets me run time intensive processes in the background asynchronously, so the browser does not need to be open.
Another plus side for this is that it's scalable, as it's easy to increase the number of queue workers running.
Basically with PHP, you just need a cron job that runs every once in a while that starts a worker that checks a Queue database for pending tasks. If none are found it keeps running in a loop until one shows up.
I was wondering how I could check if someone has left the site/page and perform an action after they left. I was reading on here and I found this:
No, there isn't. The best you can do is send an AJAX request every X seconds (perhaps only if the user moves the mouse). If the server doesn't receive any requests for 2X seconds, assume that the user left.
That's what I had planned for before but how could you make the server do something (in my case it's to remove them from the DB) if they stop sending the request? An example I can think of is how on Facebook when you go to the site you tell them you're here and online which marks you as online in chat but when you leave it marks you as offline. How is that possible?
Edit: After a while of using cron jobs I found out that web hosting sites don't like running cron jobs often enough to generate a "live" feelings on your site. So instead I found node.js and it works a 1000x better and is much simpler. I'd recommend anyone with the same issue to use it, it's very cheap to buy hosting for and it's simple to learn, if you know Javascript you can build in it no problem. Just be sure to know how Async works.
A not uncommon approach is to run a cron job periodically that checks the list of users, and does XYZ if they have been inactive for X minutes.
Facebook uses the XMPP protocol via the Jabber service to have a constant or real-time connection with the user. However, implementing one isn't an easy task at all. The most simple solution would be, as mentioned in the comments, to have the client make AJAX requests to the server every several seconds, so that the server may check whether the user is still viewing the site or not.
You might want to check out my question, which might be related to yours.
The only method I ever remember in my time developing is one that's not 100% reliable as a number of factors can actually and most likely cause it to either misfired, or not even run fully. Up to and including someone disabling JavaScript. Which grant it isn't highly likely with the way websites of today are put together. But people have the option to turn it off, and then people who are acting maliciously tend to have it off as well.
Anyway, the method I have read about but never put much stock in is, the onunload() event. That you tie into something like the <body> tag.
Example:
<body onunload="myFunction()">
Where myFunction() is a bit of JavaScript to do whatever it is your seeking to have done.
Again not superbly reliable for many reasons, but I think in all it's the best you have with almost all client side languages.
For a homework project, I'm creating a PHP driven website which main function is aggregating news about various university courses.
The main problem is this: (almost) each course has it's own website. These are usually just plain HTML or built using some simple free CMS system.
As a student, participating in 6-7 courses, almost every day you go through 6-7 websites checking if there are any news. The idea behind the project is that you don't have to do that, instead, you just check the aggregation site.
My idea is the following: each time a student logs in, go through his course list. For every course, get it's website (recursively, like with wget), and create a hash value of it. If the hash is different then one stored in database, we know that site has changed, and we notify the student.
So, what do you think, is this reasonable way to achieve the functionality?
And if yes, what is (technically) the best way to go about this? I was checking php_curl, put I don't know if it can get a website recursively.
Furthermore, there's a slight problem I have somewhat limited resources, only a few MB of quota on public (university) server. However, if that's a big problem, I could use a seperate hosting solution.
Thanks :)
Just use file_get_contents, or cURL if you absolutely have to (in case you need COOKIES).
You can use your hashing trick to check for modifications but it's not very elegant. What you want to know is when was it last changed. I doubt this information is on the website, but maybe they offer an RSS feed or some webservice or API you can use for this purpose.
Don't worry about doing recursive requests. Just make a new request each time.
"When all else fails, build a scraper"
Good time of the day, dear developers!
I am not any kind of network programming pro, but it happened that I have faced necessity to develop socket-server on php (no way for using Java) for flash multiplayer browser-game (standard features like locations, team battles, etc).
The main problem is that TCP is point-to-point protocol and it completely occupies given port.
Of course it is possible to create some kind of queue, which will manage connections to the socket, but this solution doesn't seems to be the fittest one.
It seems to me that using interval of "fair" ports (from 2000 to 2200, for example) is more fitting solution, because one request may take a lot of time to execute and players won't be happy to wait in queue.
But how can I implement this "port inteval" strategy?
The solutions that I see are:
launcing php-script per every port (he-he, 2 hundreds of launched scripts!);
somehow forking the initial process to new processes (2 hundreds of processed? Not nice too), one for every port;
additional while-loop, which listens all the ports (looks very bad);
using threading or something like that (the problem is that php is single-threaded, as far as I know; pcntl?).
But somehow I don't like any of them, or at least don't know how to implement them in the best possible way.
What is the best existing strategy to handle multiple requests from multiple users per time unit without delays, and how to implement this strategy in php? We have our own Debian-server, so it is possible to use any required php extensions.
Any advice about development, planning and implementation of such kind of systems is highlhy appreciated.
Thank You!
Edit 1:
By the way, I've forgotten to mention some extra details.
For example, if we are trying to develop chat application, we need some sort of fixed (I mean persistent) connections for each user. For example, we have 80 users in chat, and then one of them posted a message, which server tries to handle and send to all other connected users, also putting an entry to the history file or something like that.
In this situation polling server for new messages every 10 seconds from each of 80 users is craziness, so the need in persistent connection gets obvious.
But I don't know what is the best way to implement something like this, considering that not all requests are handled instanlty.
have you looked at http://www.smartfoxserver.com/ or http://www.electro-server.com/ ?
Best way for multi user communication for Flash is RTMP. Look for FMS or Red5. It's not in php, but I think this is correct way.