I'm using PHP to work with an API that limits me to 1 request every 10 seconds. The most results it can return in one go is 10.
That means to get 1000 results I need to make 100 calls which if done one after the other would take about 17mins so doing it 'on the fly' is not really an option.
If I need to get 1000 rows from the API, which would be the best way to go about it?
Is there any way I could get the API data 'in the background' so that when I need the info it's already in my database? The API is only updated every 4 weeks so it would only need to synchronise once in that time period.
I've though about using a cron job to do this but i'm not sure how it would work considering how long the script would have to run for.
Using a cron job is a great way to store values from an API in a database. You need to use updated time fields for both the API and the local database. Check if the date of the last record update of the API is greater than the date of the last record update in your local database. If so, fetch the new data.
If you want to do this same process for the local database to the server, you can. This is a common operation to sync project flow and reduce API requests.
Related
I am creating a project management system and in need to do push notifications when an activity took place.
Question : If I do a jquery to refresh and fetch notification from mysql database, say every 30seconds, will there be a huge impact in the server? What are the minimum requirements?
So basically, I'm looking at 10 notifications/day for 20 employees.
Assuming you're talking about an AJAX request to the server in order to update DOM elements, most basic web servers would very much be able to handle a few requests every 30 seconds or so. More important is how well-optimized the server-side code that finds & returns the notifications is. Assuming you'll have a few clients requesting every 30 seconds, I would suggest making sure the code only takes a few seconds to process the request and respond with the updated data.
Info about the website I'm working:
Website related to soccer live score
API provides real-time data for soccer score (they got no webhooks)
What I want:
Want to deliver the real-time score in frontend
Also, I want to save that data in Redis temporarily and after the match finishes, want to push from Redis to the database.
Preferably don't use any external JS libraries ( http://socket.io ), pusher, etc. Laravel Broadcasting+ Redis is my preferred way since I won't need pusher or socket js code to load.
Parts of this problem:
Part 1: Pulling external API data to database(or Redis).
--> So far, the only way I've managed to pull data from the API to the database is, I've created a route which will trigger the load data from external API. Again, this is so useless as of now, because live score data in API is updated almost every second, and so far I need to trigger the route(or refresh the URL every second) just to pull up data from API. Also, not to forget, it will take 2-3 minimum second just to completely transfer API data to the database. This section is not dependant on whether to pull only if the user(frontend is requesting). It should do its job even if there are 0 users online.
So, my question is what is the best, most efficient and complete way to pull API data real-time and save it in Redis until the match is finished? (we can know the status of the match by checking in API data example: {match_id{status: finished}}xxxx). Because after the match is finished, I will push Redis to the database.
Part 2: Publishing that data real-time from the database(or Redis).
-> Okay this one for me is fairly easier than part 1, I've already found ways to publish Redis data real-time via pusher and socket.io. But other than that, what is the best way to do in my scenario? Also, do I need any JS libraries if I have to use a combination of Redis+ Laravel broadcasting?
Thank you for your suggestion!
Possible answer for part 1:
I would use Task Scheduling to ->daily(); or ->hourly(); an Artisan Console Command to check when the next soccer match is and write a record in the matches table, or update existing records in case the start/end time changes.
Another Console Command on ->cron('* * * * *'); (change to desired seconds) that executes every few seconds can check the matches table; if the current time is between the starts_at and ends_at of a match, retrieve realtime data.
To prevent multiple executions of the command (if for some reason an API call takes a bit longer) at the same time, Symfony's Lockable Trait might be used.
User keys in search parameters, then we make a request to a data provider and redirect user to a loading page. The response from the data provider hits a callback url, in which case we parse the results and store about 200 rows into the db. Meanwhile the loading page uses ajax to query the db every second and when the results are all there we display the results to the user.
The issue is that insert into the mysql db is too slow. We know the response back from the data provider comes back within seconds, but the processing of the script and inserting of rows into the db is very slow. We do use multirow insert.
Any suggestions to improve? FYI, the code is hugely long... that's why not displaying right now.
There are multitude of factors affecting your insertions:
1) slow hardware and bad server speeds.
Sol : Contact your server administrator
2) Use something other than InnoDB
3) Use a surrogate key , other than your primary key that is numeric and sequential along with your natural primary key.
OR
4) Try this https://stackoverflow.com/a/2223062/3391466.
Suggestion: Instead of running the code on one page and having the user wait the whole process, why not have the php page store the instructions in a php queue? The instructions would then be executed by a separate php script (for instance a Cron Job) and the user wouldn't have to wait for the whole process to take place.
However, in this situation it would be ideal to let the user know that the changes made can take a bit of time to update.
Cron jobs are very easy to implement. In CPanel there is an option for Cron Jobs where you specify which script you want to run and in which intervals. You can let your script know to run once every 1 minute (or more or less depending on how much demand there is). From there your script would check the queue and could keep on running until the queue is empty again.
Let me know if that helped!
I'm trying to build a service that will collect some data form web at certain intervals, then parse those data, finally upon result of parse - execute dedicated procedures. Typical schematic of service run:
Request item list to be updated to
Download data of listed items
Check what's not updated yet
Update database
Filter data that contains updates (get only highest priority updates)
Perform some procedures to parse updates
Filter data that contains updates (get only medium priority updates)
Perform some procedures to parse ...
...
...
Everything would be simple if there ware not so many data to be updated.
There is so many data to be updated that at every step from 1 to 8 (maybe besides 1) scripts will fail due to restriction of 60 sec max execution time. Even if there was an option to increase it this would not be optimal as the primary goal of the project is to deliver highest priority data as first. Unlucky defining priority level of an information is based on getting majority of all data and doing lot of comparisons between already stored data and incoming (update) data.
I could resign from the service speed to get at least high priority updates in exchange and wait longer time for all the other.
I thought about writing some parent script (manager) to control every step (1-8) of service, maybe by executing other scripts?
Manager should be able to resume unfinished step (script) to get it completed. It is possible to write every step in that way that it will do some small portion of code and after finishing it mark this small portion of work as done in i.e. SQL DB. after manager's resuming, step (script) will continue form the point it was terminated by server due to exceeding max exec. time.
Known platform restrictions:
remote server, unchangeable max execution time, usually limit to parse one script at the same time, lack of the access to many apache features, and all the other restrictions typical to remote servers
Requirements:
Some kind of manager is mandatory as besides calling particular scripts this parent process must write some notes about scripts that ware activated.
Manager can be called by crul, one minute interval is enough. Unlucky, making for curl a list of calls to every step of service is not an option here.
I also considered getting new remote host for every step of service and control them by another remote host that could call them and ask for doing their job by using ie SOAP but this scenario is at the end of my list of wished solutions because it does not solve problem of max execution time and brings lot of data exchange over global net witch is the slowest way to work on data.
Any thoughts about how to implement solution?
I don't see how steps 2 and 3 by themself can execute over 60 seconds. If you use curl_multi_exec for step 2, it will run in seconds. If you are getting your script over 60 seconds at step 3, you would get "memory limit exceeded" instead and a lot earlier.
All that leads me to a conclusion, that the script is very unoptimized. And the solution would be to:
break the task into (a) what to update and save that in database (say flag 1 for what to update, 0 for what not to); (b) cycle through rows that needs update and update them, setting flag to 0. At ~50 seconds just shut down (assuming that script is run every few minutes, that will work).
get a second server and set it up with a proper execution time to run your script for hours. Since it will have access to your first database (and not via http calls), it won't be a major traffic increase.
Our company deals with sales. We receive orders and our PHP application allows our CSRs to process these orders.
There is a record in the database that is constantly changing depending on which order is currently being processed by a specific CSR - there is one of these fields for every CSR.
Currently, a completely separate page polls the database every second using an xmlhhtp request and receives the response. If the response is not blank (only when the value has changed on the database) it performs an action.
As you can imagine, this amounts to one databse query per second as well as a http request every second.
My question is, is there a better way to do this? Possibly a listener using sockets? Something that would ping my script when a change has been performed without forcing me to poll the database and/or send an http request.
Thanks in advance
First off, 1 query/second, and 1 request/second really isn't much. Especially since this number wont change as you get more CSRs or sales. If you were executing 1 query/order/second or something you might have to worry, but as it stands, if it works well I probably wouldn't change it. It may be worth running some metrics on the query to ensure that it runs quickly, selecting on an indexed column and the like. Most databases offer a way to check how a query is executing, like the EXPLAIN syntax in MySQL.
That said, there are a few options.
Use database triggers to either perform the required updates when an edit is made, or to call an external script. Some reference materials for MySQL: http://dev.mysql.com/doc/refman/5.0/en/create-trigger.html
Have whatever software the CSRs are using call a second script directly when making an update.
Reduce polling frequency.
You could use an asynchronous architecture based on a message queue. When a CSR starts to handle an order, and the record in the database is changed, a message is added to the queue. Your script can either block on requests for the latest queue item or you could implement a queue that will automatically notify your script on the addition of messages.
Unless you have millions of these events happening simultaneously, this kind of setup will cause the action to be executed within milliseconds of the event occuring, and you won't be constantly making useless polling requests to your database.