I am using an API to fetch certain data but due to its large amount, I can not fetch it each time the user accesses the website. Instead I thought of fetching it once, storing in the database and query whatever the user needs and display it. The point is I might need to send a request to API some other time in order to fetch newly added data and for this issue I thought either scheduling a certain time where the system fetches the data again from the API or manually send a request to obtain the new data.
To clarify my point, here's how I imagine the sequence:
Request for the first time to API to obtain data
Save the data in my DB and query what the user needs, thus
minimizing the time needed to fetch the data and also the number of
request sent to the server
Set automatically/manually a certain time schedule where I would resend
another request to the API to find if new data has been added
My question: Is my approach doable and efficient or could all of that be done in much more easier way?
UPDATE:
I would be using gAdventures API to list different tours along with their details.
Max number of tours returned per request is 100.
To fetch tour details, tour id returned should have a request sent to fetch all of the details concerning it, so we are talking about thousands of trips. Exact size of data is unavailable till now.
I recommend using my Swiss Cache library. It supports file based caching and APC. Even image cache (WIP). If the amount of data is big, my library supports Gzip compression enabled by default. So large amounts of data is automatically compressed.
Example Code of my library:
//Your API request function...
require_once("swiss_cache.php");
$cache = new swiss_cache();
$data = $cache->get("gAdventures_response");
if(!$data)
{
$data = someFunctionThatCallsTheService();
$cache->set("gAdventures_response", $data, 3600); //Save data for one hour.
}
Just make sure you set the cache path in the swiss_cache.php! :) If you have any issues or questions with my library. Just send me a message! The documentation is being worked on.
https://github.com/ajm113/swiss_cache
Related
I have a data provider(REST Api) that stores info about 400-500k items that gets updated daily. The API methods I can call returns info for 1000 items only (but I have a pagination mechanism so i can loop through all data) .
I'm working with PHP/MySQL and my task is to check a website database(containing 10k to 100k items) against the data provided by this API. All I need to do is to check that the item ID from the website database is present in the provider database. If not, I will delete the record from the website database.
What will be the best method to do this daily ?
Should I first do a loop, get all the data from the data provider and store that into a file ? (considering it is 400-500k ids I don't think an array will do ) Then check each ID from the local database against that file ?
I would refer to the "Rules Of Optimization Club" - specifically rules 1 and 2:
You do not optimize.
You do not optimize, without measuring first.
So build a solution that works with what you think of first. Then measure how it performs. If it performs badly, see what parts of it is slow (server responses / saving data / looping through data) and only then start to think about optimization.
This is specifically in response to "considering it is 400-500k ids I don't think an array will do" -- did you try and did it fail?
I've recently implemented Redis into one of my Laravel projects. It's currently more of an technical exercise as opposed to production as I want to see what it's capable of.
What I've done is created a list of payment transactions. What I'm pushing to the list is the payload which I receive from a webhook every time a transaction is processed. The payload is essentially an object containing all the information to do with that particular transaction.
I've created a VueJS frontend that then displays all the data in a table and has pagination so it's show 10 rows at a time.
Initially this was working super quick but now that the list contains 30,000 rows which is about 11MB worth of data, the request is taking about 11seconds.
I think the issue here is that I'm using a list and am fetching all the rows from the list using LRANGE.
The reason I used a list was because it has the LPUSH command so that latest transactions go to the start of the list.
I decided to do a test where I got all the data from the list and outputted the value to a blank page and this took about the same time so it's not an issue with Vue, Axios, etc.
Firslty, is this read speed normal? I've always heard that Redis is blazing fast.
Secondly, is there a better way to increase read performance when using Redis?
Thirdly, am I using the wrong data type?
In time I need to be able to store 1m rows of data.
As I realized you get all 30,000 rows in any transaction update and then paginate it in frontend. In my opinion, the true strategy is getting lighter data packs in each request.
For example, use Laravel pagination in response to your request.
In my opinion:
Firstly: As you know, Redis is blazing fast and Redis is really fast. Because Redis data always in memory, you say read 11MB data about use 11s, you can check your bandwidth
Secondly: I'm sorry I don't know how to increase in this env.
Thirdly: I think your choice ok.
So, you can check your bandwidth first(redis server).
I'm developing an Android app for salesman, so they can use their device to save their order. Specifically, every morning the salesman would go to the office and fetch the data for that day.
Currently, I can get the data by sending a request to php file, and like common practice we insert those data into sqlite in the Android so it can work offline. However, with current approach the device needs 6-8 seconds on getting the data and inserting those data to sqlite. As the data grow bigger I think it would make it slower. What I had found is that the process of inserting data into sqlite takes quite amount of time.
So, I've been thinking about dumping all data that is needed by the salesman into a sqlite file, so I could send only that file which I guess is more efficient. Can you please lead me on how to do that? Or is there any other way which is more efficient approach for this issue?
Note:
Server DB: Mysql
Server: PHP
You can do here different approach to achieve loading speed:
If your data can be pre-loaded with apk, you can just store those inside .apk and when user download app, it will be there, you just need to call remaining updated data.
If you need refreshed data every time, you can do call chunk of data from server in multiple calls, which will fetch and store data in database and update on UI.
If data is not too much, (I say, if there are 200-300 data, we should not consider it much more) you can do simple thing:
When you call network call for fetching data, you should pass that data objects to database for storing and at same time (before storing in db), just return the entire list object to Activity/Fragment, so it will have the data and you can see those data to user, in mean time, it will store in database.
Also, you can use no-sql in side sqlite, so you don't need to parse objects every time (which is costly compare to no-sql) and store all data in sql as entire object and whenever require, just fetch from db and parse it as per requirement.
Thanks to #Skynet for mentioning transaction, it does improve the process alot.. So I'll stay with this approach for now..
You can do something like so:
db.beginTransaction();
try {
saveCustomer();
db.setTransactionSuccessful();
} catch {
//Error in between database transaction
} finally {
db.endTransaction();
}
For more explanation: Android Database Transaction..
I am working on this project that I cant seem to get right.
Basically the application makes a request to my PHP server and displays the data on a listView.
Due to the large amount of data I have on my server, and the length of time it takes to get all the data on my listview. I decided to implement an OnScrollListener on the android side, to determine if the last item is visible, then I load more data. This I achieved by selecting all the IDs of the data I want to load when the the initial request is made. The IDs are then sorted based on my requirements (time created, points, etc) after which the first five ids are used to select the initial data which is returned to the android app along with the IDs. Then when the last item is visible, i send the next five ids from the list to a function on php which returns data corresponding to the five IDs.
So far this approach works but it is still unsatisfactory due to the amount large amount of data that needs to be processed during the initial request.
I need help with an alternative technique to achieve my objective with minimal delays while performing the initial request or subsequent request.
Any help will be much appreciated.
From what I read in your question, you are loading all the data at the initial request?
I suggest you to did pagination in your server side so you can minimalize the number of data, and call the next portion/page of data only when you need to do it (in this case you can implement it in OnScrollListener)
For more details about pagination
- http://www.phpfreaks.com/tutorial/basic-pagination
- http://www.phpeasystep.com/phptu/29.html
- http://code.tutsplus.com/tutorials/how-to-paginate-data-with-php--net-2928
I'm attempting a couple different data pulls using the BigCommerce PHP API.
In one attempt, I need to pull all of my customers and their addresses. In the other, I need to pull all of my orders and the coupon data (if any) associated with them.
The problem I'm having is a combination of the way BigCommerce returns the data, and with the amount of data I am attempting to pull.
When I pull a list of customers the address data is not stored with the results. Instead I have to query a separate JSON file. Example:
https://STORE-ID.mybigcommerce.com/api/v2/customers/2104/addresses.json
According to the quickstarts and the response from their API team, they expect me to simply iterate through each customer/order ID and then make an additional request to pull the address/coupon data for each ID.
Due to the amount of data I have here, this operation results in either script timeouts (30+ seconds), or in PHP running out of memory.
Yes, I know the general solution with PHP is throw more hardware at it, but there has to be a more efficient way to do this than to simply make a ton of single-shot long requests, right?
I'm thinking something in the way of multiple threads or jobs, though I personally am not aware of any such functionality.
As suggested by Chirag B, I ended up using Node.JS and splitting this into multiple async calls.