Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 3 years ago.
Improve this question
I'm currently building an API in Laravel (PHP) for retrieving prices from different hosts. There are currently 756 different 'coins' with different hosts to retrieve the prices from.
For example: Coin X
Host 1
Host 2
Host 3
-- up to 30 hosts
Coin Y
Host 1
Host 4
Host 5 -- up to 30 hosts
etc.
The problem here is, that ideally each coin should be updated every 10 seconds. This means that each coin needs to call all of its hosts, calculate the average price, save the price in the DB and finally save a JSON file with the total history of the coin. (Perhaps it would be better to also save the current price as JSON to save some time)
I've tried to putting all of this in a class for each host, but the execution time is way too long (around 5 minutes using CURL).
I'm thinking to create a task (cron job) for each coin. This way the 'updating' of the coins can go in sync (multiple coins at once). But i'm not quitte sure this would be the best way.
What approach would you guys recommend? All tips are welcome.
Your overall update time is going to be limited by number of coins x individual host update speed. You might want to time how long one (or avg) request takes (with script) to see just how long the whole thing will take. As, if it will likely take too long, doing them one after the other, you will have to set it up to do parallel (at same time) requests.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
We're using PHP 7 and have a MySQL DB running on a Webserver with only 128 MB RAM.
We have a problem with processing tons of datasets.
Simple description: We have 40.000 products and we want to collect data to these products to find out, if they need to be updated or not. The query which is collecting the specific data from another table with 10 Million datasets takes 1.2 seconds, because we have some SUM functions in it. We need to do the query for every product individually, because the time range which is relevant for the SUM, differs. Because of the mass of queries the function which should iterate over all the products returns a time out (after 5 min) - that's why we decided to implement a cronjob, which calls the function and the function continues with the product it ended the last time. We call the cronjob every 5 min.
But still, with our 40.000 products, it takes us ~30 hours until all the products were processed. Per cronjob, our function processes about 100 products...
How is it possible to deal with such a mass of data - is there a way to parallelize it with e.g. pthreads or does somebody have another idea? Could a server update be a solution?
Thanks a lot!
Nadine
Parallel processing will require resources as well, so on 128 MB it will not help.
Monitor your system to see where the bottleneck is. Most probably the memory since it is so low. Once you find the bottleneck resource, you will have to increase it. No amount of tuning and tinkering will solve an overloaded server issue.
If you can see that it is not a server resources issue (!), it could be at the query level (to many joints, need some indexes, ...). And your 5 min. timeout could be increased.
But start with the server.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I am creating a simple web application using PHP, Codeigniter and Google Books (on an WIndows 7 XAMPP localhost environment).
I have a MySQL list of books (a few hundred) and corresponding ISBN numbers. When a user views a book / visits a URL for the first time, and API call is made to Google Books and the title, author and description of the book is saved to my database.
Ideally i'd like to populate the database myself and not rely on the user. So, I was thinking of visiting each URL manually. However there a lot of items!
Is there a script I can use for such a task? I was hoping to run the script once every 5 minutes over a 24 hour period.
My URL is in the following format:
/items/itemView/1 // <-- book 1
/items/itemView/2 // <-- book 2
/items/itemView/3 // <-- book 3
// etc
// etc
Thanks
Short Answer:
A storage API exists so you don't have to catalogue everything.
Long Answer:
It sounds like what you are trying to do is take the API and scour through every single entry and record them for your own purposes.
While this can usually done fairly simply, instead of telling you how to do this, I'm going to tell you why you shouldn't.
An API to a huge database exists so that you don't have to store it all, as the resources required can be absolutely huge, usually more than most enthusiasts would even have.
It's better to have it as you do now, cache what is visited on the chance it is visited again and make sure periodically that any records you DO keep, you compare to it's source so that you don't have an out-of-date record (another pitfall of local caching).
I hope this helps at least show you why people tend not to duplicate large data sources.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I have a website like SO. I mean when user creates an account, my website creates an avatar for him (based on his IP and name). The image is created by PHP. The size of that image (in the expected width/height) is 10kb.
My question: Should I store the avatar of each user on the server? Or should I create it every time by PHP and no need to store it?
Actually I'm worry about the space. To store such images (for 2 million users) I have to devote 20 gb of the server's space:
2,000,000 * 10 kb = 20,000,000 kb = 20 gb
Well it doesn't seem reasonable to me. Am I right? If yes, so what should I do?
As a real answer, between using processor resources and data storage, you almost always choose data storage because :
Processing resources are generally more expensive than data storage,
Processing more means a slower server response time, which is generally a high priority to get your users (and your SEO) happy.
If you have 2 million users, 20 gig of server space should be the least of your worries. Calculate with realistic numbers first.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am wondering, when you want to make a php based games, that requires the player to wait for something, for example: I paid 100 gold to explore, and every 5 minutes I will receive loot. The exploration will ends in 30 minutes for example. I want to know, which is the best and why. Here are the options:
Keep record of starting time of the exploration command issued, then every time the one specific exploring player open the page, calculate everything and show the result then keep it in the database.
Make a cron job to calculate exploration of EVERY player currently exploring every 5 minutes and update it to database.
Make a cron job every 30 minutes to calculate and update everything for EVERY PLAYER, but also allow SPECIFIC PLAYER to update just like option 1.
option 3 is basically combination of option 1 and 2. Thanks for the help. I am not sure about the performance issue so I need to know from people who already had experience in this.
These are just some personal opinion, might not be the best choice.
2) is more of a general approach for multiplayer game that has player interaction, but it puts constant strain on the server, which seems to be over kill as I seriously doubt your game would have complex interaction between players.
1) is probably the way to go unless your calculation is very complex and take a long time. The possible drawback is that you'll probably have trouble handling lots of simultaneous request to update. But from what you describe I don't think that'll happen.
3)This is hard to comment on because I have no idea if your calculation-time would depends on how much time it has pass since last update. If you calculation is time-indepentdent, then it's a horrible method as you spend time to update data that no one might need AND you are open to traffic spike as well.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
A server has a page that calls 10 different PHP files which in total up take 10 ms (1% of CPU) to execute and 1MB of memory.
If the website begins to get lots of traffic and this individual page request that calls these 10 PHP files takes 10 ms (1% of CPU) happens to gain 90 hits per second does the CPU percent increase? Or balances at 1%? Also does the memory increase?
What would the load (CPU and memory) look like at 100 hits? 1,000 hits? 10,000 hits? and 100,000 hits?
Keeping with the above specifications.
Also, if there were another 10 different pages, calling 5 unique PHP files and 5 of the same PHP files from the above call? What happens to load at 100 hits, 1,000 hits, 10,000 hits and 100,000 hits per second? Does it partially increase? Balance?
There isn't much information on heavy loading behavior for PHP online, so I'm asking to get a better understanding, of course. Thanks! :o)
Your question has a difficult answer and I cannot tell you the accurate ratio of the increase of server's resources. But, keep these two things in mind:
More the number of users, more the use of resources. So, it doesn't matter that you are calling the same files, but the thing which matter is that you are calling it 90 times.
Your system's usage would increase definitely, but one thing would make it a little less. And, that is caching. Your CPU would load these files into its cache (when they would be accessed very much) and hence, it would make the process a bit faster.