I am making a program that would pose a question and users would be able to answer it. This is a Internet of things project and as such there will be a lot of calls to the database, which would only want to increase the value of answer A or B by one. For the record I am using PHP and Mysql and there are many unique items that would send update requests to the server.
What can I do to reduce the calls to the database?
The solution I came up with was to store somehow the data on the server, then sync the data with the database on a scheduled interval.
To update the results I would need to know only 3 things - item id, and both results. For code clarity and simplicity I made a model object with those attributes.
So far I came up with/found several ideas:
Sessions - make session an array and just put the model objects inside it
Create a file on the server that would store the data
Use superglobal variables
Create a PHP class that would have an array, in which the objects would go to and interact with the class
Use some API - but I would be completely dependent on it working
Which one of the solutions given be the best in terms of simplicity, security and performance or is there a better way to do this whole thing?
You can have one database for row data and once you authorize then you can sync the data into actual database which your application needs. After sync, you can run a job of deleting unwanted data from the row database. Let me know if it works for you.
Related
My stack is php and mysql.
I am trying to design a page to display details of a mutual fund.
Data for a single fund is distributed over 15-20 different tables.
Currently, my front-end is a brute-force php page that queries/joins these tables using 8 different queries for a single scheme. It's messy and poor performing.
I am considering alternatives. Good thing is that the data changes only once a day, so I can do some preprocessing.
An option that I am considering is to create run these queries for every fund (about 2000 funds) and create a complex json object for each of them, store it in mysql indexed for the fund code, retrieve the json at run time and show the data. I am thinking of using the simple json_object() mysql function to create the json, and json_decode in php to get the values for display. Is this a good approach?
I was tempted to store them in a separate MongoDB store - would that be an overkill for this?
Any other suggestion?
Thanks much!
To meet your objective of quick pageviews, your overnight-run approach is very good. You could generate JSON objects with your distilled data, or even prerendered HTML pages, and store them.
You can certainly store JSON objects in MySQL columns. If you don't need the database server to search the objects, simply use TEXT (or LONGTEXT) data types to store them.
To my way of thinking, adding a new type of server (mongodb) to your operations to store a few thousand JSON objects does not seem worth the the trouble. If you find it necessary to search the contents of your JSON objects, another type of server might be useful, however.
Other things to consider:
Optimize your SQL queries. Read up: https://use-the-index-luke.com and other sources of good info. Consider your queries one-by-one starting with the slowest one. Use the EXPLAIN or even the EXPLAIN ANALYZE command to get your MySQL server to tell you how it plans each query. And judiciously add indexes. Using the query-optimization tag here on StackOverflow, you can get help. Many queries can be optimized by adding indexes to MySQL without changing anything in your php code or your data. So this can be an ongoing project rather than a big new software release.
Consider measuring your query times. You can do this with MySQL's slow query log. The point of this is to identify your "dirty dozen" slowest queries in a particular time period. Then, see step one.
Make your pages fill up progressively, to keep your users busy reading while you get the data they need. Put the toplevel stuff (fund name, etc) in server-side HTML so search engines can see it. Use some sort of front-end tech (React, maybe, or Datatables that fetch data via AJAX) to render your pages client-side, and provide REST endpoints on your server to get the data, in JSON format, for each data block in the page.
In your overnight run create a sitemap file along with your JSON data rows. That lets you control exactly how you want search engines to present your data.
So, I have situation and I need second opinion. I have database and it' s working great with all foreign keys, indexes and stuff, but, when I reach certain amount of visitors, around 700-800 co-current visitors, my server hits bottle neck and displays "Service temporarily unavailable." So, I had and idea, what if I pull data from JSON instead of database. I mean, I would still update database, but on each update I would regenerate JSON file and pull data from it to show on my homepage. That way I would not press my CPU to hard and I would be able to make some kind of cache on user-end.
What you are describing is caching.
Yes, it's a common optimization to avoid over-burdening your database with query load.
The idea is you store a copy of data you had fetched from the database, and you hold it in some form that is quick to access on the application end. You could store it in RAM, or in a JSON file. Some people operate a Memcached or Redis in-memory database as a shared resource, so your app can run many processes or threads that access the same copy of data in RAM.
It's typical that your app reads some given data many times for every single time it updates the data. The greater this ratio of reads to writes, the better the savings in terms of lightening the load on your database.
It can be tricky, however, to keep the data in cache in sync with the most recent changes in the database. In other words, how do all the cache copies know when they should re-fetch the data from the database?
There's an old joke about this:
There are only two hard things in Computer Science: cache invalidation and naming things.
— Phil Karlton
So after another few days of exploring and trying to get the right answer this is what I have done. I decided to create another table, instead of JSON, and put all data, that was suposed to go in JSON file, in the table.
WHY?
Number one reason is MySQL has ability to lock tables while they're being updated, JSON has not.
Number two is that I will downgrade from few dozens of queries to just one, simplest, query: SELECT * FROM table.
Number three is that I have better control over content this way.
Number four, while I was searching for answer I found out that some people had issues with JSON availability if a lot of co-current connections were making request for same JSON, I would never have a problem with availability.
For large arrays, is it better to save the data to global variables or query the database each time I need them? In my situation keeping them local scope and passing them to functions isn't an option.
I'm using wordpress and in most pages I get every user and all metadata attached to them. Often times I use these variables in multiple places on the same page. Unfortunately wordpress won't let me pass variables between templates so I'm stuck either using global variables or calling the database each time. Eventually, this will be hundreds of users with a lot of metadata attached to each. Should I call the database each time to keep the variables local, or should save them to global variables to save on database queries? What are the considerations? Should I worry about performance, overhead, and/or other issues?
Thanks so much!
The only real solution to your problem is using some kind of cache system (Memcache and Redis are your best options). Fortunately, there are plenty of Wordpress plugins that make the integration an easy thing. For instance:
Redis: https://wordpress.org/plugins/redis-object-cache/
Memcache: https://wordpress.org/plugins/memcached/
EDIT
If you only want to cache a few databases calls, you can forget about Wordpress plugins and start coding a bit. Let's say you only want to cache the call for retrieving the list of users from database, and let's assume you are using Memcache to accomplish this task (Memcache stores key-value pairs and allows super fast access to a value given a key).
Query Memcache asking for the key "users".
Memcache still doesn't have such key, so you'll have a cache fail and after it, you'll query your database to retrieve the user list. Now serialize the database response (serialize and json_encode are two different ways to do this) and store the key "users" along this serialized value in your memcache.
Next time you query your memcache asking for "users", you'll get a hit. In this moment you just have to unserialize the value and work with your user list.
And that's all. Now you just have to decide what you want to cache and apply this procedure to those elements.
You shouldn't have to perform the calls but once per page, you might have to execute the call once for every page. So I would suggest you creating some sort of class to interact with your database that you can call on to get the data that you need. I would also recommend using stored procedures and functions on your database instead of straight queries since this will help both with security and separation of application logic and data functionality.
I have a ajax based PHP app (without any frameworks etc.).
I need to retrieve some records from the database (html select element items) ONCE, and once only, during application startup, store it in a PHP array, and have this array available for future use to prevent future database calls, for ALL future users.
I could do this easily in Spring with initializing beans. And this bean would have the application scope (context) so that it could be used for ALL future user threads needing the data. That means the database retrieval would be once, only during app boot, and then some bean would hold the dropdown data permanently.
I can't understand how to replicate the usecase in PHP.
There's no "application" bootstrapping as such, not until the first user actually does something to invoke my php files.
Moreover, there is no application context - records retrieved for the first user will not be available to another user.
How do I solve this problem? (Note: I don't want to use any library like memcache or whatever.)
If you truly need to get the data only the first time the app is loaded by any user, than you could write something that gets the data from your database, and then rewrites the html page that you're wanting those values in. That way when the next user comes along, they are viewing a static page that has been written by a program.
I'm not so sure that 1 call to the database everytime a user hits your app is going to kill you though. Maybe you've got a good reason, but avoiding the database all but 1 time seems rediculous IMO.
If you need to hit the database one time per visitor, you could use $_SESSION. At the beginning of your script(s) you would start up a session and check to see if there are values in it from the database. If not, it's the user's first visit and you need to query the database. Store the database values in the $_SESSION superglobal and carry on. If the data is in the session, use it and don't query the database.
Would that cover you?
Assumption
I understand that it's not good to store to much data and it is needed to be as simple.
State today
Now I use as minimum needed and using simple data types (int and strings)
mainly for storing user's id and to tell if he is logged in.
must of my functions are static or singleton that has to be built each post/get.
I have trouble to representing the current state and changing it.
and get a largely static site.
most of state representing goes into javascript .
Target
for the other hand if I'll create a object that represent the entire website it will be much easier for me to maintain user's input , including database interaction.
simple question, how much data should be stored there?
example
One of the things i want to implement is
objects that relate to Database tables,
Let's take a page for a "car.update()".
Now if i store an object for it, that extends a connection to the Database with methods
for CRUD.
When I handle a post back from that page with details i could just put them in properties needed and call the update method.
situation now: I need to create a new object with that details and make an static update
Another example
storing previous search result and filter it using new data
In many cases the ideal amount would be none. Store the username in a cookie along with an HMAC hash used to verify the cookie was created by your site, and get everything else from the database (or cache) as needed. This makes it easy to load balance across servers because any server can handle any request and there's no state that needs to be shared between them.
This approach wouldn't be appropriate for banking or other top-security uses because if someone gets your cookie they connect as you. But for sites where you're not doing anything super critical it's great. The risk can also be mitigated somewhat by adding an expiration mechanism to your cookie handling. See chubbards great answer related to another HMAC question for more info.
note you can switch the way PHP stores data using session_set_save_handler. Then you don't have to change the calls and you improve performances/maintenance with the efficiency of database.
The minimum would be the user I.D.—assuming it is a logging in type of interface. But it is often helpful to include the most common aspects of that, like the user's permission and other items which are stored in the database, but are frequently referenced when constructing pages.
You shouldn't store an enormous amount of data, but you can without problems store some user-information if it helps you server you pages faster.
But if you want to build a more dynamic website, you will probably retreive more and more data from the database. So when you're connecting to a database after all, you could skip storing all kinds of information in the session, because you can just as well get them from the database. Most databases (including MySQL) have a quite efficient query cache that will make repeated queries lightning fast.
So in that case you'll need to save little more than the userid and maybe a small amount of flags.