Context: I am creating a website where I will need to show statistics. The statistics are calculated in Python and I need a place to store the calculated stats so it can be read and be presented in the website. The statistics are calculated by going through around 70000 JSON files so the calculations are done beforehand. The data is not dynamic so all I need to do is just read the data and therefore there is no writing or changing the data.
Solutions:
MySQL approach: I put the statistics in the DB beforehand and use PHP to connect to the MYSQL database and use SELECT statements to get the data and present it.
AJAX (JavaScript) approach: I put the statistics I need into a JSON file and put the file into my server. I use an AJAX call to get the JSON data and parse it and show the statistics from JavaScript.
Question: Which would be the best approach to take?
If speed is top priority, PHP/MYSQL is definitely faster.
With AJAX, I assume that your 70,000 JSON files are split up, and your AJAX call queries the "right one". Depending on your client, the user experience might be nicer since you can get new data without doing a page refresh.
One "happy medium" solution could be do make an ajax call to a query.php file which does the MySQL/PHP lookup, but returns a JSON object so that you can get the best of both worlds!
use the php/mysql approach
why
faster
doesn't consume a lot of resources on the client side and doesn't slow the browser.
Related
My stack is php and mysql.
I am trying to design a page to display details of a mutual fund.
Data for a single fund is distributed over 15-20 different tables.
Currently, my front-end is a brute-force php page that queries/joins these tables using 8 different queries for a single scheme. It's messy and poor performing.
I am considering alternatives. Good thing is that the data changes only once a day, so I can do some preprocessing.
An option that I am considering is to create run these queries for every fund (about 2000 funds) and create a complex json object for each of them, store it in mysql indexed for the fund code, retrieve the json at run time and show the data. I am thinking of using the simple json_object() mysql function to create the json, and json_decode in php to get the values for display. Is this a good approach?
I was tempted to store them in a separate MongoDB store - would that be an overkill for this?
Any other suggestion?
Thanks much!
To meet your objective of quick pageviews, your overnight-run approach is very good. You could generate JSON objects with your distilled data, or even prerendered HTML pages, and store them.
You can certainly store JSON objects in MySQL columns. If you don't need the database server to search the objects, simply use TEXT (or LONGTEXT) data types to store them.
To my way of thinking, adding a new type of server (mongodb) to your operations to store a few thousand JSON objects does not seem worth the the trouble. If you find it necessary to search the contents of your JSON objects, another type of server might be useful, however.
Other things to consider:
Optimize your SQL queries. Read up: https://use-the-index-luke.com and other sources of good info. Consider your queries one-by-one starting with the slowest one. Use the EXPLAIN or even the EXPLAIN ANALYZE command to get your MySQL server to tell you how it plans each query. And judiciously add indexes. Using the query-optimization tag here on StackOverflow, you can get help. Many queries can be optimized by adding indexes to MySQL without changing anything in your php code or your data. So this can be an ongoing project rather than a big new software release.
Consider measuring your query times. You can do this with MySQL's slow query log. The point of this is to identify your "dirty dozen" slowest queries in a particular time period. Then, see step one.
Make your pages fill up progressively, to keep your users busy reading while you get the data they need. Put the toplevel stuff (fund name, etc) in server-side HTML so search engines can see it. Use some sort of front-end tech (React, maybe, or Datatables that fetch data via AJAX) to render your pages client-side, and provide REST endpoints on your server to get the data, in JSON format, for each data block in the page.
In your overnight run create a sitemap file along with your JSON data rows. That lets you control exactly how you want search engines to present your data.
I'm developing an Android app for salesman, so they can use their device to save their order. Specifically, every morning the salesman would go to the office and fetch the data for that day.
Currently, I can get the data by sending a request to php file, and like common practice we insert those data into sqlite in the Android so it can work offline. However, with current approach the device needs 6-8 seconds on getting the data and inserting those data to sqlite. As the data grow bigger I think it would make it slower. What I had found is that the process of inserting data into sqlite takes quite amount of time.
So, I've been thinking about dumping all data that is needed by the salesman into a sqlite file, so I could send only that file which I guess is more efficient. Can you please lead me on how to do that? Or is there any other way which is more efficient approach for this issue?
Note:
Server DB: Mysql
Server: PHP
You can do here different approach to achieve loading speed:
If your data can be pre-loaded with apk, you can just store those inside .apk and when user download app, it will be there, you just need to call remaining updated data.
If you need refreshed data every time, you can do call chunk of data from server in multiple calls, which will fetch and store data in database and update on UI.
If data is not too much, (I say, if there are 200-300 data, we should not consider it much more) you can do simple thing:
When you call network call for fetching data, you should pass that data objects to database for storing and at same time (before storing in db), just return the entire list object to Activity/Fragment, so it will have the data and you can see those data to user, in mean time, it will store in database.
Also, you can use no-sql in side sqlite, so you don't need to parse objects every time (which is costly compare to no-sql) and store all data in sql as entire object and whenever require, just fetch from db and parse it as per requirement.
Thanks to #Skynet for mentioning transaction, it does improve the process alot.. So I'll stay with this approach for now..
You can do something like so:
db.beginTransaction();
try {
saveCustomer();
db.setTransactionSuccessful();
} catch {
//Error in between database transaction
} finally {
db.endTransaction();
}
For more explanation: Android Database Transaction..
I would like to use PHP to write a website, which I need good performance.
I want to use api-style to get json-formatted-data. For example,
If someone want a page to show the 2nd page(50 students per page) of the 1st Grade students, he access the following URL in the browser, such as
http://mywebsite.com/student-list.php?grade=1&page=2.
Inside the above PHP script, I first send a request using curl/file_get_contents to the following link to fetch json-formatted-data:
http://localhost/API/data-api.php?action=students&grade=1&page=2
In ./API/data-api.php, I use $_GET to get the params, and fetch data from database as needed, I may reform the data to an array. Maybe then foreach the array to add a little other customized things to the array, THEN JSON_ENCODE THE ARRAY, AND ECHO IT AS THE LAST OF THE FILE.
(Maybe I also should use content-type:application/json at the head of this data-api.php file? I also will add some memcached logic to cache the data when fetching, that's some other things not involved in this question.)
Maybe I would also use this way to fetch other data from other api-files in this student-list.php. Then show them all in the html template.
The question is : IS THIS WAY OK?
I know single user using the website may be just ok, but is the performance good when there's 2000 or more people accessing the same page? Because I'm using this '
http://localhost/API/data-api.php...
thing, is there some network-delay thing involved? and I use curl/file_get_contents to fetch data from the API file, is this way consuming huge CPU resources? I just want to make the data-json-like and seperate the data-layer from my other view/controller layer as maximum as possible.
I'm stuck at it, don't know whether I should use this way to start writing the website? Cause once I started it will be hard to change.
Thanks for any answer.
I'm rewriting a data visualisation web tool which was written 3 years ago. Since that time, JavaScript engine of browser have become way faster so i was thinking to transfer part of the job from server to client.
On the page, data is visualized in a table and in a map (or chart), it's using the same data, but in a different way so the two algorithm to prepare the data for display are different.
Before at every interaction of the user with the data dropdown selectors (3 main + 2sub depending on the 3 main), 3 ajax request were sent, php doing all the work and sending back only necesary data (in html for the table/xml for the chart) very tiny response, no performance issue and javascript was appending response and doing not much more than chasing change events.
So performance was ok but at every single change of criteria user has to wait for ajax response :/
Now my idea is to send back a json object in one ajax request, only at every change of the main 3 criteria combination and then have javascript populating the data in the table and the chart/map on ajaxsuccess and then also on change of the 2 sub criteria.
My hesitation concerns the structure of the json send by the server, the balance of the payload.
Indeed, if there were only one algorithm necessary to create the wanted json structure to display the data from the raw data, i would have php processing the data into this object ready for javascript to deal with it without any additional treatment; but there are 2.
So
if I make php process the data to create 2 objects (one for table/one for chart), I will double the size of the json response & increase memory usage on the client side; i don't like this approach because this two object contain the same data, just structured differently & redundancy is evil, isn't it ?
if i send the raw object and let javascript search for what to display and where i'm giving lot of job to the client - this also at every subcriteria change (or i could create all the json objects once on ajaxsuccess so they are ready in case of this subcriteria change ?)- here i'm little worry for users with old browser/small ram...
(The raw json object untreated, depending on criteria vary between 3kb and 12kb, between 500 and 2000 records)
I'm failing to spot the best approach...
So for this single raw data to multiple structured objects job, would you have php (increasing response size and sending redundant data) or javascript (increasing javascript payload) processing the raw data ?
Thanks a ton for your opinion
I found an appropriate solution, so I will answer my own question.
I have followed #Daverandom's advice:
PHP sends raw data (along with a couple of parameters that depends on the combination of the main criteria)
JavaScript processes the raw data and render it in the page
JavaScript reprocesses the raw data if sub-criteria are changed, as upon testing the looping process appears to be very fast and doesn't freeze the browser whatsoever, so there is no need to keep the structured object in the scope
Aggressive caching headers are sent with the JSON AJAX response (those data never change - only new records are added every year) in case user re-consults data that has already been consulted: so raw data is not kept in the JavaScript scope if it is not being displayed
On top of that, the JSON strings echoed by php are cached on the server (because those data never change) so this reduces database queries and improves response time
The final code is neat, easy to maintain, and the application works flawlessly.
Thanks to #Daverandom for the help.
I have a MySQL table with about 9.5K rows, these won't change much but I may slowly add to them.
I have a process where if someone scans a barcode I have to check if that barcode matches a value in this table. What would be the fastest way to accomplish this? I must mention there is no pattern to these values
Here Are Some Thoughts
Ajax call to PHP file to query MySQL table ( my thoughts would this would be slowest )
Load this MySQL table into an array on log in. Then when scanning Ajax call to PHP file to check the array
Load this table into an array on log in. When viewing the scanning page somehow load that array into a JavaScript array and check with JavaScript. (this seems to me to be the fastest because it eliminates Ajax call and MySQL Query. Would it be efficient to split into smaller arrays so I don't lag the server & browser?)
Honestly, I'd never load the entire table for anything. All I'd do is make an AJAX request back to a PHP gateway that then queries the database, and returns the result (or nothing). It can be very fast (as it only depends on the latency) and you can cache that result heavily (via memcached, or something like it).
There's really no reason to ever load the entire array for "validation"...
Much faster to used a well indexed MySQL table, then to look through an array for something.
But in the end it all depends on what you really want to do with the data.
As you mentions your table contain around 9.5K of data. There is no logic to load data on login or scanning page.
Better to index your table and do a ajax call whenever required.
Best of Luck!!
While 9.5 K rows are not that much, the related amount of data would need some time to transfer.
Therefore - and in general - I'd propose to run validation of values on the server side. AJAX is the right technology to do this quite easily.
Loading all 9.5 K rows only to find one specific row, is definitely a waste of resources. Run a SELECT-query for the single value.
Exposing PHP-functionality at the client-side / AJAX
Have a look at the xajax project, which allows to expose whole PHP classes or single methods as AJAX method at the client side. Moreover, xajax helps during the exchange of parameters between client and server.
Indexing to be searched attributes
Please ensure, that the column, which holds the barcode value, is indexed. In case the verification process tends to be slow, look out for MySQL table scans.
Avoiding table scans
To avoid table scans and keep your queries run fast, do use fixed sized fields. E.g. VARCHAR() besides other types makes queries slower, since rows no longer have a fixed size. No fixed-sized tables effectively prevent the database to easily predict the location of the next row of the result set. Therefore, you e.g. CHAR(20) instead of VARCHAR().
Finally: Security!
Don't forget, that any data transferred to the client side may expose sensitive data. While your 9.5 K rows may not get rendered by client's browser, the rows do exist in the generated HTML-page. Using Show source any user would be able to figure out all valid numbers.
Exposing valid barcode values may or may not be a security problem in your project context.
PS: While not related to your question, I'd propose to use PHPexcel for reading or writing spreadsheet data. Beside other solutions, e.g. a PEAR-based framework, PHPExcel depends on nothing.