For Example
Let say we have 1000 products in a single category.
And we would like to Filter through these products.
As we Filter through the products using json.
Each time we have need run a separate query to the DB.
We were wondering if any one knows if it's possible display a preload the products table.
For example preload bar: initializing search (0 - 100%)
So the whole system would only initialize once on load then we would hope the search results could then be instant.
Unfortunately tweaking customers servers isn't really an option for us, so hopefully someone here may have a better suggestion
Thanks in advance!
On the PHP end, you could preload the records into something like a memcache bucket.
I strongly doubt, though, that the problem is the database query. It's much more likely that the json request is what takes so long.
It's certainly possible to preload all the values (i.e. fetch them in one big request). You are not giving any details about your system, but it would probably simply be a big json request.
I don't see any easy way to implement a progress bar for the Ajax request. Maybe a simple "loading" animation will do as well.
You would probably have to use some sort of local storage to persist the data across pages for it to make sense. For that, see this question.
1000 data sets isn't that much to query through, but it is a significant amount to store client side. However.. to answer your question:
Javscript can create an array of objects. Let's say
<script type="text/javascript">
var products = new Array()
products[0] = {name:'First', price:29.99};
products[1] = {name:'Second', price:29.99};
products[2] = {name:'Third', price:29.99};
</script>
... create these via php. Also instead of {...} you can create a function product(name, price) and then call products[0] = new product("first", 29.99);
When you have this set up all information is stored in the clients browser. So now you can use a search/filter via only javascript.
Loading bar can be neatly done through jquery-ui. Searching involves an array loop.
If you have multiple categories that you can separate before hand - you can just create different arrays, instead of storing everything in products array.
Good luck!
Related
My stack is php and mysql.
I am trying to design a page to display details of a mutual fund.
Data for a single fund is distributed over 15-20 different tables.
Currently, my front-end is a brute-force php page that queries/joins these tables using 8 different queries for a single scheme. It's messy and poor performing.
I am considering alternatives. Good thing is that the data changes only once a day, so I can do some preprocessing.
An option that I am considering is to create run these queries for every fund (about 2000 funds) and create a complex json object for each of them, store it in mysql indexed for the fund code, retrieve the json at run time and show the data. I am thinking of using the simple json_object() mysql function to create the json, and json_decode in php to get the values for display. Is this a good approach?
I was tempted to store them in a separate MongoDB store - would that be an overkill for this?
Any other suggestion?
Thanks much!
To meet your objective of quick pageviews, your overnight-run approach is very good. You could generate JSON objects with your distilled data, or even prerendered HTML pages, and store them.
You can certainly store JSON objects in MySQL columns. If you don't need the database server to search the objects, simply use TEXT (or LONGTEXT) data types to store them.
To my way of thinking, adding a new type of server (mongodb) to your operations to store a few thousand JSON objects does not seem worth the the trouble. If you find it necessary to search the contents of your JSON objects, another type of server might be useful, however.
Other things to consider:
Optimize your SQL queries. Read up: https://use-the-index-luke.com and other sources of good info. Consider your queries one-by-one starting with the slowest one. Use the EXPLAIN or even the EXPLAIN ANALYZE command to get your MySQL server to tell you how it plans each query. And judiciously add indexes. Using the query-optimization tag here on StackOverflow, you can get help. Many queries can be optimized by adding indexes to MySQL without changing anything in your php code or your data. So this can be an ongoing project rather than a big new software release.
Consider measuring your query times. You can do this with MySQL's slow query log. The point of this is to identify your "dirty dozen" slowest queries in a particular time period. Then, see step one.
Make your pages fill up progressively, to keep your users busy reading while you get the data they need. Put the toplevel stuff (fund name, etc) in server-side HTML so search engines can see it. Use some sort of front-end tech (React, maybe, or Datatables that fetch data via AJAX) to render your pages client-side, and provide REST endpoints on your server to get the data, in JSON format, for each data block in the page.
In your overnight run create a sitemap file along with your JSON data rows. That lets you control exactly how you want search engines to present your data.
I would like to use PHP to write a website, which I need good performance.
I want to use api-style to get json-formatted-data. For example,
If someone want a page to show the 2nd page(50 students per page) of the 1st Grade students, he access the following URL in the browser, such as
http://mywebsite.com/student-list.php?grade=1&page=2.
Inside the above PHP script, I first send a request using curl/file_get_contents to the following link to fetch json-formatted-data:
http://localhost/API/data-api.php?action=students&grade=1&page=2
In ./API/data-api.php, I use $_GET to get the params, and fetch data from database as needed, I may reform the data to an array. Maybe then foreach the array to add a little other customized things to the array, THEN JSON_ENCODE THE ARRAY, AND ECHO IT AS THE LAST OF THE FILE.
(Maybe I also should use content-type:application/json at the head of this data-api.php file? I also will add some memcached logic to cache the data when fetching, that's some other things not involved in this question.)
Maybe I would also use this way to fetch other data from other api-files in this student-list.php. Then show them all in the html template.
The question is : IS THIS WAY OK?
I know single user using the website may be just ok, but is the performance good when there's 2000 or more people accessing the same page? Because I'm using this '
http://localhost/API/data-api.php...
thing, is there some network-delay thing involved? and I use curl/file_get_contents to fetch data from the API file, is this way consuming huge CPU resources? I just want to make the data-json-like and seperate the data-layer from my other view/controller layer as maximum as possible.
I'm stuck at it, don't know whether I should use this way to start writing the website? Cause once I started it will be hard to change.
Thanks for any answer.
Ok so i am new to making complex database structure into the page. I have a basic people table with a few categories. Students, teachers, parents and mods. There are again tables one for parents, students and teachers/mods. Its basically a school's website.
Now for example for a profile page where a student's info is showed to parents. Info like who are teachers, subjects, attendance, homework, etc. This will query a lot of tables. So what is my best bet here? I plan to do it in a web-app way. I was thinking maybe i can JSON data to page with ajax and let javascript do the heavy lifting of calculation. All the tables will only be queried once.
So is it even ok to do so? or will i face some hidden problems when i have dug my feet too deep in it? Maybe i can go one more level deep and make a huge JSON with entire database being sent to user and then it is cached in the browser. I dont have to support really old browsers :)
Also things like attendance and marks/result will need to be calculated every time. Is there any way to cache the result per student? like wen a parent views a student's result, it calculates it once and then caches it for x number of days on the server somehow.
EDIT:
Another non JSON approach i can think of is traditional way. I do all the things on server side with php. This means i wont have to worry about browser compatibility. Maybe i can make a call to all the needed tables in the beginning and store it is an array. That way a table only gets called once too.
EDIT 2
I read somewhere about battlefield 3 battlelog website. It is also made the same way. All the data is pulled from server with JSON and then calculated on client side. If that helps put my idea in perspective.
Probably to move away some misconceptions, but this is more a lengthy comment than an answer:
Your database has tables and fiels. That's perfectly valid. It's the job of a database server to store the data and handle the querying for you. Your own code will never be better in joining tables than the code of the database. Because the database server is optimized for the task.
So I think the idea to query all data and put it into your webapp via JSON is just a bad idea. Instead contact your server if you need specific data, the server will build the related SQL query, fire it up to the database server, get's the result back, converts the result into JSON and sends it back to the browser.
AJAX (Asynchronus Javascript and XML) just allows you to fetch data on the fly after the main portion of the page is painted. All things being equal it might be easier for you to design it as a standard (fetch data then paint the page) layout. If you design the application correctly the calls to do the individual components (Attendance, Teachers, Grades, etc.) can be gathered pre-page-render or post page render. There's obvious security concerns (URL hacking) by going the AJAX route, and I personally design non-AJAX as it's less flukey when things start going wierd.
A client has given me a task to do that I've not done before, and so I'm looking for the best way to do it. They have a form they want users to fill in, but for one field, they want an option of thousands to be placed into three dropdown menus.
For example:
So a user will only be able to choose a Venue once they've chosen a City, only a City once they'd chosen a State. (A nice way to break up the thousands of options.)
I suppose I could this quite easily using POSTBACKs and a simple database, but I imagine that doing something with AJAX and a simple database would be the slicker solution.
Are there any other ways that this problem might be tackled? If not, does anyone have any links to tutorials or code snippets I could grab? Secondly, how long do you think it would take you to implement such a system?
I've never done this before so I'm hoping to avoid as many unforeseen pitfalls as possible. Thanks.
How about a simple jQuery Autocomplete solution?
I've done such a thing, also with several thousands of entries.
The problems:
it's difficult for the end user to navigate trough the list if there are hundreds of cities to choose
dropdowns as they are terrible for such things
querying a database to obtain info is stressful because the query is basically the same, with same results, nearly never-changing.
So on to solutions:
I still stood by dropdowns, but I added (trough UI) options for users to filter the list a bit. I won't post the code or the layout, if you are fine with the dropdowns as they are - keep them.
I loaded all of the countries, cities and areas via JS once. Now, why - first off, it wasn't that huge of a deal, it was about 20ish kilobytes gzipped if I'm not mistaken. I didn't want the "please choose a country" > "please wait, loading cities" > "choose a city" > "please wait, loading areas" > "choose an area" thing, I absolutely hate waiting so I don't want anyone to wait if they don't have to :)
So, the whole structure is loaded at once (when page is requested) and kept inside an object. If the browser supports sessionStorage, I check whether I have the object there in the first place - if not, I load it via jQuery's $.ajax call.
On the web server, the script returning JSON object actually loads the data from Memcache. If there's no entry in the Memcache, then I query the db and load all the data and I store it with Memcache for later use.
Now, what happens is that I've got a JS object representing all countries, cities and areas - with relations, meaning I can render the dropdowns in any way I need to, showing only relevant cities for current country selection.
Now, as you have similar structure - you can apply the same logic.
Load the item when the page loads, but check if you have sessionStorage available. If yes, check if you got object there. No? Do a $.ajax call and obtain it. When dropdowns fire change event, render the appropriate data.
Hopefully this helps a little, I'm typing this in a rush :)
A few responses:
This is a good use of AJAX, no need to look for another method. You wouldn't want to force the client to pre-load all of the javascript arrays for the possible state/city/theater combinations...
jQuery Autocomplete is a good tool to use to implement the UI
The list of cities and states can be obtained from GeoNames
How long it would take to implement depends on the skill of the implementer ;)
Somewhat working example: http://jsfiddle.net/rA9gU/36/
Hope this might help
UPDATE: Added the NO theater found near you option
http://jsfiddle.net/rA9gU/39/
You've got it. On the "on change" event for each dropdown, run an AJAX request for the options for the next dropdown.
With jQuery it's pretty simple.
$(document).ready(function () {
$("#state").change(function () {
// AJAX call w/ $.get or $.post to a script to return and set city options
});
// Same for city to retrieve cinema options
....
});
jQuery is by no means a requirement -- just wraps things up nicely and in cross-browser fashion.
Be happy to provide a more specific example if you like.
I have a table with just 3,000 records.
I render these 3000 records in the home page without pagination, my client is not interested in pagination...
So to show page completely it takes around 1 min, 15 sec. What can be done to make the page load more quickly?
My table structure:
customer table
customer id
customer name
guider id
and few columns
guider table
guider id
guider name
and few columns
Where's the slow down? The query or the serving?
If the former, see the comments above. If the latter:
Enable gzip on the server. Otherwise capture the [HTML?] output to a file, compress it (zip), then serve it as a download. Same for any other format if you think something else can render it better than a browser (CSV and Open Office).
If you're outputting the data into a HTML table then you may have an issue where the browser is waiting for the end of the table before rendering it. You can either break this into multiple table chunks like every 500 records/rows or try CSS "table-layout: fixed;".
Check the Todos
sql Connection (dont open the
connection in loop) for query it
should be one time connection
check your queries and analyse it if you are using some complex logic
which can be replaced
use standard class for sql connection and query ; use ezsql
sql query best practice
While you could implement a cache to do this, you don't necessarily need to do so, an introducing unnecessary cache structures can often cause problems of its own. Depending on where the bottleneck is, it may not even help you much, or at all.
You need to look in two places for your analysis:
1) The query you're using to get your data. Take a look at its plan, or if you're not comfortable doing that, run it in your favorite query tool and see how long it takes to come back. If it doesn't take too long, you've got a pretty good idea that your bottleneck isn't the query. If the query itself takes a long time, that's where you should focus your efforts.
2) How your page is rendering. What is the size of your page, in bytes? It may be too big. Can you cut the size down by formatting? Can you more effectively use CSS to eliminate duplicate styling on the page? Are you using a fixed or dynamic table layout? Dynamic is generally going to be quite a bit slower, especially for large tables. Try to avoid nesting tables. Do everything you can to make the page as small as possible, and keep testing!
while displaying records i want to
display guidername so , i did once
function that return the guider name
Sounds like you need to use a JOIN. Here's a simple example:
SELECT * FROM customer JOIN guider ON guider.id=customer.guider_id
This will change your page from using N + 1 (3001) queries to just one.
Make sure both guider.id and customer.guider_id are indexed and of appropriate data types (such as integers).
This is a little list, what you should think about for improving the performance, the importance is relative to each point, so the first ist not to be the most important to you - which depends on the details of your project.
Check your database structure. If there are just these two tables, their might be little you can do. But keep in mind that there is stuff like indices and with an increasing number of records a second denormalizes table structure will improve the speed of retrieving results.
Use rather one Query for selecting your data, than iterating through ids and doing selects repeatedly
Run a separate Query for the guiders, I assume there are only a few of them. Save all guiders in a data structure, e.g. a dictionary, first and use the foreign key to apply the correct one to the current record - this might save a lot of data which has to be transmitted from the database to your web server.
Get your result set by using something like mysqli_result::fetch_all() which returns a 2-dimensional array with all results. This should be faster than iteration through each row with fetch_row()
Sanitize your HTML Output, use (external) CSS. This will save a lot of output space if you format your stuff with style=" ... a lot of formatting code ..." attributes in each line. If you use one large table, split them up in multiple tables (some browsers wait for the complete table to load before rendering it).
In a lot of languages very important: Use a string builder for concatenating your results into the output string!
Caching: Think about generating the output once a day or once an hour. Write it to a cachefile which is opened instead of querying the database and building the same stuff on every request. Maybe you want to offer this generated file as download, rather than displaying it as plain HTML Site on the web.
Last but not least, check the connections to webserver and database, the server load as well as the number of requests. If your servers are running on heavy load everything ales here might help reducing the load or you just have to upgrade hardware.
LOL
everyone is talking of big boys toys, like database structure, caching and stuff.
While the problem most likely lays in mere HTML and browsers.
Just to split whole HTML table in chunks will help first chunk to show up immediately while others will eventually come.
Only ones were right who said to profile whole thing first.
Trying to answer without profiling results is shooting in the dark.