I have a client that needs to retrieve information from various tables in a mysqli database on a web server. I currently have one AJAX query set up on the client that posts an AJAX request to a php page that queries the database and returns a JSON object. The client then iterates through the resulting object and inserts the necessary data into a similar table in a local database. However this query only contains information from one table. I am going to need to populate multiple tables in the client database. Eventually there could be a large amount of requests to the web server occurring at one time when populating these database. The possible solutions that I've come up with are as follows:
Design multiple ajax queries on the client that each post to the same php page with separate handler queries depending on the type of post received so that different JSON objects are returned for the client to iterate and insert into the local database
(Many AJAX -> 1 PHP)
Design many AJAX queries that each post to a different php page with a single resulting JSON for each AJAX/PHP query to reduce the traffic on any single PHP page.
(Many AJAX -> Many PHP)
Design 1 large AJAX query that makes a single request to the PHP page and returns all the necessary information from the database and have the client insert the different pieces it needs into the local database.
(1 AJAX -> 1 PHP)
Do any of these ideas seem better than the others?
I see flaws in all of them so I'm wondering if there is already an ideal solution to minimize the amount of work done on the client as well as reduce the traffic/maintenance on the server that someone may know of. Any help/criticism is appreciated.
Options 1 and 3 are not mutually exclusive: your server-side code can return small (or partial, depends on how you see it) data sets as in option 1 and at the same time reply to a different type of request with the full data set. The client-side code decides what query to make depending on information that it has regarding the operation being performed. For example, it might use one query if the requested operation was "update one record" and another if the operation was "update all records".
Broadly speaking that's what I would do, so I recommend to leave yourself the flexibility to choose the appropriate query type down the road.
Related
Right now I am learning to write an android api in php for some CRUD operation.
I am able fetch data from the database without any issue.
Now I want to fetch 200, 500 and 600 records from three different tables.
At the end I am going to show it the UI by grouping them at their appropriate position.
My question is, should I write multiple php files for fetching the records from each table and send it back to the user separately (obviously I have call the api separately one after the another from the app after getting the response of each of the previous call).
OR
Only one php file where I will fetch all the records from the three tables and send it back to the user in one shot.
I am returning the data in json format.
Please help in figuring out which one of the above method should I use and its advantage or disadvantage if any..
I am working on this project that I cant seem to get right.
Basically the application makes a request to my PHP server and displays the data on a listView.
Due to the large amount of data I have on my server, and the length of time it takes to get all the data on my listview. I decided to implement an OnScrollListener on the android side, to determine if the last item is visible, then I load more data. This I achieved by selecting all the IDs of the data I want to load when the the initial request is made. The IDs are then sorted based on my requirements (time created, points, etc) after which the first five ids are used to select the initial data which is returned to the android app along with the IDs. Then when the last item is visible, i send the next five ids from the list to a function on php which returns data corresponding to the five IDs.
So far this approach works but it is still unsatisfactory due to the amount large amount of data that needs to be processed during the initial request.
I need help with an alternative technique to achieve my objective with minimal delays while performing the initial request or subsequent request.
Any help will be much appreciated.
From what I read in your question, you are loading all the data at the initial request?
I suggest you to did pagination in your server side so you can minimalize the number of data, and call the next portion/page of data only when you need to do it (in this case you can implement it in OnScrollListener)
For more details about pagination
- http://www.phpfreaks.com/tutorial/basic-pagination
- http://www.phpeasystep.com/phptu/29.html
- http://code.tutsplus.com/tutorials/how-to-paginate-data-with-php--net-2928
I'm rewriting a data visualisation web tool which was written 3 years ago. Since that time, JavaScript engine of browser have become way faster so i was thinking to transfer part of the job from server to client.
On the page, data is visualized in a table and in a map (or chart), it's using the same data, but in a different way so the two algorithm to prepare the data for display are different.
Before at every interaction of the user with the data dropdown selectors (3 main + 2sub depending on the 3 main), 3 ajax request were sent, php doing all the work and sending back only necesary data (in html for the table/xml for the chart) very tiny response, no performance issue and javascript was appending response and doing not much more than chasing change events.
So performance was ok but at every single change of criteria user has to wait for ajax response :/
Now my idea is to send back a json object in one ajax request, only at every change of the main 3 criteria combination and then have javascript populating the data in the table and the chart/map on ajaxsuccess and then also on change of the 2 sub criteria.
My hesitation concerns the structure of the json send by the server, the balance of the payload.
Indeed, if there were only one algorithm necessary to create the wanted json structure to display the data from the raw data, i would have php processing the data into this object ready for javascript to deal with it without any additional treatment; but there are 2.
So
if I make php process the data to create 2 objects (one for table/one for chart), I will double the size of the json response & increase memory usage on the client side; i don't like this approach because this two object contain the same data, just structured differently & redundancy is evil, isn't it ?
if i send the raw object and let javascript search for what to display and where i'm giving lot of job to the client - this also at every subcriteria change (or i could create all the json objects once on ajaxsuccess so they are ready in case of this subcriteria change ?)- here i'm little worry for users with old browser/small ram...
(The raw json object untreated, depending on criteria vary between 3kb and 12kb, between 500 and 2000 records)
I'm failing to spot the best approach...
So for this single raw data to multiple structured objects job, would you have php (increasing response size and sending redundant data) or javascript (increasing javascript payload) processing the raw data ?
Thanks a ton for your opinion
I found an appropriate solution, so I will answer my own question.
I have followed #Daverandom's advice:
PHP sends raw data (along with a couple of parameters that depends on the combination of the main criteria)
JavaScript processes the raw data and render it in the page
JavaScript reprocesses the raw data if sub-criteria are changed, as upon testing the looping process appears to be very fast and doesn't freeze the browser whatsoever, so there is no need to keep the structured object in the scope
Aggressive caching headers are sent with the JSON AJAX response (those data never change - only new records are added every year) in case user re-consults data that has already been consulted: so raw data is not kept in the JavaScript scope if it is not being displayed
On top of that, the JSON strings echoed by php are cached on the server (because those data never change) so this reduces database queries and improves response time
The final code is neat, easy to maintain, and the application works flawlessly.
Thanks to #Daverandom for the help.
I have a MySQL table with about 9.5K rows, these won't change much but I may slowly add to them.
I have a process where if someone scans a barcode I have to check if that barcode matches a value in this table. What would be the fastest way to accomplish this? I must mention there is no pattern to these values
Here Are Some Thoughts
Ajax call to PHP file to query MySQL table ( my thoughts would this would be slowest )
Load this MySQL table into an array on log in. Then when scanning Ajax call to PHP file to check the array
Load this table into an array on log in. When viewing the scanning page somehow load that array into a JavaScript array and check with JavaScript. (this seems to me to be the fastest because it eliminates Ajax call and MySQL Query. Would it be efficient to split into smaller arrays so I don't lag the server & browser?)
Honestly, I'd never load the entire table for anything. All I'd do is make an AJAX request back to a PHP gateway that then queries the database, and returns the result (or nothing). It can be very fast (as it only depends on the latency) and you can cache that result heavily (via memcached, or something like it).
There's really no reason to ever load the entire array for "validation"...
Much faster to used a well indexed MySQL table, then to look through an array for something.
But in the end it all depends on what you really want to do with the data.
As you mentions your table contain around 9.5K of data. There is no logic to load data on login or scanning page.
Better to index your table and do a ajax call whenever required.
Best of Luck!!
While 9.5 K rows are not that much, the related amount of data would need some time to transfer.
Therefore - and in general - I'd propose to run validation of values on the server side. AJAX is the right technology to do this quite easily.
Loading all 9.5 K rows only to find one specific row, is definitely a waste of resources. Run a SELECT-query for the single value.
Exposing PHP-functionality at the client-side / AJAX
Have a look at the xajax project, which allows to expose whole PHP classes or single methods as AJAX method at the client side. Moreover, xajax helps during the exchange of parameters between client and server.
Indexing to be searched attributes
Please ensure, that the column, which holds the barcode value, is indexed. In case the verification process tends to be slow, look out for MySQL table scans.
Avoiding table scans
To avoid table scans and keep your queries run fast, do use fixed sized fields. E.g. VARCHAR() besides other types makes queries slower, since rows no longer have a fixed size. No fixed-sized tables effectively prevent the database to easily predict the location of the next row of the result set. Therefore, you e.g. CHAR(20) instead of VARCHAR().
Finally: Security!
Don't forget, that any data transferred to the client side may expose sensitive data. While your 9.5 K rows may not get rendered by client's browser, the rows do exist in the generated HTML-page. Using Show source any user would be able to figure out all valid numbers.
Exposing valid barcode values may or may not be a security problem in your project context.
PS: While not related to your question, I'd propose to use PHPexcel for reading or writing spreadsheet data. Beside other solutions, e.g. a PEAR-based framework, PHPExcel depends on nothing.
Ok so i am new to making complex database structure into the page. I have a basic people table with a few categories. Students, teachers, parents and mods. There are again tables one for parents, students and teachers/mods. Its basically a school's website.
Now for example for a profile page where a student's info is showed to parents. Info like who are teachers, subjects, attendance, homework, etc. This will query a lot of tables. So what is my best bet here? I plan to do it in a web-app way. I was thinking maybe i can JSON data to page with ajax and let javascript do the heavy lifting of calculation. All the tables will only be queried once.
So is it even ok to do so? or will i face some hidden problems when i have dug my feet too deep in it? Maybe i can go one more level deep and make a huge JSON with entire database being sent to user and then it is cached in the browser. I dont have to support really old browsers :)
Also things like attendance and marks/result will need to be calculated every time. Is there any way to cache the result per student? like wen a parent views a student's result, it calculates it once and then caches it for x number of days on the server somehow.
EDIT:
Another non JSON approach i can think of is traditional way. I do all the things on server side with php. This means i wont have to worry about browser compatibility. Maybe i can make a call to all the needed tables in the beginning and store it is an array. That way a table only gets called once too.
EDIT 2
I read somewhere about battlefield 3 battlelog website. It is also made the same way. All the data is pulled from server with JSON and then calculated on client side. If that helps put my idea in perspective.
Probably to move away some misconceptions, but this is more a lengthy comment than an answer:
Your database has tables and fiels. That's perfectly valid. It's the job of a database server to store the data and handle the querying for you. Your own code will never be better in joining tables than the code of the database. Because the database server is optimized for the task.
So I think the idea to query all data and put it into your webapp via JSON is just a bad idea. Instead contact your server if you need specific data, the server will build the related SQL query, fire it up to the database server, get's the result back, converts the result into JSON and sends it back to the browser.
AJAX (Asynchronus Javascript and XML) just allows you to fetch data on the fly after the main portion of the page is painted. All things being equal it might be easier for you to design it as a standard (fetch data then paint the page) layout. If you design the application correctly the calls to do the individual components (Attendance, Teachers, Grades, etc.) can be gathered pre-page-render or post page render. There's obvious security concerns (URL hacking) by going the AJAX route, and I personally design non-AJAX as it's less flukey when things start going wierd.