I'm rewriting a data visualisation web tool which was written 3 years ago. Since that time, JavaScript engine of browser have become way faster so i was thinking to transfer part of the job from server to client.
On the page, data is visualized in a table and in a map (or chart), it's using the same data, but in a different way so the two algorithm to prepare the data for display are different.
Before at every interaction of the user with the data dropdown selectors (3 main + 2sub depending on the 3 main), 3 ajax request were sent, php doing all the work and sending back only necesary data (in html for the table/xml for the chart) very tiny response, no performance issue and javascript was appending response and doing not much more than chasing change events.
So performance was ok but at every single change of criteria user has to wait for ajax response :/
Now my idea is to send back a json object in one ajax request, only at every change of the main 3 criteria combination and then have javascript populating the data in the table and the chart/map on ajaxsuccess and then also on change of the 2 sub criteria.
My hesitation concerns the structure of the json send by the server, the balance of the payload.
Indeed, if there were only one algorithm necessary to create the wanted json structure to display the data from the raw data, i would have php processing the data into this object ready for javascript to deal with it without any additional treatment; but there are 2.
So
if I make php process the data to create 2 objects (one for table/one for chart), I will double the size of the json response & increase memory usage on the client side; i don't like this approach because this two object contain the same data, just structured differently & redundancy is evil, isn't it ?
if i send the raw object and let javascript search for what to display and where i'm giving lot of job to the client - this also at every subcriteria change (or i could create all the json objects once on ajaxsuccess so they are ready in case of this subcriteria change ?)- here i'm little worry for users with old browser/small ram...
(The raw json object untreated, depending on criteria vary between 3kb and 12kb, between 500 and 2000 records)
I'm failing to spot the best approach...
So for this single raw data to multiple structured objects job, would you have php (increasing response size and sending redundant data) or javascript (increasing javascript payload) processing the raw data ?
Thanks a ton for your opinion
I found an appropriate solution, so I will answer my own question.
I have followed #Daverandom's advice:
PHP sends raw data (along with a couple of parameters that depends on the combination of the main criteria)
JavaScript processes the raw data and render it in the page
JavaScript reprocesses the raw data if sub-criteria are changed, as upon testing the looping process appears to be very fast and doesn't freeze the browser whatsoever, so there is no need to keep the structured object in the scope
Aggressive caching headers are sent with the JSON AJAX response (those data never change - only new records are added every year) in case user re-consults data that has already been consulted: so raw data is not kept in the JavaScript scope if it is not being displayed
On top of that, the JSON strings echoed by php are cached on the server (because those data never change) so this reduces database queries and improves response time
The final code is neat, easy to maintain, and the application works flawlessly.
Thanks to #Daverandom for the help.
Related
Context: I am creating a website where I will need to show statistics. The statistics are calculated in Python and I need a place to store the calculated stats so it can be read and be presented in the website. The statistics are calculated by going through around 70000 JSON files so the calculations are done beforehand. The data is not dynamic so all I need to do is just read the data and therefore there is no writing or changing the data.
Solutions:
MySQL approach: I put the statistics in the DB beforehand and use PHP to connect to the MYSQL database and use SELECT statements to get the data and present it.
AJAX (JavaScript) approach: I put the statistics I need into a JSON file and put the file into my server. I use an AJAX call to get the JSON data and parse it and show the statistics from JavaScript.
Question: Which would be the best approach to take?
If speed is top priority, PHP/MYSQL is definitely faster.
With AJAX, I assume that your 70,000 JSON files are split up, and your AJAX call queries the "right one". Depending on your client, the user experience might be nicer since you can get new data without doing a page refresh.
One "happy medium" solution could be do make an ajax call to a query.php file which does the MySQL/PHP lookup, but returns a JSON object so that you can get the best of both worlds!
use the php/mysql approach
why
faster
doesn't consume a lot of resources on the client side and doesn't slow the browser.
I am working on this project that I cant seem to get right.
Basically the application makes a request to my PHP server and displays the data on a listView.
Due to the large amount of data I have on my server, and the length of time it takes to get all the data on my listview. I decided to implement an OnScrollListener on the android side, to determine if the last item is visible, then I load more data. This I achieved by selecting all the IDs of the data I want to load when the the initial request is made. The IDs are then sorted based on my requirements (time created, points, etc) after which the first five ids are used to select the initial data which is returned to the android app along with the IDs. Then when the last item is visible, i send the next five ids from the list to a function on php which returns data corresponding to the five IDs.
So far this approach works but it is still unsatisfactory due to the amount large amount of data that needs to be processed during the initial request.
I need help with an alternative technique to achieve my objective with minimal delays while performing the initial request or subsequent request.
Any help will be much appreciated.
From what I read in your question, you are loading all the data at the initial request?
I suggest you to did pagination in your server side so you can minimalize the number of data, and call the next portion/page of data only when you need to do it (in this case you can implement it in OnScrollListener)
For more details about pagination
- http://www.phpfreaks.com/tutorial/basic-pagination
- http://www.phpeasystep.com/phptu/29.html
- http://code.tutsplus.com/tutorials/how-to-paginate-data-with-php--net-2928
I have a client that needs to retrieve information from various tables in a mysqli database on a web server. I currently have one AJAX query set up on the client that posts an AJAX request to a php page that queries the database and returns a JSON object. The client then iterates through the resulting object and inserts the necessary data into a similar table in a local database. However this query only contains information from one table. I am going to need to populate multiple tables in the client database. Eventually there could be a large amount of requests to the web server occurring at one time when populating these database. The possible solutions that I've come up with are as follows:
Design multiple ajax queries on the client that each post to the same php page with separate handler queries depending on the type of post received so that different JSON objects are returned for the client to iterate and insert into the local database
(Many AJAX -> 1 PHP)
Design many AJAX queries that each post to a different php page with a single resulting JSON for each AJAX/PHP query to reduce the traffic on any single PHP page.
(Many AJAX -> Many PHP)
Design 1 large AJAX query that makes a single request to the PHP page and returns all the necessary information from the database and have the client insert the different pieces it needs into the local database.
(1 AJAX -> 1 PHP)
Do any of these ideas seem better than the others?
I see flaws in all of them so I'm wondering if there is already an ideal solution to minimize the amount of work done on the client as well as reduce the traffic/maintenance on the server that someone may know of. Any help/criticism is appreciated.
Options 1 and 3 are not mutually exclusive: your server-side code can return small (or partial, depends on how you see it) data sets as in option 1 and at the same time reply to a different type of request with the full data set. The client-side code decides what query to make depending on information that it has regarding the operation being performed. For example, it might use one query if the requested operation was "update one record" and another if the operation was "update all records".
Broadly speaking that's what I would do, so I recommend to leave yourself the flexibility to choose the appropriate query type down the road.
Let say I have a page with 100 objects and each page is around 700 bytes when converted to json.
In order to save the objects to the php based controller I have the following options.
Option 1
For each objects (100 objects) do the following
Take object definition
2 .convert to json
Do http post to the php controller
the php controller saves it to a file or database.
Option 2
Variable bigJsonString;
For each objects (100 objects) do the following
Take object definition
2 .convert to json
append the json to a string variable "bigJsonString" with a delimitter to indicate end of object.
After the big fat bigJsonString is constructed
Do http post to the php controller by sending "bigJsonString"
the php controller saves it to a file or database.
In option 1, I am doing 100 http posts one after another. Does this raise any alarms?
Is this normal for web applications doing ajax post?
The second option seems safe but then the only concern is when the 100 objects become say 500 objects or to a point where the "bigJsonString" goes several Megabytes long.
The third option we can introduce is a hybrid of option 1 and 2 where we start by constructing the "bigJsonString" and if the length goes to a certain limit then do a ajax post. Flush the string and build the string again for remaining objects.
What are the pitfalls and what is the normal or standard practice. if someone can point to resources where this is already analysed, that would be great.
Thanks very much.
Browsers generally limit the number of connections to a single domain to a low number (under 20 by default for most browsers). In the meantime, many of your requests will block.
On the other hand, larger requests will take longer to fully process because there are less opportunities for parallelization.
Ideally, you would try both methods and see which one works most effectively.
A note: for the second method, you could create an array of the objects then serialize the array as JSON, instead of manually dealing with the JSON string. (JSON supports arrays, not just objects!)
I guess its situational. However, I don't see any situation in which sending 100 requests to the server all at one time (per se) is good.
Personally, I'd just push each object to a javascript array and send a JSON representation of the array to PHP, that way you don't have to worry about delimiters.
It depends how fast you are posting the objects to the server. If the json objects are being posted say every second, 1 object per post isn't too bad. But if you are posting 100 in a second, you really need to build up a large request.
There will be significant lag for each request. Building a large multi object json string is preferable in terms of performance.
What if there is an error in one of the objects? You will need to make sure it doesn't dump stop processing of all the other objects or the user will have to upload all that data again.
If you do multiple requests, you can give better user feedback client side since you know exactly where you are in the queue of objects.
It is up to you to balance all this.
Good luck.
Go for option 2, bigJsonString. You should have no trouble passing messages that are several megabytes long - the same infrastructure is used to pass much larger html, image, style, script and video files over the internets.
Dropping my lurker status to finally ask a question...
I need to know how I can improve on the performance of a PHP script that draws its data from XML files.
Some background:
I've already mapped the bottleneck to CPU - but want to optimize the script's performance before taking a hit on processor costs. Specifically, the most CPU-consuming part of the script is the XML loading.
The reason I'm using XML to store object data because the data needs to be accessible via a browser Flash interface, and we want to provide fast user access in that area. The project is still in early stages though, so if best practice would be to abandon XML altogether, that would be a good answer too.
Lots of data: Currently plotting for roughly 100k objects, albeit usually small ones - and they must ALL be taken up into the script, with perhaps a few rare exceptions. The data set will only grow with time.
Frequent runs: Ideally, we'd run the script ~50k times an hour; realistically, we'd settle for ~1k/h runs. This coupled with data size makes performance optimization completely imperative.
Already taken an optimization step of making several runs on the same data rather than loading it for each run, but it's still taking too long. The runs should generally use "fresh" data with the modifications done by users.
Just to clarify: is the data you're loading coming from XML files for processing in its current state and is it being modified before being sent to the Flash application?
It looks like you'd be better off using a database to store your data and pushing out XML as needed rather than reading it in XML first; if building the XML files gets slow you could cache files as they're generated in order to avoid redundant generation of the same file.
If the XML stays relatively static, you could cache it as a PHP array, something like this:
<xml><foo>bar</foo></xml>
is cached in a file as
<?php return array('foo' => 'bar');
It should be faster for PHP to just include the arrayified version of the XML.
~1k/hour, 3600 seconds per hour, more than 3 runs a second (let alone the 50k/hour)...
There are many questions. Some of them are:
Does your php script need to read/process all records of the data source for each single run? If not, what kind of subset does it need (~size, criterias, ...)
Same question for the flash application + who's sending the data? The php script? "Direct" request for the complete, static xml file?
What operations are performed on the data source?
Do you need some kind of concurrency mechanism?
...
And just because you want to deliver xml data to the flash clients it doesn't necessarily mean that you have to store xml data on the server. If e.g. the clients only need a tiny little subset of the availabe records it probably a lot faster not to store the data as xml but something more suited to speed and "searchability" and then create the xml output of the subset on-the-fly, maybe assisted by some caching depending on what data the client request and how/how much the data changes.
edit: Let's assume that you really,really need the whole dataset and need a continuous simulation. Then you might want to consider a continuous process that keeps the complete "world model" in memory and operates on this model on each run (world tick). This way at least you wouldn't have to load the data on each tick. But such a process is usually written in something else than php.