Its known fact that we can use Json/XML parsing or Database to maintain a limited amount of data consistent on a given application and Website at any given time.
However, the dilemma is a project having few display textlines 3k-4k that are supposed to be consistent on both(App and website)on a selected UI, these text files may change at any given point of time. What will be the optimized method or steps of implementation for this technique?
Just my bit would be to store the data in on server as a text file and later use the website or the app to parse the given text file at given location to display it
Related
I have no access to create/edit any databases but due to the massive amount of content I need some sort of management system which is why I created my own. Here's how it works:
Each blog post has got its own .php file which loads static parts of the website like the header or the menu bar. But there are many category sites that display previews of the respective posts. It would be sooo annoying having to edit the same preview on 10 sites due to a misspelled word or so. That's why I store those previews (not the full content since there's no need to) in a JSON file.
Is that bad practice? Could this lead to long loading times if the number of previews rose? And could I prevent that by creating multiple JSON files?
Thanks for your advice!
If you are working on a small scale, then using JSON files is fine however it would definitely be beneficial to switch to a database management system whenever possible for storage when it comes to PHP (Or the majority of languages for that matter).
It can be considered bad practice if JSON is used to store large amounts of data or if a lot of data is stored in the same file in which case yes, using multiple JSON files rather than one large one is indeed more viable since the input stream when reading the file does not have to go over as much data.
I have thousands of Images (wihch are less then 16MB in size) in RAW BINARY DATA in MONGODB with its Meta-Data in JSON as Date, Time, Location etc from the Small Satellite (BSON Documents). I have to make REST API which can query the Images with its respective Meta-Data. Following things needs to be taken under observation.
Data = Data Screenshot
User's will query the Meta-Data with RestAPI, based on time, location etc.
Server will get the Request from Cilent with Query and DO Image-Processing and Returns the Images
Image Processing will be done on Server Side.
Requested Images will travel through RESTAPI from Server to Client with the GET Request.
NOTE : Just see the Attached Picture to Get the Idea of the DATA.
Tools Used : Data-Base = MongoDB
Questions
Which Server Side Programming Language is More feasible? PHP, Python or Node.js?
How I could do Image-Processing in this scenario? With Libraries on PHP, Python or Node.js?
Which Technology to be Used for making REST API for MongoDB which is best with Binary Data and Images.
How Images will travel from Server to client i-e In binary data. and then Renders at Client Side.
NodeJS and Python are equally suited for basic image processing. I would make a server side language determination more on the expertise of your team and/or current environment in that regard.
With Python PIL or Pillow is the primary image library to use. I've used https://github.com/aheckmann/gm for NodeJS which was not difficult to do basic image processing.
(and 4) In terms of retrieving images from an API I typical using a typical REST/CRUD set up to get the metadata and put the image one level deeper using by adding .../image? to the endpoint. For example:
GET ../picture/<id>
Would return the picture metadata with the image url included.
GET ../picture/<id>/image?<processing params>
Would return the image itself.
If you are making a web application using the <img> tag and the correct image URL is sufficient for displaying the image.
I would also recommend storing the images on the filesystem directly (not in the database) unless you have a specific reason to store them in the database. It tends to simplify both you storage code and retrieval since you don't need to deal with sending a BLOB to the database. Nginx (https://www.nginx.com/resources/wiki/modules/upload/) for example lets you set file upload location so literally just need to get the filename from the headers and copy/rename it to the location you want to store the file permanently. This also lets you easily remap a URL to the location on the filesystem. The biggest benefit is that it lets the webserver worry about the upload and download and you just have keep track of the filename in your code.
According to your questions, I will give you answer one by one.
nodejs is better as they have more then 15000 modules.
you can do image processing by using nodejs modules like sharp, Jimp and many more.
MongoDB is a document-oriented NoSQL database (Big Data ready). It stores data in JSON-like format and allows users to perform SQL-like queries against it, and nodejs is best for this purpose.
for this there are number of modules for travels from server to client, like webcamjs.
I am trying to create a website that display Google map mark up with my proprietary data (in form of multiple polygons) on top of it.
I have been studying Google Map API and found the only way to do it is to publish my proprietary data in GeoJSON file then use the following api function to load the data to Google map: map.data.loadGeoJson(xxxxxx);
This means I need to publish my proprietary data by web service in GeoJSON format. However, I don't want users to download my proprietary data and use it for another purpose other than my site. This is similar to a website which allows video streaming but not allowing download the whole video offline.
How can I achieve the purpose? Can I use some language like PHP to generate the map (with markup) at server then send to web client in form of HTML? Or if I cannot achieve this by Google Map API, can other map API support it (like Bing?)
Thanks very much for your help!!!!
Code Mon key
One option is to turn your data into a tile layer. This will limit the user to only seeing an image of the data but would not give them access to the raw data. In a worse case scenario they would only be able to take the images and view the data and not do any kind of analytics against it unless they manually trace all the data.
As an added benefit of rendering the data as a tile layer, you will be able to visualize a lot more data. I've built a few systems that can render 500M rows of polygon data on a map using this approach. The cool thing, if you store the data in a spatial database like SQL Azure, you can easily make your data interactive by taking the point a user clicks on a map and searching the database for any shapes that intersect with that point.
I wrote a simple blog post on how to create a web service that does this many years ago here: https://rbrundritt.wordpress.com/2009/11/26/dynamic-tile-layers-in-the-bing-maps-silverlight-control/
There is also a good open source project here that uses ASP.NET: http://ajaxmapdataconnector.codeplex.com/
I have a whitepaper that is a lot more up to date than my blog post that will be published soon. If you email me at ricky_brundritt at Hotmail.com, I'll send you a draft copy.
Very sorry if this is not in the right place etc. I have been researching this for a while but its raised more questions!
I have developed a spreadsheet which I use to set a teams duties for a shift. There are 5 teams, each with staff which can change day to day.
the spreadsheet works fine, but its too complicated for some users. I am therefore trying to develop a straightforward web based form.
All the data is on the spreadsheet, held on a network drive (essentially locally).
I need to be able to have several combo/select boxes which get their values from a range of cells from the XLS. Along with the ability to output the final selections to a XLS sheet.
Finally, it needs to be able to load the previous day values on load.
What is the best way of developing a web page for this? Is Jscript the best option? Can I access a local file with jScript?
Thanks in advance
Adrian
The easiest option for you is to use google web forms. These allow creation of forms that will submit data to a google spreadsheet. Which is essentially an uploaded version of your local spreadsheet and can be downloaded to excel.
In case if you want more control and programming, pure javascript cant play with files, you need server side too. Javascript is not necessary unless you want to make your app do some visually fancy stuff. Since you mentioned php as a tag of this question, it seems you are a bit familiar with php. The task you have mentioned can be done using php programming as below:
Read excel file using an excel plugin
Parse relevant data using a text matching function, may require regular expressions knowledge.
display the form by building up the html and putting in the variables using the data obtained above.
Write a method to save the data submitted by the form to the same excel file using the excel plugin.
As its not convenient to play around with excel files. A better option would be to generate csv file or use a database using a database class . csv files can be parsed easily using text.
I am trying to create a world application using jQuery (JS) and PHP. I originally tried doing this by using a MySQL database, which didn't work well - the server got overloaded with database queries and crashed.
This time I want to store the data in a text file... maybe use JSON to parse it? How would I do this? The three main things I want are:
Name
x-position
y-position
The x and y positions are given from the JS. So, in order:
User loads page and picks username
User moves character, the jQuery gets the x and y position
The username, x and y position are sent to a PHP page in realtime using jQuery's $.post()
The PHP page has to find some way to store it efficiently without crashing the database.
The PHP page sends back ALL online users' names and x and y coordinates to jQuery
jQuery moves the character; everyone sees the animation.
Storing the data in the file instead of the MySQL database isn't an option if you want to improve performance. Just because MySQL stores its data in the files too, but is use some technics to improve performance like caching and using indexes.
The fastest method to save and retrieve data on server is using RAM as a storage. Redis for example do that. It stores all the data in the RAM and can backup it to the hard drive to prevent data loss.
However I don't think the main problem here is MySQL itself. Probably you use it in an inappropriate way. But I can't say exactly since I don't know how many read and write requests your users generate, what the structure of your tables etc.
Text files are not the best performing things on Earth. Use a key-value store like Redis (it has a PHP client) to store them. It should be able to take a lot more beating than the MySQL server.
You can store the data in a text file in CSV (Comma separated values) format.
For example, consider your requirements.
1,Alice,23,35
2,Bob,44,63
3,Clan,435,322
This text file can be stored and read anytime, and use explode function to separate values.