I have thousands of Images (wihch are less then 16MB in size) in RAW BINARY DATA in MONGODB with its Meta-Data in JSON as Date, Time, Location etc from the Small Satellite (BSON Documents). I have to make REST API which can query the Images with its respective Meta-Data. Following things needs to be taken under observation.
Data = Data Screenshot
User's will query the Meta-Data with RestAPI, based on time, location etc.
Server will get the Request from Cilent with Query and DO Image-Processing and Returns the Images
Image Processing will be done on Server Side.
Requested Images will travel through RESTAPI from Server to Client with the GET Request.
NOTE : Just see the Attached Picture to Get the Idea of the DATA.
Tools Used : Data-Base = MongoDB
Questions
Which Server Side Programming Language is More feasible? PHP, Python or Node.js?
How I could do Image-Processing in this scenario? With Libraries on PHP, Python or Node.js?
Which Technology to be Used for making REST API for MongoDB which is best with Binary Data and Images.
How Images will travel from Server to client i-e In binary data. and then Renders at Client Side.
NodeJS and Python are equally suited for basic image processing. I would make a server side language determination more on the expertise of your team and/or current environment in that regard.
With Python PIL or Pillow is the primary image library to use. I've used https://github.com/aheckmann/gm for NodeJS which was not difficult to do basic image processing.
(and 4) In terms of retrieving images from an API I typical using a typical REST/CRUD set up to get the metadata and put the image one level deeper using by adding .../image? to the endpoint. For example:
GET ../picture/<id>
Would return the picture metadata with the image url included.
GET ../picture/<id>/image?<processing params>
Would return the image itself.
If you are making a web application using the <img> tag and the correct image URL is sufficient for displaying the image.
I would also recommend storing the images on the filesystem directly (not in the database) unless you have a specific reason to store them in the database. It tends to simplify both you storage code and retrieval since you don't need to deal with sending a BLOB to the database. Nginx (https://www.nginx.com/resources/wiki/modules/upload/) for example lets you set file upload location so literally just need to get the filename from the headers and copy/rename it to the location you want to store the file permanently. This also lets you easily remap a URL to the location on the filesystem. The biggest benefit is that it lets the webserver worry about the upload and download and you just have keep track of the filename in your code.
According to your questions, I will give you answer one by one.
nodejs is better as they have more then 15000 modules.
you can do image processing by using nodejs modules like sharp, Jimp and many more.
MongoDB is a document-oriented NoSQL database (Big Data ready). It stores data in JSON-like format and allows users to perform SQL-like queries against it, and nodejs is best for this purpose.
for this there are number of modules for travels from server to client, like webcamjs.
Related
Its known fact that we can use Json/XML parsing or Database to maintain a limited amount of data consistent on a given application and Website at any given time.
However, the dilemma is a project having few display textlines 3k-4k that are supposed to be consistent on both(App and website)on a selected UI, these text files may change at any given point of time. What will be the optimized method or steps of implementation for this technique?
Just my bit would be to store the data in on server as a text file and later use the website or the app to parse the given text file at given location to display it
I am trying to create a website that display Google map mark up with my proprietary data (in form of multiple polygons) on top of it.
I have been studying Google Map API and found the only way to do it is to publish my proprietary data in GeoJSON file then use the following api function to load the data to Google map: map.data.loadGeoJson(xxxxxx);
This means I need to publish my proprietary data by web service in GeoJSON format. However, I don't want users to download my proprietary data and use it for another purpose other than my site. This is similar to a website which allows video streaming but not allowing download the whole video offline.
How can I achieve the purpose? Can I use some language like PHP to generate the map (with markup) at server then send to web client in form of HTML? Or if I cannot achieve this by Google Map API, can other map API support it (like Bing?)
Thanks very much for your help!!!!
Code Mon key
One option is to turn your data into a tile layer. This will limit the user to only seeing an image of the data but would not give them access to the raw data. In a worse case scenario they would only be able to take the images and view the data and not do any kind of analytics against it unless they manually trace all the data.
As an added benefit of rendering the data as a tile layer, you will be able to visualize a lot more data. I've built a few systems that can render 500M rows of polygon data on a map using this approach. The cool thing, if you store the data in a spatial database like SQL Azure, you can easily make your data interactive by taking the point a user clicks on a map and searching the database for any shapes that intersect with that point.
I wrote a simple blog post on how to create a web service that does this many years ago here: https://rbrundritt.wordpress.com/2009/11/26/dynamic-tile-layers-in-the-bing-maps-silverlight-control/
There is also a good open source project here that uses ASP.NET: http://ajaxmapdataconnector.codeplex.com/
I have a whitepaper that is a lot more up to date than my blog post that will be published soon. If you email me at ricky_brundritt at Hotmail.com, I'll send you a draft copy.
I will try to keep this simple without lots of extra information. I have been investigating MongoDB and I believe it will work well for my next project. There is one thing I am fuzzy about though: storing and retrieving files (chunks) from GridFS.
Lets take a CMS for example. If I wanted to display (output via the browser) an image, I would go my MySQL database, find the key and pull the metadata that would include the file source on the File System and then display an image tag with that href. I know that I can store image/video/etc files in Mongo beautifully and I can retrieve binary, but if I wanted to display that file (push it to the browser) would I have to write the contents to a temporary file and then echo my img tag with the href? That can't be more efficient.
I feel like I'm missing something. For this circumstance, is MongoDB any better?
For clarification: I'm using PHP and Apache on a typical LAMP stack (development not production) and working on a platform to enable creative collaboration between artists. So, I would have several artists collaborating on the same files, and I would like to be able to search inside those files, index them, keep all metadata together and employ sharding. I really seems like MongoDB is the way to go.
Thanks!
Michael
If you want to store binary files in the database, you can serve them directly without writing them to a filesystem first. All you have to do is write a PHP script (or whatever) to send the data in response to an incoming HTTP request, with the proper media type in the response HTTP header.
So if we're talking about images, in your HTML you'd do something like this:
<img src="/your_img_script.php?image=1234"></img>
Then just write your_img_script.php to lookup image ID 1234 and dump the data out with a image/jpeg (or whatever) content type header.
I have an iOS app that lists local places in a table view. Each cell has a picture, text, and subtext.
Each cell's detail view also has multiple pictures of the relevant location, as well as a decent amount of text. JSON is the interchange format.
Currently I am sending bit blobs and constructing it into a jpeg once loaded to the device but I am worried this is intensive on both the device and the server. So I was considering sending a link to the picture and asynchronously downloading each picture, but I am unaware of what repercussions this would have. Especially considering that I am currently using a cheap PHP/MySQL shared hosting plan for the backend.
I am looking for a list of pros and cons for sending the raw image data through JSON vs a link to that image. Any other options for quickly and efficiently populating a view with multiple network images is welcome.
I think the difference is as following:
1- the user will download : (link+image > image) more data stream.
2- if the image is on another server -> might be slower than your server or faster -> affect the image loading speed provided for the user and minimize transmitted data size between your server and the client.
3- if the image is on another server -> do you guarantee that it will be there when your website is up ?
4- loading data using ajax is already an asynchronous method and you don't have to worry about another server if you use it. well, unless your server is as slow as hell then you should consider using another server for the big images as the synchronization won't be your major concern as it is the load you are applying to your server.
if other points come to my mind, I'll post them here ..
I've done a little bit of research into this and asked a few colleagues, so I'll share what tidbits I've come up with.
At some point, the raw image data is going to have to be sent- that is unavoidable.
But I can benefit from lazily loading the image data- so that if my user only looks through 14 tableview cells, I only spend time loading 14 images instead of however many total results are returned from the server (And even less if I implement proper caching).
My solution so far is to return 30 (the number of tableview cells I load at one time) JSON objects, each having an "Image_URL":"..." field and putting those into a dictionary. Then, in cellForRowAtIndexPath:, I check to see if the image for that cell is already cached and if not, I make a request for that picture and update the cell.image in the network callback.
This is pretty simple to do on your own, but SDWebImage seems like a pretty good library for handling corner cases, caching, and other things that aren't covered in a basic implementation. I should note that AFNetworking also includes functionality for asynchronous image downloading.
I am building a Flash AS3 application that allows users to modify images, and then submit-save them to a server.
The user will then have the ability to log in and access these saved images in a thumbnail gallery. They can either delete an image or click a thumbnail to view it at original size.
I am comfortable with the front end having built something similar in AS2 and Flash 8 a few years back.
What will be required for the backend?
I assume some type of PHP-MySQL database is needed. Not sure about hosting issues requirements as the AS2 application I built never sent any actual binary data, but rather data describing the image transformations. I assume I will need to make use of byteArray?
Is there an existing tutorial or code sample that does something similar available for viewing-download?
Are there any security restriction 'gotchas' associated with FP9 -10 I need to be aware of?
the most simple way is to create the image in the client ... get a BitmapData snapshot of the image using BitmapData::draw ... convert this to JPEG or PNG using the as3corelib, that offers encoders for both formats ... and then just send the raw binary data to the server (store it into the data property of your URLRequest) and there, store it to the file system (retrieve it in $HTTP_RAW_POST_DATA) ... so the whole storage process is just a couple of lines ...
you will need a database of course, for session management (you could rely on PHPSESSION only, but personally i don't trust it), login, registration and to store which image belongs to which user ...
so yeah, the whole netcode/backend/storage thing etc. will be quite a piece of cake (btw. you might wanna look into amfphp) ... designing a good interface and implementing the galery view etc. will be the biggest chunk i guess ...
there are no real security gotchas, as long as your SWF comes from the same server, that it communicates to ...
so good luck then ... ;)
greetz
back2dos
If you are on a shared host, php and mysql are probably already available to you, that is a good way to get started.
In terms of flash communicating with the server, you will have to find a way to turn your pictures into a stream of bytes (byteArray sure), and then use flash's send() to post them to the server. Sending XML Out From Flash
Using php you can receive the images and save them to the db, and show them (turn the stream of bytes back into an image with gd -- gd docs)
Also: you may not ever have to send a stream of bytes if you can find a way to have flash describe the transformations, and have gd repeat them, just a thought.
Are there any security restriction 'gotchas' associated with FP9 -10 I need to be aware of?
Maybe, if you are posting data to a different server, you have to enable it with some xml Send data from Flash to PHP on a different server