I have a very specific scenarion of storing images to mySql DB via PHP. Each image is send to server via POST request in JSON base64 encoded with other attributes and has in average 1MB. Image is written via PHP script to one dedicated table for images as string. Then it is usually read right after because of synchronization. Then it can be read several times in next 24 hours. Up to 20, average 2-3 times. Then image is not read anymore, only in special case it can be read by administrator. There are in average written 100-500 images in 24 hours range. Images older then 1 year are deleted.
In this case does it has benefit to change it and store images in file system ?
a) Store it via PHP script in file system as copy of received base64 data (needs more space)
b) Store it via PHP script in file system as jpg image (needs more performance)
Api via POST/JSON can not be changed
I read a lot of articles about such problems, but I am asking specificaly for my scenario if it is worth to change it, and what will be gained.
There are a lot things you must consider in order to know what is the best solution for you. Both approaches have their pros and cons. You gave to little information about your scenario. What's the average size os those images? What's the database size ?
You should check this thread:
Storing Images in DB - Yea or Nay?
Related
I am working in ionic framework. Currently designing a posts page with text and images. User can post there data and image and all are secure.
So, i use base 64 encoding and save the image in database.
encodeURIComponent($scope.image)
Each time when user request, i select rows from table and display them along with text and decode them.
decodeURIComponent($scope.image)
with HTML "data:image/jpeg;base64,_______" conversion.
Works fine, but take so much time that i expected. Hence, image are 33% bigger size, and totally looks bulgy.
Then i decide to move on file upload plugin of cordova. But i realize, maintain file in this way is so much risk and complected. I also try to save binary data into database. But failed.
Text selecting without base64 data are dramatically reduce time. If it is possible to select image individually in another http call, after selecting other column and display. Is it a right mechanism to handle secure images?
As a rule of thumb, don't save files in the database.
What does the mysql manual have to say about it?
http://dev.mysql.com/doc/refman/5.7/en/miscellaneous-optimization-tips.html
With Web servers, store images and other binary assets as files, with
the path name stored in the database rather than the file itself. Most
Web servers are better at caching files than database contents, so
using files is generally faster. (Although you must handle backups and
storage issues yourself in this case.)
Don't save base4 encoded files in a database at all
Works fine, but take so much time that i expected. Hence, image are
33% bigger size, and totally looks bulgy.
As you discovered, unwanted overhead in encoding/decoing + extra space used up which means extra data transfer back and forth as well.
As #mike-m has mentioned. Base64 encoding is not a compression method. Why use Base64 encoding is also answered by a link that #mike-m posted What is base 64 encoding used for?.
In short there is nothing to gain and much to loose by base64 encoding images before storing them on the file system be it S3 or otherwise.
What about Gzip or other forms of compression without involving base64. Again the answer is that there is nothing to gain and much to lose. For example I just gzipped a 1941980 JPEG image and saved 4000 bytes that's 0.2% saving.
The reason is that images are already in compressed formats. They cannot be compressed any further.
When you store images without compression they can be delivered directly to browsers and other clients and they can be cached. If they are compressed (or base64 encoded) they need to be decompressed by your app.
Modern browsers are able to display base64 images embedded to the HTML but then they cannot be cached and the data is about 30% larger than it needs to be.
Is this an exception to the norm?
User can post there data and image and all are secure.
I presume that you mean a user can download images that belong to him or shared with him. This can be easily achieved by savings the files off the webspace in the file system and saving only the path in the database. Then the file is sent to the client (after doing the required checks) with fpassthru
What about when I grow to a 100000 users
How they take care about images file. In performance issue, when large
user involved, it seams to me, i need 100000 folder for 100000 user
and their sub folder. When large amount of user browse same root
folder, how file system process each unique folder.
Use a CDN or use a file system that's specially suited for this like BTRFS
Database has good searching facility, good thread safe connection, good session management. Is this scenario changed when large operation involved
Yes Indeed. Use it to the fullest by saving all the information about the file and it's file path in the database. Then save the file itself in the file system. You get best of both worlds.
Since it's just personal files, your could store them in S3.
In order to be safe about file uploads, just check the file's mime type before uploading for whatever storage you choose.
http://php.net/manual/en/function.mime-content-type.php
just run a quick check on the uploaded file:
$mime = mime_content_type($file_path);
if($mime == 'image/jpeg') return true;
no big deal!
keeping files on the database is bad practise, it should be your last resource. S3 is great for many use cases, but it's expensive for high usages and local files should be used only for intranets and non-public available apps.
In my opinion, go S3.
Amazon's sdk is easy to use and you get a 1gb free storage for testing.
You could also use your own server, just keep it out of your database.
Solution for storing images on filesystem
Let's say you have 100.000 users and each one of them has 10 pics. How do you handle storing it locally?
Problem: Linux filesystem breaks after a few dozens of thousands images, therefore you should make the file structure avoid that
Solution:
Make the folder name be 'abs(userID/1000)*1000'/userID
That way when you have the user with id 989787 it's images will be stored on the folder
989000/989787/img1.jpeg
989000/989787/img2.jpeg
989000/989787/img3.jpeg
and there you have it, a way of storing images for a million users that doesn't break the unix filesystem.
How about storage sizes?
Last month I had to compress a 1.3 million jpegs for the e-commerce I work on. When uploading images, compress using imagick with lossless flags and 80% quality. That will remove the invisible pixels and optimize your storage. Since our images vary from 40x40 (thumbnails) to 1500x1500 (zoom images) we have an average of 700x700 images, times 1.3 million images which filled around 120GB of storage.
So yeah, it's possible to store it all on your filesystem.
When things start to get slow, you hire a CDN.
How will that work?
The CDN sits in front of your image server, whenever the CDN is requested for a file, if it doesn't find it in it's storage (cache miss) it will copy it from your image server. Later, when the CDN get's requested again, it will deliver the image from it's own cache.
This way no code is needed to migrate to a CDN image deliver, all you will need to do is change the urls in your site and hire a CDN, the same works for a S3 bucket.
It's not a cheap service, but it's waaaaay cheaper then cloudfront and when you get to the point of needing it, you can probably afford it.
I would suggest you to continue with base64 string only, you can use LZ string compression technique to reduce the string size. I've been using and it's working pretty well.
I don't know how am I near to your question, but hope this will help you out.
Here is LZ compression technique : https://github.com/pieroxy/lz-string/
Lets say I have a file called "getimage.php" to obtain an image all we need to do is getimage.php?imgid=1
My Server guy is warning me that pages with multiple images become very processor heavy. So would you suggest making a copies on the file server of theses image? and have a cron job delete files not being used after x amount of time?
or can I some how make 1 MYSQL query to obtain all the images? would this be less heavy then the getimage.php calling 1 SQL per image?
Any other suggestions?
I don't think you can get all the images in one shot from the database. In any case, is there any reason on why you're storing images in the database? This move has its benefits but, on the other hand, it has its drawbacks.
The only huge advantage I personally see from storing images in the database is when these images are subject to permissions (e.g. some users can see those images, some can't).
I have an iOS app that lists local places in a table view. Each cell has a picture, text, and subtext.
Each cell's detail view also has multiple pictures of the relevant location, as well as a decent amount of text. JSON is the interchange format.
Currently I am sending bit blobs and constructing it into a jpeg once loaded to the device but I am worried this is intensive on both the device and the server. So I was considering sending a link to the picture and asynchronously downloading each picture, but I am unaware of what repercussions this would have. Especially considering that I am currently using a cheap PHP/MySQL shared hosting plan for the backend.
I am looking for a list of pros and cons for sending the raw image data through JSON vs a link to that image. Any other options for quickly and efficiently populating a view with multiple network images is welcome.
I think the difference is as following:
1- the user will download : (link+image > image) more data stream.
2- if the image is on another server -> might be slower than your server or faster -> affect the image loading speed provided for the user and minimize transmitted data size between your server and the client.
3- if the image is on another server -> do you guarantee that it will be there when your website is up ?
4- loading data using ajax is already an asynchronous method and you don't have to worry about another server if you use it. well, unless your server is as slow as hell then you should consider using another server for the big images as the synchronization won't be your major concern as it is the load you are applying to your server.
if other points come to my mind, I'll post them here ..
I've done a little bit of research into this and asked a few colleagues, so I'll share what tidbits I've come up with.
At some point, the raw image data is going to have to be sent- that is unavoidable.
But I can benefit from lazily loading the image data- so that if my user only looks through 14 tableview cells, I only spend time loading 14 images instead of however many total results are returned from the server (And even less if I implement proper caching).
My solution so far is to return 30 (the number of tableview cells I load at one time) JSON objects, each having an "Image_URL":"..." field and putting those into a dictionary. Then, in cellForRowAtIndexPath:, I check to see if the image for that cell is already cached and if not, I make a request for that picture and update the cell.image in the network callback.
This is pretty simple to do on your own, but SDWebImage seems like a pretty good library for handling corner cases, caching, and other things that aren't covered in a basic implementation. I should note that AFNetworking also includes functionality for asynchronous image downloading.
Lets say I'm building a image gallery using PHP, where users would be able to upload their photos.
Every user would have 1 folder on server side with all their images there.
Now lets say I need to provide information in browser. Users would be able to browse images and should see lots of information about them, like image size, image dimensions, even EXIF information etc.
I could do this in 2 ways:
Save all information about image into database when uploading image.
Use PHP functions to browse through folder, and get information from every image.
I have something like file manager class, that can do all manipulations with files on server side, like deleteDir, deleteFile, countItems, getFileSize, getDirSize.
And it would be easy to only write one more class that would inspect images, and then I could just upload images, and get their information right from the folders without a need for relation database.
And now the question you all been waiting for is: ... :)
What would be faster, first or second solution? Lets say that site gets loads of traffic.
What solution would be better if I want it to be fast, and not to stress server to much?
actually, I got this situation like yours, this is my solution:
Save all information about image into database when uploading image.
Why?
I tested 2 ways:
Using php to get the image info for 1000 times.
Getting image info from database for 1000 times.
And the result is :
Getting image info from database is faster and faster.
Last but not least:
What would you do if you want to do a image info analystics?
If you save all info in database ,you can easily get them and analyse them ,but if you using php to get the info? it's hard to image.
So, just save all information about image into database when uploading image.
Good luck.
storing it in the database once
reading the data from the database and store it in cache,
redoing things always costs especially if it happens all the time
Depending on the size of these images, you probably want to show thumbnails instead of the original when people are browsing, which means you need to generate them. I would generate the thumbnail on upload and grab all the file info. Then save the file info in the database and put the original and thumbnail in the file system. If you get a lot of traffic, throw memcache on there too.
Storing data in separate places has a way of creating maintenance headache. I would just serialize the metadata for images in each folder and dump it to a file there. If you use gzip compression on the file, retrieval and storage should be very fast.
I'm creating a blog with a featured image on each post. I have a dilemma, I'm unsure what to do with my image data...
Should I insert image data into my MYSQL database using BLOB?
Or should I just create an uploader which makes a directory into the users images folder and upload the photo that way...then just reference it directly in the image field when adding a Blog Post?
Is there a standardised way?
Kind regards,
adam
Upload the files to your server and save the location of the file in your database. Less strain on your DB and your HTTP daemon is better at serving images than MySQL.
The general approach is not to store files in DB, unless you understand why do you need it to be stored there. So, since you are not sure, it's much simplier storing them in upload folder.
But, just in case you decide you need storing files (no matter images or some other) in DB, you have to declare BLOB field and then save it using some BLOB-supporting DB mechanism. 'PHP's MySQLi extension: Storing and retrieving blobs' is a good example of how it can be made
You should store images in folder. Click on below link from where you can get idea how to crop different-different size images and store images name in to database table:
How can I upload images in a normal insert form (MySql)? after upload the image should have three versions of different sizes and different names
convert the image data to base64. This can be done within PHP:
<?
$image=file_get_contents("image.png");
$image=base64_encode($image);
?>
Storing images in a DB is a good idea for secure images.
Always store images, music files etc in system files on disk, and then store the urls to them in the database. That will make it
1) faster
2) easier to configure security settings
3) better in any ways I can imagine
Disadvantage
If file system is corrupted you will have hard time recovering.
You can also use third party Image hosting sites too, you can use Amazon S3 or Mosso Cloud Files.
Problem with file system is it is difficult to scale.
Facebook uses cassandra to store images.
Since it is blog you can store images in filesystem.
Both are valid approaches.
They have different advantages/disadvantages.
Storing it in the database means you need to add extra code to change the image to a representation which will fit inside a INSERT/UPDATE statement (base64 is one approach, and requires equivalent decode, but you could just use mysql_real_escape_string()). Although you can't query the image directly (other than finding exact matches) it may reduce the number of seek and I/O operations required to retrieve the data compared with looking up the path in the database then retrieving the file.
It's also a lot simpler to set up replication of a database compared with setting up replication of the database AND the filesystem if you run on multiple nodes. And there's the issue og keeping filesystem and database backups synchronized.
OTOH, using a filesystem makes your data tables much smaller, and therefore faster to retrieve records from.
which makes a directory into the users images folder
You certainly don't want to allow users to upload content directly into your webserver's document tree - regardless of which route you take, the data should be stored in a location not directly accessible by the webserver but accessible by your code.