I have a site where we store mostly text. Recently, I have seen huge increase in database size and I was worried, so I started to work on this problem.
After doing some research I came up with a solution that will compress text with lz4 compression algorithm and then store it to the database.
https://github.com/kjdev/php-ext-lz4
Now the problem is: when I save it to the database, it is not saved correctly. For example,"?" signs get added, like here:
?Home - PER'AN and POOR
I have set the table encoding to
utf8mb4_general_ci
So does anyone have an idea how to fix it, or any other suggestion to reduce the database size?
database column type :: text
Related
I am developing a blog, where my client wants to use lot of images, for(articles, titles, advertisement, etc.). he hardly wants any text there, as the blog he wants it to be developed in arabic and he is not happy with any of the supporting font by web browser, nor he wants to adopt the EOT, he will be updating the blog daily (like just uploading the pictures),
what data type do you think i should be using for it? BLOB or VARCHAR ?
PS: i am using MySQL..
Check out the following site which
uses the same concept as of my
clients, although they are rivals,
they have used images for links, news
or advertisements and still the site
is not that heavy..
http://www.sahilonline.net/
Update: I misunderstood that you want to store the text as images. The recommendations below are for storing native text, not image data. I have to agree with #Col that this is a very bad idea - performance-wise, in regards to search engine visibility (no indexing will take place), accessibility for people with visual or other impairments, different screen resolutions, mobile devices... Although I can understand that the selection of browser-available fonts does arabic characters even less justice than they do ours, I would try and get the client away from this idea.
For new projects, definitely VARCHAR or TEXT / LONGTEXT with a UTF-8 character set.
The main reason being that only (VAR)CHAR and TEXT can do fulltext search
How you store your data in the data base has nothing to do with arabic font support or encoding issues. A UTF-8 table can store arabic text without problems.
For some very thorough basic reading on encoding issues, there's Joel Spolsky's famous Unicode article.
mySQL 5 String type overview
Blob stands for "Binary Object" while varchar stands for "variable number of characters", so, the answer is obvious, in my opinion
Though I cannot keep myself from commenting: keeping a blog post title as an image in the database is the most ridiculous solution I have ever seen in my life.
you can save your Image or other Blob object in directory hierarchy and just save the file address in database then you can use Varchar or Text for your field!
I suggest use this method other than save blob object
It will be better if you use blob. As it stores the data in format what you have inserted & displays as it is. also VARCHAR have some limit.
Please do not go for text as you are saying that "hardly wants any text there". It will be better to store Images & any Unicode formatted text in the data type blob.
Just take care of editor that you will be giving to the client for inputting the data, which should support Arabic. Plug-ins for those are available.
I happen to have a database with pictures stored as blob fields. Can't help it, it was the previous developer's choice.
Now I need that data in a new site and the provider won't let me copy the data the easy way (file has become 11Mb big - won't upload that and I don't have shell access).
So I thought I'd write a script that opens a connection in db1, selects all the records, and then copies each into a table in the new db2.
Works all fine if I exclude the blobs. If I want to copy them too, it won't insert.
Anyone had something similar before?
Should I treat the blobs differently when it comes to inserting?
Thanks for any ideas or help.
11MB isn't a huge file, I'm surprised your host has such a low max upload size.
Have you thought about exporting as SQL, splitting the file in two (in Notepad++ or something) then uploading it in smaller sections? Wouldn't take long.
Perhaps check to see if you can increase the max_allowed_packet setting on your mysql DB. I'm not sure if it affects inserts, but I remember having to adjust this setting when I worked on a web-app that allowed users to download 3-5MB binaries from blob fields in the DB.
This link may be helpful, from a quick google search: http://www.astahost.com/info.php/max_allowed_packet-mysql_t2725.html
I want to save an uploaded image to database. Now my question is:
What should be the data type for SQL Server 2000 for image?
a. image
b. varbinary
what is the code for saving image in database?
How to retrieve the image from database and show?
Please refer me any tutorial or guidline. Or if you can please share with me.
Thanks in advance.
The data type should be text because the best way to save an image to a database is to save its path. Let your OS do the job of storing the actual files.
Typically on SQL Server, you would use a BLOB, Binary Large OBject, to store images. We used it for Word documents on a previous project, and it worked just fine. See this article on Database Journal for more info, although a quick Google for the BLOB type will throw up lots more examples.
I wrote this article a while back on this subject. It should help you with #2 and #3. It uses MySQL, but the basics are the same, you just need to replace the MySQL calls with the MSSQL calls.
As to #1, I would go with the obvious choice: "image".
I am not 100% sure of the differences between the two, however. It just seems obvious :)
Edit, according to this, it appears that the image datatype is deprecated. It will be removed in future versions. Not sure how much this affects you, seeing as you are using a 10 year old version, but it is worth keeping in mind.
In SQLServer IMAGE or VARBINARY are allmost the same. IMAGE is 2GB but VARBINARY() needs a length argument.
It's not good idea to store images in a database, the size increases a lot, with each Backup you need to save all images, with increasing size also increases the time to perform a Backup and Restore. You also need to change the network packet size (at the server properties, advanced, network, network packet size) In recents SQLServer versions the most adecuate datatype is varbinary(MAX)
I want to upload a large file of maximum size 10MB to my MySQL database. Using .htaccess I changed PHP's own file upload limit to "10485760" = 10MB. I am able to upload files up to 10MB without any problem.
But I can not insert the file in the database if it is more that 1 MB in size.
I am using file_get_contents to read all file data and pass it to the insert query as a string to be inserted into a LONGBLOB field.
But files bigger than 1 MB are not added to the database, although I can use print_r($_FILES) to make sure that the file is uploaded correctly. Any help will be appreciated and I will need it within the next 6 hours. So, please help!
You will want to check the MySQL configuration value "max_allowed_packet", which might be set too small, preventing the INSERT (which is large itself) from happening.
Run the following from a mysql command prompt:
mysql> show variables like 'max_allowed_packet';
Make sure its large enough. For more information on this config option see
MySQL max_allowed_packet
This also impacts mysql_escape_string() and mysql_real_escape_string() in PHP limiting the size of the string creation.
As far as I know it's generally quicker and better practice not to store the file in the db as it will get massive very quickly and slow it down. It's best to make a way of storing the file in a directory and then just store the location of the file in the db.
We do it for images/pdfs/mpegs etc in the CMS we have at work by creating a folder for the file named from the url-safe filename and storing the folder name in the db. It's easy just to write out the url of it in the presentation layer then.
Some PHP extensions for MySQL have issues with LONGBLOB and LONGTEXT data types. The extensions may not support blob streaming (posting the blob one segment at a time), so they have to post the entire object in one go.
So if PHP's memory limit or MySQL's packet size limit restrict the size of an object you can post to the database, you may need to change some configuration on either PHP or MySQL to allow this.
You didn't say which PHP extension you're using (there are at least three for MySQL), and you didn't show any of the code you're using to post the blob to the database.
The best answer is to use an implementation that is better and also works around that issue.
You can read an article here. Store 10MB, 1000MB, doesn't matter. The implementation chunks/cuts the file into many smaller pieces and stores them in multiple rows.. This helps with load and fetching so memory doesn't also become an issue.
You could use MySQL's LOAD_FILE function to store the file, but you still have to obey the max_allowed_packet value and the fact that the file must be on the same server as the MySQL instance.
You don't say what error you're getting (use mysql_error() to find out), but I suspect you may be hitting the maximum packet size.
If this is the case, you'd need to change your MySQL configuration max_allowed_packet
You don't say what error you're getting (use mysql_error() to find out), but I suspect you may be hitting the maximum packet size.
If this is the case, you'd need to change your MySQL configuration max_allowed_packet
Well I have the same problem. And data cannot be entered in the mysql database chunck by chunck in a "io mode"
loop for :
read $data from file,
write $data to blob
end loop
close file
close blob
A solution seems to create a table with multi-part blobs like
create table data_details
(
id int pk auto_increment,
chunck_number int not null,
dataPart blob
);
???
I'm making a little forum for my clans website. I'm wondering if I should store the thread text in TEXT or BLOB? Whats the difference? I've seen that phpBB does that.
What is BLOB anyway? cant find much about it on Google.
A blob is just a bunch of bytes. An arbitrary number of bytes, nothing more.
If you were to store text as a blob, you'd have to worry about encoding (the process of translating text to bytes). But if you store things as text whatever database transport your using will make sure that the text stored in the database is properly encoded and decoded for both efficient storage and easy use.
If you're planning to store text, you should store text.
phpBB could implement text encoding and decoding themselves and that could be one reason to use blob instead of text. It's unlikely but sometimes text data types have a maximum length, the blob might be a work around for phpBB in this particular instance.
Re the "what" - BLOB is Binary Large OBject; compare to CLOB: Character Large OBject. Different databases call them different things, though - for example, on SQL Server you have image/varbinary(max) for BLOB, and text/varchar(max) for CLOB.
If a system only supports a BLOB, then one option is to encode strings - for example using UTF8. This might be what is happening.
BLOB is for binary data. I don't know the reason why phpBB 3 stores everything in binary but I have noticed it myself. My guess is that they are compressing/encoding whatever they put into the database. You could try looking through the phpBB source code to see if there is any comments explaining it.