Storing huge arrays in the database - php

Question

I am working on an image processing project. I am using php for the GUI and matlab for the algorithm and Mysql as my database. I have 30,000 of images stored in the database for now. My matlab program will generate 3 arrays for each image containing 300 elements. So, my question is
Whether to save that arrays generated from matlab for all images in a single txt file or create a txt file for each image. Which method will be easier to retrieve datas and store into the database?
How hard it is to copy array form a txt file and saving it into the database? Is there any standard process for this?
The elements of the array must be retrieved for further computation. Can we use serialize and deserialize for this purpose?
I have to compare 2 array elements at a time and obtain a third array with the minimum values from both array. Eg A=[1 2 3 4] and B=[6 1 4 2] I have to compare each elements of this 2 array and generate a third array c=[1 1 3 2] that is comparing each elements of the array with its corresponding elements of the other array and storing the minimum element in the third array. This process is repeated with thousands of arrays comparing with 1 fixed array. Is there any php function to do this?
Any suggestions and help will be highly appreciable. Thank you.

Answer 1

The best solution in this case is to create a separate meta table that stores the data(arrays) that relates to your images.
Here's a simple example EER:
The combination of image_id (referencing foreign key), array, and index make up the primary key for the meta table.
I'm assuming you can safely represent the keys of your arrays as just 0-indexed, and all values of your arrays are also numbers (which is why all fields are of type INT, but if not, adjust the datatypes accordingly).
image_id represents the linked image_id.
array represents the specific array (you said up to 3 arrays, so values in this column would be anywhere from 1-3).
index is the numerical index in the array.
value is the value in the array paired with the index.
Example data in your meta table might look like:
image_id | array | index | value
------------------------------------------------------
256 | 1 | 0 | 5
256 | 1 | 1 | 9
256 | 1 | 2 | 4
256 | 1 | 3 | 23
256 | 1 | 4 | 1
256 | 2 | 0 | 9
256 | 2 | 1 | 15
256 | 2 | 2 | 8
256 | 2 | 3 | 19
256 | 2 | 4 | 11
In the above example data, we have two arrays (each represented by 1 and 2 in the array column) with 5 elements each (key represented in the index column, and value represented in the value column.
You can store however many arrays you want, with however many indexes you want.
You can also perform your needed array comparison calculation right in MySQL using GROUP BY. Here is how you can find the minimum value for each key across all arrays for image_id 256 (let's say there are 3 arrays):
SELECT index, MIN(value) AS minvalue
FROM image_meta
WHERE image_id = 256
GROUP BY index
Since a composite index is set up on (image_id, array, index), this should be extremely quick.
This design also allows you to have a variable number of indexes and arrays per image.

Answer 2

Reading through your problems, regardless of my comment, a relational database would be better for you, and if performance is not a big issue, then SQL would be the language to solve your issues best, not on the PHP level.
And yes, PHP has easy ways also to solve those problems, albeit not as easy as in SQL.