PHP loading structed files faster - php

Question

I have multiple dataset as below, and I want to handle them using PHP.
Dataset #1 (75 cols * 27,000 rows)
col #1 col #2 ...
record #1
record #2
...
Dataset #2 (32 cols * 7,500 rows)
....
Dataset #3 (44 cols * 17,500 rows)
....
Here, the number of records and columns are different so it is hard to use database structure.
And note that each 'cell' of dataset is only consists of either real number or N/A... and the dataset is perfectly fixed, i.e., there will be no any change.
So what I've done so far is make them as a file-based table, and write a starting offset of each record in the file.
Using this way, quite nice access speedup was achieved, but not satisfactory so far, because an access to each record requires parsing it as PHP data structure.
What I ultimately want to achieve is eliminating the parsing step. But serialization was not a good choice because it loads entire dataset. Of course it is possible to serialize each record and keep their offset as I've done but without serialization, but it seems me to not so fancy.
So here's the question, is there any method to load a part of dataset without any parsing step, but more better than the partial serialization what I suggested?
Many thanks in advance.
More information
Maybe I made the viewers a little bit confused.
Each dataset is separated and they exist as independent files.
Usual data access pattern is row-wise. Each row have unique ID by string, and an ID in one dataset could be exists in other dataset, but not necessarily. But above of that, what I concern is accelerating an access speed when I have some query to fetch specific row(s) in the dataset. For example, let there is a dataset like below.
Dataset #1 (plain-text file)
obs1 obs2 obs3 ...
my1 3.72 5.28 10.22 ...
xu1 3.44 5.82 15.33 ...
...
qq7 8.24 10.22 47.54 ...
And there is a corresponding index file, serialized using PHP. A key of each item represents unique ID in the dataset, and their value represents their offset in the file
Index #1 (PHP-serialized one, not same as actual serialized one)
Array (
"my1" => 0,
"xu1" => 337,
...
"qq7" => 271104
)
So it is possible to know record "xu1" starts at 337 bytes from the beginning of dataset file.
In order to access and fetch some rows using their unique ID,
1) Load serialized index file
2) Find matching IDs with query
3) Access to those position and fetch rows, and parsing them as an array of PHP.
The problems what I have is
1) Since I using exact matching, it is impossible to fetch multiple rows that partially matching with query (for example, fetch "xu1" row from query "xu")
2) Even though I indexed dataset, fetch speed is not satisfactory (took 0.05 sec. from single query)
3) When I tried to solve above problem by serializing an entire dataset, (maybe of course) the loading speed become substantially slower.
The only easiest way to solve above problems is make them as database I would do so,
but hope to find better way as keep them with plain text or some text-like format (for example, serialized or json-coded).
Many thanks and interests about my problem!

Answer 1

I think I understand your question to some extent. You've got 3 sets of data, that can be or cannot be related, with different number of columns and rows.
This may not the most cleanest looking solution, but I think it could solve the purpose. You can use mysql to store the data to avoid parsing the file every now and again. You can store the data in three tables or put them in one table with all the columns (the rows without need for a set column can have "null" for the field value).
You can also use sql unions, in case you want to run queries on all the three datasets collectively, by using tricks like
select null as "col1", col2, col3 from table1 where col2="something"
union all
select col1,null as "col2", null as "col3" from table2 where co1="something else"
order by col1