Import path of folders and subfolders into a faster database engine - php

Question

I created a proxy/crawler a while ago and it ended up logging a lot of files. I thought this would be a simple and OK solution to begin with, but realized I came across more and more problems once it came close 1 000 000 files. Searching the database can take up to 15 seconds, and I have experience the server crash twice in the last week. I tested restarting apache2, search for "test" and spam "free -m" command in terminal. I notice the ram went up high imminently, and it's probably the ram that causes crash. I'm not sure what makes a search engine fast, but would really like to know.
All files are stored under:
database/*/*/*.txt
And use this code to go through them all:
$files = array();
$dir = '/var/www/html/database';
foreach (glob($dir . '/*/*/*.txt', GLOB_NOCHECK) as $path) {
$title = basename($path, ".txt");
if(strripos($title,$search) !== false){
array_push( $files, $path );
}
}
The code is much longer, but I just wanted to show the basics of how it works.
Each file contains about 6 lines of useful info.
So I started looking for a solution, and thought. What if I parse the search to something that can search faster than PHP like Java or C? Ahh, it would be a mess.. So I thought about MySQL. But how should I be able to transfer all files from the folders and subfolders to MySQL? Server is running Debian, with 4 GB ram and i3 processor.
I haven't taken any actions yet because MySQL was confusing and hasn't found any other solution. What should I do?

Answer 1

This question is asking for too much. It's not just a click and go. I thought more people had problems like this, but then I realized that everyone are using premade search engines.
I ended up download the whole database to my windows computer, and code a program in c# that automatically goes through all files, gets the content and POST it to an elasticsearch database which I installed on the Debian server. I should probably have created a file to file converter instead of a file to pure POST request.
Drawback of doing this, is that speeds are not too high and it took 2 hours to transfer 700 000 files to the database.
Program will not be released publicly because of specific strings I used in the files. So this was way harder than I expected.
C# Appllication result: