Efficient way to write 2^20 files to ext4 filesystem - php

Question

I'm trying to write 2^20 files with about 45k lines in each, each line is a word.
I need this to be flat, no sql for storage purpose, I optimized it not to take too much disk space.
As of now the files are written in 16 directory, making it 65536 file in each directory. The file names are 4 characters long.
That didn't seem too much for me, my script takes a huge file, read each line, and then write each line in its dedicated file.
I first tried to make this with 2^16 files, so 4096 files in each directory, worked like a charm but I wanted to make the lookup faster. It seems to hit a wall at this number of files. Yet from what I saw on the internet, it's not even close the ext4 maximum file number.
After say 45gB it became very very very slow (like wrote 100 lines in each file in more than an hour).
My script is taking advantage of the server 64gB ram by having a buffer, I have 1.7tB SSD drive with more than 700mB writing speed but still looks like it cannot manage that much files. Even a du -h /dir/ takes forever.
Is there any way to do this task faster, or should I just go with 65536 files ?
Thank you for your help :)
Have a great day.
EDIT : Here's the code sorry for the mistake.
Thank you for your answer and sorry I didn't post any code. Here's the code :
$time_pre = microtime(true);
$bufferSize = (1<<9);
$dico = fopen('/path/to/file1',"r+");
$hexa = array("0","1","2","3","4","5","6","7","8","9","a","b","c","d","e","f");
for($i = 0;$i < 16;$i++)
{
for($j = 0;$j < 16;$j++)
{
for($k = 0;$k < 16;$k++)
{
for($l = 0;$l < 16;$l++)
{
for($m = 0;$m < 16;$m++)
{
$ijklm = "$hexa[$i]$hexa[$j]$hexa[$k]$hexa[$l]$hexa[$m]";
$$ijklm = "";
}
}
}
}
}
while(!feof($dico))
{
$ligne = rtrim(fgets($dico));
$md4 = hash("md5",$ligne);
$add4 = "$md4[0]$md4[1]$md4[2]$md4[3]$md4[4]";
$$add4 .= "$md4[5]$md4[6]$ligne\n";
if(isset($$add4{$bufferSize}))
{
$baseFichier = '/path/to/'.$md4[0].'/'.$md4[1].'/'.$md4[2].'/'.$md4[3].'/'.$md4[4];
$fichier = fopen($baseFichier,"a");
fwrite($fichier, $$add4);
fclose($fichier);
$$add4 = "";
}
}
for($i = 0;$i < 16;$i++)
{
for($j = 0;$j < 16;$j++)
{
for($k = 0;$k < 16;$k++)
{
for($l = 0;$l < 16;$l++)
{
for($m = 0;$m < 16;$m++)
{
$ijklm = "$hexa[$i]$hexa[$j]$hexa[$k]$hexa[$l]$hexa[$m]";
if($$ijklm != "")
{
$baseFichier = '/Path/to/'.$hexa[$i].'/'.$hexa[$j].'/'.$hexa[$k].'/'.$hexa[$l].'/'.$hexa[$m];
$fichier = fopen($baseFichier,"a");
fwrite($fichier, $$ijkl);
fclose($fichier);
$$ijklm = "";
}
}
}
}
}
}
fclose($dico);
I used fflush and some locks for the files at first, because I had 6 workers at the same time, to avoid writing problems. But for testing sake I removed it. On this one I tried to create 16^4 directories instead of one directory with 65.536 files in it.
I also tried a smaller buffer, to check if this was better on SSD.
It's running right now but looks like it's slower than previous test. The input file is 75gig, I have 6 like this to process.
What would you do to speed things up ?
I'm using php8. I tried using C but was the same issue so I went with a more high-level language for facility purpose.
Thank you :)
PS : Sorry if the code isn't pro it's a hobbie for me.