Import massive csv data with symfony command too slow with doctrine - php

Question

I need to import a lot of data from a csv file (45 Mo) in myqsl database with Symfony. I imported League\Csv\Reader library
I made a command with doctrine.
It works but I is very slow.
How can I accelerate this ?
I tried to :
adding : $this->em->clear() after $this->em->flush();
adding : //Disable SQL Logging: to avoid huge memory loss.
$this->em->getConnection()->getConfiguration()->setSQLLogger(null);
.
namespace App\Command;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use App\Entity\Developer;
use App\Entity\BadgeLabel;
use Doctrine\ORM\EntityManagerInterface;
use League\Csv\Reader;
class CsvImportCommand extends Command
{
public function __construct(EntityManagerInterface $em){
parent::__construct();
$this->em = $em;
}
// the name of the command (the part after "bin/console")
protected static $defaultName = 'app:import-developpers';
protected function configure()
{
$this
// the short description shown while running "php bin/console list"
->setDescription('Import a new developper.')
// the full command description shown when running the command with
// the "--help" option
->setHelp('This command allows you to import a develpper...')
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
$io = new SymfonyStyle($input, $output);
$io->title('Importation en cours');
$reader = Reader::createFromPath('%kernel.root_dir%/../src/Data/developers_big.csv')
->setHeaderOffset(0)
;
$results = $reader->getrecords();
$io->progressStart(iterator_count($results));
//Disable SQL Logging: to avoid huge memory loss.
$this->em->getConnection()->getConfiguration()->setSQLLogger(null);
foreach ($results as $row) {
$developer = $this->em->getRepository(Developer::class)
->findOneBy([
'firstName' => ($row['FIRSTNAME']),
'lastName'=> ($row['LASTNAME'])
])
;
if (null === $developer) {
$developer = new developer;
$developer
->setFirstName($row['FIRSTNAME'])
->setLastName($row['LASTNAME']);
$this->em->persist($developer);
$this->em->flush();
$this->em->clear();
}
$badgeLabel = $this->em->getRepository(BadgeLabel::class)
->findOneBy([
'name' => ($row['BADGE LABEL']),
'level'=> ($row['BADGE LEVEL'])
])
;
if (null === $badgeLabel) {
$badgeLabel = new BadgeLabel;
$badgeLabel
->setName($row['BADGE LABEL'])
->setLevel($row['BADGE LEVEL']);
$this->em->persist($badgeLabel);
$this->em->flush();
$this->em->clear();
}
$developer
->addBadgeLabel($badgeLabel);
$io->progressAdvance();
}
$this->em->flush();
$this->em->clear();
$io->progressFinish();
$io->success('Importation terminée avec succès');
}
}
The command works put its to slow. After 15 min, only 32% was updload in my Mysql database. I Expected it in 2 minutes max

Answer 1

Method1: (not the best)
When flush method is called, Symfony go throught all listeners. So, you could avoid to flush on each loop. You can replace each flush by this code:
if (0 === ($batchSize++ % $input->getOption('fetch'))) {
$this->entityManager->flush();
$this->entityManager->clear();
}
fetch option can be declared in configure method:
const BATCH_SIZE = 1000; // As example
/**
* Configure the command.
*/
protected function configure()
{
$this
// the short description shown while running "php bin/console list"
->setDescription('Import a new developper.')
//This option helps you to find a good value and use BATCH_SIZE constant as default
->addOption('fetch', 'f', InputArgument::OPTIONAL, 'Number of loop between each flush', self::BATCH_SIZE)
// the full command description shown when running the command with
// the "--help" option
->setHelp('This command allows you to import a develpper...')
;
Method2: More efficient
You can create a command which writes all SQL queries with update or insert in a sql file. Then, you launch a native command that read the files and execute queries.
Method3: Using DBAL
As suggested in comments, youcould use DBAL to avoid unnecessary object hydration with Doctrine.