Optimize Import CSV data into neo4j - php

Question

I wrote an import snippet to populate my Neo4J DB with nodes for towns and related to them counties. The code looks like
<?php
function readCSV($csvFile){
$file_handle = fopen($csvFile, 'r');
$lineCount=0;
while (!feof($file_handle) ) {
$line_of_text[] = fgetcsv($file_handle, 1024, ';', '"');
$lineCount++;
}
fclose($file_handle);
return array($line_of_text,$lineCount);
}
// Create an Index for Town and for Country
$queryString = '
CREATE INDEX ON :Country (name)
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString);
$result = $query->getResultSet();
$queryString = '
CREATE INDEX ON :Town (name)
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString);
$result = $query->getResultSet();
// Set path to CSV file
$importFile = 'files/import_city_country.csv';
$completeResult = readCSV($importFile);
$dataFile = $completeResult[0];
$maxLines = $completeResult[1];
for ($row = 1; $row < $maxLines; ++ $row) {
$countryData = array();
if(!is_null($dataFile[$row][0]))
{
// Define parameters for the queries
$params =array(
"nameCountry" => trim($dataFile[$row][0]),
"nameTown" => trim($dataFile[$row][1]),
"uuid" => uniqid(),
);
# Now check if we know that country already to avoid double entries
$queryString = '
MATCH (c:Country {name: {nameCountry}})
RETURN c
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString,$params);
$result = $query->getResultSet();
if(COUNT($result)==0) // Country doesnt exist!
{
$queryString = '
MERGE (c:Country {name: {nameCountry}} )
set
c.uuid = {uuid},
c.created = timestamp()
RETURN c
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString,$params);
$result = $query->getResultSet();
}
# Now check if we know that town already
$queryString = '
MATCH (t:Town {name: {nameTown}})
RETURN t
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString,$params);
$result = $query->getResultSet();
if(COUNT($result)==0) // Town doesnt exist!
{
$queryString = '
MERGE (t:Town {name: {nameTown}} )
set
t.created = timestamp()
RETURN t
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString,$params);
$result = $query->getResultSet();
// Relate town to country
$queryString = '
MATCH (c:Country {name: {nameCountry}}), (t:Town {name: {nameTown}})
MERGE (t)-[:BELONGS_TO]->(c);
';
$query = new Everyman\Neo4j\Cypher\Query($client, $queryString,$params);
$result = $query->getResultSet();
}
} // Excel Last Line is not Null - go on
} // Next Row
?>
A typical CSV line looks like
Country City
Albania Tirana
This all works fine - but it takes more than 30 minutes on a pc to import 9.000 lines. I know the system needs to check each record if already existing and also do a relation between town and country but it seems quite long though for such an amount of CSV-lines.
Do you have maybe suggestions how to improve the import code?
Thanks,
Balael
BTW: Any chance to insert here code without editing every row and adding 4 spaces - kinda boring for longer code.....

Answer 1

Use LOAD CSV inside of the neo4j-shell if at all possible, and don't write your own code to process the CSV.
What this will do for you primarily is to allow you to batch many items into a single transaction USING PERIODIC COMMIT.
If you want to use the REST API remotely (as I assume you're doing here) then look into your language binding's support for batch operations. Your code as written is going to spend a lot of time going back and forth to the server, probably turning each line of the CSV into a request/response, which will be slow. Better to batch up many at a time and run them as one operation, which will help minimize how much protocol overhead you have.