I want to import data using php mysql from excel sheet containing 1.6 million records. What is the best way to do this?
this is the sample code I have used to iterate excel file and insert data in database:
public function iterateData($file_name) {
$fileDirectory = '';
$file_name = $fileDirectory . $file_name;
if (file_exists($file_name)) {
$this->truncateTable();
include 'PHPExcel2/Classes/PHPExcel/IOFactory.php';
$objReader = PHPExcel_IOFactory::createReader('Excel2007');
$objPHPExcel = $objReader->load($file_name);
$count = 1;
foreach ($objPHPExcel->getWorksheetIterator() as $worksheet) {
foreach ($worksheet->getRowIterator() as $row) {
$cellIterator = $row->getCellIterator();
$cellIterator->setIterateOnlyExistingCells(true); // Loop all cells, even if it is not set
$cellValues = array();
foreach ($cellIterator as $cell) {
if (!is_null($cell)) {
$cellValues[] = $cell->getCalculatedValue();
}
}
if (isset($cellValues[0]) && $cellValues[0] != 'Product' && $cellValues[0] != '') {
$this->inserInDatabase($cellValues);
} elseif (empty($cellValues[0]) && empty($cellValues[1]) && empty($cellValues[2])) {
continue;
}
}
if ($objPHPExcel->getSheetCount() == $count) {
return TRUE;
}
$count++;
}
} else {
return FALSE;
}
} private function inserInDatabase($data) {
$dbDetails = array(
'db_name' => '*',
'db_pass' => '*',
'db_host' => 'localhost',
'db_user' => '*'
);
$dbh = dbConnect::connect($dbDetails);
$date = date('Y-m-d H:i:s');
$sql = "INSERT INTO product_description (product_id, prpoduct_description, price, created_date) values ('" . mysql_escape_string($data[0]) . "', '" . mysql_escape_string($data[1]) . "', '" . mysql_escape_string($data[2]) . "', '$date')";
if (!$dbh->dbh->query($sql)) {
die('Database Connection Failed.');
}
}
export you excel data to csv format, and then import the csv format to mysql
you can import using ;
if($request->hasFile('excelFile')){
$inputFileType = 'Xlsx';
$inputFileName = $request->file('excelFile')->getRealPath();
/** Create a new Reader of the type defined in $inputFileType **/
$reader = \PhpOffice\PhpSpreadsheet\IOFactory::createReader($inputFileType);
/** Advise the Reader that we only want to load cell data **/
$reader->setReadDataOnly(true);
/** Load $inputFileName to a Spreadsheet Object **/
$spreadsheet = $reader->load($inputFileName);
foreach ($spreadsheet->getActiveSheet()->toArray() as $key => $row) {
$data['question'] = $row[0];
$data['option1'] = $row[1];
$data['option2'] = $row[2];
$data['option3'] = $row[3];
$data['option4'] = $row[4];
$data['correct'] = $row[5];
$data['status'] = 1;
$data['receiver'] = 'all';
$data['createdOn'] = date("Y-m-d H:i:s");
if(!empty($data)) {
DB::table('questions')->insert($data);
}
}
}
Related
Is it possible by code to skip the Header (Top Row) in Excel Spreadsheet?
I'm using PHPExcel_Reader to process upload into database.
this image is my data excel
this is my code:
<?php require_once('../../php-excel-reader/excel_reader2.php'); require_once('../../SpreadsheetReader.php'); if (isset($_POST["import"])) { $allowedFileType = ['application/vnd.ms-excel','text/xls','text/xlsx','application/vnd.openxmlformats-officedocument.spreadsheetml.sheet']; if(in_array($_FILES["file"]["type"],$allowedFileType)){ $targetPath = '../assets/uploads/'.$_FILES['file']['name']; move_uploaded_file($_FILES['file']['tmp_name'], $targetPath); $Reader = new SpreadsheetReader($targetPath); $sheetCount = count($Reader->sheets()); for($i=0;$i<$sheetCount;$i++) { $Reader->ChangeSheet($i); foreach ($Reader as $Row) { $student_id = ""; if(isset($Row[0])) { $student_id = mysqli_real_escape_string($conn,$Row[0]); } $roll_no = ""; if(isset($Row[1])) { $roll_no = mysqli_real_escape_string($conn,$Row[1]); } $student_name = ""; if(isset($Row[2])) { $student_name = mysqli_real_escape_string($conn,$Row[2]); } $class_name = ""; if(isset($Row[3])) { $class_name = mysqli_real_escape_string($conn,$Row[3]); } ?>
I use this method for skip header :
$spreadsheet = \PhpOffice\PhpSpreadsheet\IOFactory::load('FILE.xlsx');
$sheetData = array();
foreach ($spreadsheet->getWorksheetIterator() as $worksheet) {
$worksheetTitle = $worksheet->getTitle();
$highestRow = $worksheet->getHighestRow(); // e.g. 10
$highestColumn = $worksheet->getHighestColumn(); // e.g 'F'
$highestColumnIndex = \PhpOffice\PhpSpreadsheet\Cell\Coordinate::columnIndexFromString($highestColumn);
for ($row = 1; $row <= $highestRow; ++$row) {
for ($col = 0; $col <= $highestColumnIndex; ++$col) {
$cell = $worksheet->getCellByColumnAndRow($col, $row);
$val = $cell->getValue();
$sheetData[$row][$col] = $val;
}
}
}
unset($sheetData[1]); // SKIP HEADER
foreach ($sheetData as $val) {
// set data for upload DB
}
i just unset that from array.
I am trying to populate a mysql database with an excel file using phpspreadsheet library. I am doing it in the following way but I get just the first row. How can I do it for all the rows
$spreadsheet = \PhpOffice\PhpSpreadsheet\IOFactory::load($target_file);
$worksheet = $spreadsheet->getActiveSheet();
$rows = [];
$outer = 1;
foreach ($worksheet->getRowIterator() AS $row) {
$cellIterator = $row->getCellIterator();
$cellIterator->setIterateOnlyExistingCells(FALSE); // This loops through all cells,
$cells = [];
foreach ($cellIterator as $cell) {
$cells[] = $cell->getValue();
}
$rows[] = $cells;
while($outer > 1){
$data = [
'testTaker' => $cells[1],
'correctAnswers' => $cells[2],
'incorrectAnswers' => $cells[3],
];
if($this->testModel->addTest($data)){
die('it worked');
} else {
die('Something went wrong');
}
}
$outer++;
}
This is how I am importing a XLSX spreadsheet with phpSpreadSheet into a MySQL database using PDO (modified to fit your criteria).
// read excel spreadsheet
$reader = new \PhpOffice\PhpSpreadsheet\Reader\Xlsx();
if($reader) {
$reader->setReadDataOnly(true);
$spreadsheet = $reader->load($target_file);
$sheetData = $spreadsheet->getActiveSheet()->toArray();
foreach($sheetData as $row) {
// get columns
$testTaker = isset($row[0]) ? $row[0] : "";
$correctAnswers = isset($row[1]) ? $row[1] : "";
$incorrectAnswers = isset($row[2]) ? $row[2] : "";
// insert item
$query = "INSERT INTO item(testTaker, correctAnswers, incorrectAnswers) ";
$query .= "values(?, ?, ?)";
$prep = $dbh->prepare($query);
$prep->execute(array($testTaker, $correctAnswers, $incorrectAnswers));
}
}
I'm working on phpexcel. The File I'm working on is from tally export file. I have to read this file and save the data to each user .
For example from this file I need only A2:G10 as HTML/TABLE. So i can display to the particular member (Adv. Chandra Mogan) individually.
I NEED A TABLE FOR ONLY A PORTION OF THE SHEET
What I have done so far:
protected function doExcelUpdate() {
$inputFileName = $this->getParameter('temp_directory') . '/file.xls';
if (!file_exists($inputFileName)) {
$this->addFlash('sonata_flash_error', 'File: not found in temp directory');
return;
}
$this->addFlash('sonata_flash_info', 'File: exist');
try {
$inputFileType = PHPExcel_IOFactory::identify($inputFileName);
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
$objPHPExcel = $objReader->load($inputFileName);
} catch (Exception $e) {
$this->addFlash('sonata_flash_error', 'Error in PHPExcel');
return;
}
$sheet = $objPHPExcel->getSheet(0);
if (!$sheet) {
$this->addFlash('sonata_flash_error', 'Error in reading sheet');
return;
}
$objPHPExcel->getSheet(0)
->getStyle('A1:G10')
->getProtection()
->setLocked(
PHPExcel_Style_Protection::PROTECTION_PROTECTED
);
$objWriter = PHPExcel_IOFactory::createWriter($objPHPExcel, 'HTML');
$objWriter->setSheetIndex(0);
$objWriter->save($this->getParameter('temp_directory') . '/output.html');
}
A1:G10 is not locked. Entire sheet is printed.
First point: "Locking" doesn't change the sheet size, or set a "view window"; it protects parts of the sheet from being edited.
Second point: "Locking" is a feature of Excel, and supported for excel writers, but not for HTML.
Third point: there is no direct mechanism to write only part of a worksheet.
As a suggestion, you might create a new blank worksheet, and then copy the data/style information from the cell range that you want in your your main worksheet to that new worksheet starting from cell A1; then send that worksheet to the HTML Writer.
You can't achive this using locked or hidden cells when you use HTML writer.
You can use a workaround creating a new worksheet and after adding the portion you want to display.
For mantain style on new worksheet (like font, color, border) you must extract it for each cell from the orginal worksheet and apply to the copied cells.
The same thing is for merged cells in the old worksheet.
Your function doExcelUpdate() should be like this:
protected function doExcelUpdate()
{
$inputFileName = $this->getParameter('temp_directory').'/file.xls';
if (!file_exists($inputFileName)) {
$this->addFlash('sonata_flash_error', 'File: not found in temp directory');
return;
}
$this->addFlash('sonata_flash_info', 'File: exist');
try {
$inputFileType = PHPExcel_IOFactory::identify($inputFileName);
$objReader = PHPExcel_IOFactory::createReader($inputFileType);
$originalPHPExcel = $objReader->load($inputFileName);
} catch (Exception $e) {
$this->addFlash('sonata_flash_error', 'Error in PHPExcel');
return;
}
$originalSheet = $originalPHPExcel->getSheet(0);
if (!$sheet) {
$this->addFlash('sonata_flash_error', 'Error in reading sheet');
return;
}
// Get the data of portion you want to output
$data = $originalSheet->rangeToArray('A1:G11');
$newPHPExcel = new PHPExcel;
$newPHPExcel->setActiveSheetIndex(0);
$newSheet = $newPHPExcel->getActiveSheet();
$newSheet->fromArray($data);
// Duplicate style for each cell from original sheet to the new sheet
for ($i = 1; $i < 11; $i++) {
for ($j = 0; $j <= 6; $j++) {
$style = $originalSheet->getStyleByColumnAndRow($j, $i);
$newSheet->duplicateStyle($style, PHPExcel_Cell::stringFromColumnIndex($j).(string)$i);
}
}
// Merge the same cells that are merged in the original sheet
foreach ($originalSheet->getMergeCells() as $cells) {
$inRange = false;
foreach (explode(':', $cells) as $cell) {
$inRange = $originalSheet->getCell($cell)->isInRange('A1:G11');
}
// Merge only if in range of the portion of file you want to output
if ($inRange) {
$newSheet->mergeCells($cells);
}
}
$objWriter = PHPExcel_IOFactory::createWriter($newPHPExcel, 'HTML');
$objWriter->save($this->getParameter('temp_directory').'/output.html');
}
UP TO NOW FOR THE WORK TO BE COMPLETED, I HAVE DONE THIS,
private function doExcelUpdate() {
$inputFileName = $this->getParameter('temp_directory') . '/file.xls';
$synopsis = PHPExcel_IOFactory::load($inputFileName)->getSheet(0);
$column = $synopsis->getHighestColumn();
$row = $synopsis->getHighestRow();
$this->cleanUserPayment();
$this->doExcelUpdateTable($synopsis, $column, $row);
$this->deleteExcelFile();
}
private function cleanUserPayment() {
$em = $this->getDoctrine()->getManager();
$classMetaData = $em->getClassMetadata('AppBundle\Entity\UserPayment');
$connection = $em->getConnection();
$dbPlatform = $connection->getDatabasePlatform();
$connection->beginTransaction();
try {
$connection->query('SET FOREIGN_KEY_CHECKS=0');
$q = $dbPlatform->getTruncateTableSql($classMetaData->getTableName());
$connection->executeUpdate($q);
$connection->query('SET FOREIGN_KEY_CHECKS=1');
$connection->commit();
} catch (\Exception $e) {
$connection->rollback();
}
}
private function doExcelUpdateTable($synopsis, $column, $row) {
set_time_limit(300);
$t = [];
for ($r = 1; $r <= $row; $r++) {
for ($c = "A"; $c <= $column; $c++) {
$cell = $synopsis->getCell($c . $r)->getFormattedValue();
if ($cell == 'Ledger:') {
$t[] = $r;
}
}
}
$t[] = $row+1;
$numItems = count($t);
$i = 0;
$em = $this->getDoctrine()->getManager();
foreach ($t as $key => $value) {
if (++$i != $numItems) {
$up = new UserPayment();
$up->setName($synopsis->getCell('B' . $value)->getFormattedValue());
$up->setMessage($this->doExcelUpdateTableCreate($synopsis, $column, $value, $t[$key + 1]));
$em->persist($up);
// $this->addFlash('sonata_flash_error', 'Output: ' . $synopsis->getCell('B' . $value)->getFormattedValue() . $this->doExcelUpdateTableCreate($synopsis, $column, $value, $t[$key + 1]));
}
}
$em->flush();
$this->addFlash('sonata_flash_success', "Successfully updated user bills. Total data updated::" . count($t));
}
private function doExcelUpdateTableCreate($synopsis, $column, $rowS, $rowE) {
$mr = NULL;
$x = 0;
$alphas = range('A', $column);
$oneTable = '<table border="1">';
for ($r = $rowS; $r < $rowE; $r++) {
$oneTable .= "<tr>";
for ($c = "A"; $c <= $column; $c++) {
if ($x > 0) {
$x--;
continue;
}
$mr = NULL;
$x = 0;
$cell = $synopsis->getCell($c . $r);
$cellVal = $cell->getFormattedValue();
if ($cellVal == NULL) {
$cellVal = " ";
}
$cellRange = $cell->getMergeRange();
if ($cellRange) {
$mr = substr($cellRange, strpos($cellRange, ":") + 1, 1);
$upto = array_search($mr, $alphas);
$x = ($upto - array_search($c, $alphas));
$oneTable .= "<td colspan=" . ($x + 1) . " style='text-align:right;'>" . $cellVal . "</td>";
} else {
$oneTable .= "<td>" . $cellVal . "</td>";
}
}
$oneTable .= "</tr>";
}
$oneTable .= "</table>";
return $oneTable;
}
private function deleteExcelFile() {
$filesystem = new \Symfony\Component\Filesystem\Filesystem();
$filesystem->remove($this->getParameter('temp_directory') . '/file.xls');
}
I am importing some data into a mysql table from an xls file.
On the data, i want to use the mysqli_real_escape_string function, before i insert it into the sql table.
My question is that: where should i put the array_map with the escape function in this code below?
Thanks for help.
if(isset($_POST['submitButton']))
{
if($_FILES['file']['size'] != 0 )
{
if($_FILES["file"]["size"] > 5242880 ) { $error[] = "A fájl mérete maximum 5 MB lehet."; }
$filename = $_FILES['file']['name'];
$ext = pathinfo($filename, PATHINFO_EXTENSION);
if(!array_key_exists($ext, $fajl_types)) { $error[] = "Nem engedélyezett fájl típus."; }
if(count($error) == 0 )
{
$path = "../imports/" . date( "Y-m-d" ) . '-' . rand(1, 9999) . '-' . $_FILES['file']['name'];
if(move_uploaded_file($_FILES['file']['tmp_name'], $path ))
{
$file_name = basename($path);
$objPHPExcel = PHPExcel_IOFactory::load('../imports/'.$file_name);
$dataArr = array();
foreach($objPHPExcel->getWorksheetIterator() as $worksheet) {
$worksheetTitle = $worksheet->getTitle();
$highestRow = $worksheet->getHighestRow(); // e.g. 10
$highestColumn = $worksheet->getHighestColumn(); // e.g 'F'
$highestColumnIndex = PHPExcel_Cell::columnIndexFromString($highestColumn);
for ($row = 1; $row <= $highestRow; ++ $row) {
for ($col = 0; $col < $highestColumnIndex; ++ $col) {
$cell = $worksheet->getCellByColumnAndRow($col, $row);
$val = $cell->getValue();
$dataArr[$row][$col] = $val;
}
}
}
unset($dataArr[1]); // since in our example the first row is the header and not the actual data
$user_pass = "";
$user_reg_date = date("Y-m-d-H:i:s");
$user_last_login = "";
$user_aktivation = "";
$user_vevocsoport = (int)0;
$user_newpass = "";
$user_imported = (int)1;
foreach($dataArr as $val)
{
$sql = "INSERT INTO user
(
user_vnev,
user_knev
)
VALUES
(
'".$val['0']."',
'".$val['1']."'
)";
$import = mysqli_query($kapcs, $sql) or die("IMPORT-ERROR - " . mysqli_error($kapcs));
$ok = 1;
}
}
else
{
$error[] = "A fájl feltöltése nem sikerült.";
}
}
}
else
{
$error[] = "Nem választott ki fájlt.";
}
}
You can add it when building the query. Like this, the escaping and building the query, which are both related to each other, are close to each other in your code.
foreach($dataArr as $val)
{
$escapedVals = array_map(function($value) use ($kapcs) {
return mysqli_real_escape_string($kapcs, $value);
}, array_slice($val, 0, 2));
$sql = 'INSERT INTO user
(
user_vnev,
user_knev
)
VALUES
(
"' . implode ('","', $escapedVals) . '"
)';
$import = mysqli_query($kapcs, $sql) or die("IMPORT-ERROR - " . mysqli_error($kapcs));
$ok = 1;
}
If I have 14000 records then there is memory space issue coming and if records are 1000 I am able to upload but takes 00:30 mints .
Can anyone help me to optimize this code that I can upload records in minimum time?
Here is my code:
<?php
namespace ProjectName\AdminBundle\Controller;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\RedirectResponse;
class TakeoverController extends Controller {
public function uploadtakeoverAction(Request $request) {
ini_set('max_execution_time', 0); //0=NOLIMIT
ini_set('memory_limit', '8192M');
//start - code to check ip address
$checkipaddress = $this->get('userlogin')->CheckIPAddress();
if (empty($checkipaddress)) {
return $this->redirect($this->generateUrl('admin'));
}
//end - code to check ip address
$em = $this->getDoctrine()->getManager();
$helper = $this->get('common_helper_function');
$reqImage = $_FILES;
$validExtensions = array('xls', 'xlsx');
if (isset($_FILES) && !empty($_FILES)) {
$ext = pathinfo($_FILES['file']["name"], PATHINFO_EXTENSION);
if (!in_array($ext, $validExtensions)) {
echo "1";
die;
}
}
$admin = $this->get('security.context')->getToken()->getUser();
$takeoverfile = $this->uploadtakeoverFile($reqImage);
$takeovertm = array();
$insertedrecord = array();
$existingrecord = array();
$importmulitpletakeover = array();
$atmservicesamount = array();
$errormessage = '';
$neeerro = '';
if ($takeoverfile) {
$phpExcelObject = $this->get('phpexcel')->createPHPExcelObject();
$file = $takeoverfile;
if (!file_exists($file)) {
exit("Please run 05featuredemo.php first.");
}
$objPHPExcel = \PHPExcel_IOFactory::load($file);
foreach ($objPHPExcel->getWorksheetIterator() as $worksheet) {
$i = 0;
foreach ($worksheet->getRowIterator() as $row) {
$cellIterator = $row->getCellIterator();
$cellIterator->setIterateOnlyExistingCells(false); // Loop all cells, even if it is not set
$totalindex = $row->getRowIndex();
// echo $row->getRowIndex();
if ($row->getRowIndex() != 1) { // first index is title index so not cosider
foreach ($cellIterator as $cell) {
if (!is_null($cell)) {
//the below columns are from A to AF
if ($cell->getColumn() == 'A') {
$val1= trim($cell->getValue());
}
if ($cell->getColumn() == 'B') {
$val2= $cell->getValue();
}
if ($cell->getColumn() == 'C') {
$val3= trim($cell->getValue());
}
if ($cell->getColumn() == 'D') {
$val3.=', ' . trim($cell->getValue());
}
if ($cell->getColumn() == 'E') {
$val3.=', ' . trim($cell->getValue());
}
}
// insert this fields into database from here of column A to AF
}
unlink($takeoverfile);
$response = new Response(json_encode($importmulitpletakeover));
return $response;
} else {
$message = 'Image has not been uploaded ';
echo "2";
die;
}
}
}}}
protected function uploadtakeoverFile($uploadedReceipt) {
//start - code to check ip address
$checkipaddress = $this->get('userlogin')->CheckIPAddress();
if (empty($checkipaddress)) {
return $this->redirect($this->generateUrl('admin'));
}
//end - code to check ip address
$basicPath = $_SERVER['DOCUMENT_ROOT'] . '/ProjectName/web/uploads';
$basicReceiptPath = $basicPath . '/FolderName';
if (null === $uploadedReceipt) {
return;
} else {
$extention = explode('.', #$uploadedReceipt['file']["name"]);
$filename = #$extention[0] . '_' . time() . '.' . #$extention[1];
$this->path = $basicReceiptPath . '/' . #$filename;
if (move_uploaded_file($uploadedReceipt['file']["tmp_name"], $this->path))
return $this->path;
else
return false;
}
}
}
The most effective style will be to only use Doctrine DBAL and write SQL code.
Documentation states that Doctrine ORM is not fit for mass inserts.
There are "hacks" like disabling SQLLogger and batch processing, but using only DBAL will save you more time.