SQL query via Medoo taking too long to finish - php

My script allows you to upload a zip file and then inserts the individual file information into a database using Medoo, after extracting them. My script takes way too long to finish, even after I've set the max execution time to 5 minutes I get the notice saying the max execution time has been exceeded.
There are only about 650 files in the zips that will be upload-able and the script only manages to extract and insert about half in to the DB before it times out. Is this query more memory intensive than I realize?
EDIT: I should mention that it only hangs with zip files with a larger amount of files, like the 650 figure I mentioned above, the program seems to execute fine with a small number of files.
Code (Offending query near bottom of script):
<?php
ini_set('max_execution_time', 300);
require_once 'vendor/medoo.min.php';
require_once 'scripts/class.file.php';
$database = new medoo([
'database_type' => 'mysql',
'database_name' => 'invoice_files',
'server' => 'localhost',
'username' => 'root',
'password' => 'pass',
'charset' => 'utf8'
]);
$file = new File();
$file->set("filename", $_FILES['uploaded-file']['name']);
$file->set("category", "Invoice Statement");
$file->set("file_temp_path", $_FILES["uploaded-file"]["tmp_name"]);
$file->set("uploadedFilePath", $file->path("uploads/") . basename($file->get("filename")));
$counter = 0;
if($file->getPathInfo()["extension"] == "zip")
{
$zip = new ZipArchive;
$zipFile = $file;
echo "Source: " . $zipFile->get("file_temp_path") . "<br>";
if($zip->open($zipFile->get("file_temp_path")))
{
for($i = 0; $i < $zip->numFiles; $i++)
{
$zipName = $zip->getNameIndex($i);
$zipFile->set("uploadedFilePath", $file->path("uploads/"));
$zipFile->set("filename", $zipName);
for($x = 0; $x < $zip->numFiles; $x++)
{
$extension = $zip->getNameIndex($x);
$pathInfo = pathinfo($extension);
if($pathInfo["extension"] != "pdf" && $pathInfo["extension"] != "xls")
{
echo "Non PDF or excel sheet detected<br>";
return false;
}
if($pathInfo["extension"] == "xls")
{
$excelFile = $extension;
$excelFlag = true;
}
else
{
$excelFlag = false;
}
}
if($zip->extractTo($zipFile->get("uploadedFilePath")))
{
$pathInfo = pathinfo($zipName);
$database->insert('files',[
'name' => $zipFile->get("filename"),
'category' => $zipFile->get("category"),
'date' => $zipFile->setDate(),
'extension' => $pathInfo["extension"],
'size' => filesize($zipFile->get("uploadedFilePath") . $zipFile->get("filename")) / 1000 . 'KB',
'path' => $zipFile->get("uploadedFilePath") . $zipFile->get("filename")
]);
}
else
{
echo "Failure to extract<br>";
}
}
}
if($excelFlag)
{
$url = "insert-new-clients.php?excelfile=" . urlencode($excelFile);
//header("location:$url");
}
}
else
{
echo "File not in zip format";
return false;
}
?>

I figured it out. I realized that $zip->extractTo($zipFile->get("uploadedFilePath")) was attempting to extract 650 files for each iteration of the loop, which is 650 times.
I just moved the extraction code outside the loop and the script executed quickly.

Related

Low speed of saving to the MySQL (PHP - Yii2)

I am trying to import data into MySQL from a JSON file.
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$i = 0;
foreach ($products as $product) {
$i++;
$item = new Product_dub();
$item->id_1c_product = $product->id;
$category = Category_dub::findOne(['id_1c_category' => $product->category_id]);
if (!$category) {
Answer::failure("В этом товаре отсутствует категория или такой категории не существует: " . $product->title);
}
$item->category_id = $category->id;
$item->title = $product->title;
$brand = Brands_dub::findOne(['id_1c_brand' => $product->brand_id]);
if (!$brand) {
Answer::failure("В этом товаре отсутствует бренд/изготовитель: " . $product->title);
}
$item->brand_id = $brand->id;
// $item->shortdesc = $product->shortdesc;
$item->content1 = $product->content1;
$item->content2 = $product->content2;
$item->content3 = $product->content3;
$item->link_order = $product->link_order;
$item->img = $product->img;
$item->in_stock = $product->in_stock ? 1 : 0;
$item->is_popular = $product->is_popular ? 1 : 0;
if (!$item->save()) {
Answer::failure("Не удалось импортировать: Проверьте данные в " . $product->title);
}
if ($i == 200) {
break;
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
There are about 1100 objects in my JSON file. It takes 7 seconds to add 100 rows to the database. Adding 200 lines - 15 seconds. 300 = 33 sec, 400 = 58 sec. Why does it slow down over time and how to speed up this process?
I do everything on the local OpenServer server.
PHP 7.2 version, Xeon 2620v3 processor, 16 GB DDR4, HDD.
UPD 1.
"Can you try not importing and just determine the speed of reading" - I comment $item->save() and get 1-2 sec for all of JSON files. "In each iteration of your cycle you are running 2 DB queries to load category and brand." - I tried to delete these lines for test - but the result was 1-2 seconds faster than with 2 DB queries.
UPD 2.
I changed save() to insert() - the speed has increased. Now all JSON (1107 lines) is imported in 40 seconds.
Are there faster ways to load ready-made data from JSON into the database?
What if there are 100 thousand lines or a million? Is it normal practice to wait a few hours?
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$start = time();
$categoryMap = Category_dub::find()->select(['id', 'id_1c_category'])->indexBy('id_1c_category')->column();
$brandMap = Brands_dub::find()->select(['id', 'id_1c_brand'])->indexBy('id_1c_brand')->column();
foreach ($products as $product) {
Yii::$app->db->createCommand()->insert('product_dub', [
'id_1c_product' => $product->id,
'category_id' => $categoryMap[$product->category_id] ?? '0',
'title' => $product->title,
'brand_id' => $brandMap[$product->brand_id] ?? 'No brand',
'content1' => $product->content1,
'content2' => $product->content2,
'content3' => $product->content3,
'link_order' => $product->link_order,
'img' => $product->img ?? 'no-image.png',
'in_stock' => $product->in_stock ? 1 : 0,
'is_popular' => $product->is_popular ? 1 : 0,
])->execute();
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
I changed save() to insert() - the speed has increased. Now all JSON (1107 lines) is imported in 40 seconds.
Are there faster ways to load ready-made data from JSON into the database?
What if there are 100 thousand lines or a million? Is it normal practice to wait a few hours?
public function importProductFile($file, $return = true)
{
$products = json_decode($file);
$dubTableName = Product::tableName() . "_dub";
$start = time();
if ($this->db->createDuplicateTable(Product::tableName(), $dubTableName)) {
$start = time();
$categoryMap = Category_dub::find()->select(['id', 'id_1c_category'])->indexBy('id_1c_category')->column();
$brandMap = Brands_dub::find()->select(['id', 'id_1c_brand'])->indexBy('id_1c_brand')->column();
foreach ($products as $product) {
Yii::$app->db->createCommand()->insert('product_dub', [
'id_1c_product' => $product->id,
'category_id' => $categoryMap[$product->category_id] ?? '0',
'title' => $product->title,
'brand_id' => $brandMap[$product->brand_id] ?? 'No brand',
'content1' => $product->content1,
'content2' => $product->content2,
'content3' => $product->content3,
'link_order' => $product->link_order,
'img' => $product->img ?? 'no-image.png',
'in_stock' => $product->in_stock ? 1 : 0,
'is_popular' => $product->is_popular ? 1 : 0,
])->execute();
}
}
}
$finish = time();
$res = $finish - $start . "sec. ";
if ($return) {
echo $res;
Answer::success();
}
}
You can use the bulk insert as mentioned in this answer and Yii2 docs. Using this bulk insert, you need to remember that the event will not be triggered.
Yii::$app->db->createCommand()->batchInsert('product_dub', array_keys(reset($products)), $products)->execute();

How to inspect empty rows in uploaded csv

Here i have a csv file which have contains numerous contacts rows, and these rows ae saved to my database table. Now i just want to inspect empty rows from uploaded csv by customer.
Here is an example csv with some empty rows
In this above csv, have 3rd and 6th rows are empty. so want to inspect these empty row number and discard csv with error.
Here is my csv code
$filename = $_FILES["csv_file"]["tmp_name"];
if ($_FILES["csv_file"]["size"] > 0) {
$file = fopen($filename, "r");
$importdata = fgetcsv($file, 10000, ",");
$counter = 1;
while (!feof($file)) {
if ($counter > 1) {
$alldata[] = fgetcsv($file);
}
$counter++;
}
fclose($file);
$csvfieldcounter = 1;
foreach ($alldata as $importdata) {
$userdata = $this->session->userdata();
$userId = $userdata['id'];
$status = 'Y';
if ($importdata[4] == 'Disable' || $importdata[4] == 'disable')
$status = 'N';
else if ($importdata[4] == 'Enable' || $importdata[4] == 'enable')
$status = 'Y';
$data = array(
'customer_name' => $importdata[0],
'customer_email' => $importdata[1],
'customer_mobile' => $importdata[2],
'birth_date' => $importdata[3],
'status' => $status,
'user_id' => $userId,
'cat_type' => $file_cat
);
if ($importdata[2]) {
$run = $this->db->insert('customer', $data);
$csvfieldcounter++;
$id = $this->db->insert_id();
}
}
$this->session->set_flashdata('csv_imported','Your CSV have been successfully imported.');
redirect('/customer', $csvfieldcounter);
}
I just want a little help for get that. your kind efforts would be appreciated Thanks :)
I suggest you to use below mentioned library. This will help you address above issue also it will decrease lines of code in your controller.
https://github.com/parsecsv/parsecsv-for-php
This will help you.
If you worried about how to use than check below mentioned code.
$this->load->library('Parsecsv');
$csv = new Parsecsv($file);
$users = $csv->data;
$noOfUsers = count($users);
foreach($users as $user):
if(!empty($user)):
// Write your code
endif;
endforeach;

How to upload a Large CSV file really fast in Laravel

This question has been asked so many times , I have tried couple of way also but this time I am stuck since my requirement is bit specific . None of the generic methods worked for me .
Details
File Size = 75MB
Total Rows = 300000
PHP Code
protected $chunkSize = 500;
public function handle()
{
try {
set_time_limit(0);
$file = Flag::where('imported','=','0')
->orderBy('created_at', 'DESC')
->first();
$file_path = Config::get('filesystems.disks.local.root') . '/exceluploads/' .$file->file_name;
// let's first count the total number of rows
Excel::load($file_path, function($reader) use($file) {
$objWorksheet = $reader->getActiveSheet();
$file->total_rows = $objWorksheet->getHighestRow() - 1; //exclude the heading
$file->save();
});
$chunkid=0;
//now let's import the rows, one by one while keeping track of the progress
Excel::filter('chunk')
->selectSheetsByIndex(0)
->load($file_path)
->chunk($this->chunkSize, function($results) use ($file,$chunkid) {
//let's do more processing (change values in cells) here as needed
$counter = 0;
$chunkid++;
$output = new ConsoleOutput();
$data =array();
foreach ($results->toArray() as $row)
{
$data[] = array(
'data'=> json_encode($row),
'created_at'=>date('Y-m-d H:i:s'),
'updated_at'=> date('Y-m-d H:i:s')
);
//$x->save();
$counter++;
}
DB::table('price_results')->insert($data);
$file = $file->fresh(); //reload from the database
$file->rows_imported = $file->rows_imported + $counter;
$file->save();
$countx = $file->rows_imported + $counter;
echo "Rows Executed".$countx.PHP_EOL;
},
false
);
$file->imported =1;
$file->save();
echo "end of execution";
}
catch(\Exception $e)
{
dd($e->getMessage());
}
}
So the above Code runs really fast for the 10,000 rows CSV File.
But when I upload a larger CSV its not working .
My Only restriction here is that I have to use following logic to transform each row of the CSV to KeyPair value json data
foreach ($results->toArray() as $row)
{
$data[] = array(
'data'=> json_encode($row),
'created_at'=>date('Y-m-d H:i:s'),
'updated_at'=> date('Y-m-d H:i:s')
);
//$x->save();
$counter++;
}
Any suggestions would be appreciated , Its been more than and Hour now and still only 100,000 rows have been inserted
I find this is really slow
Database : POSTGRES

how to handle duplicate records in SQL?

I am uploading a file to mysql database. Now the file contains records of login and logout of users. Below is its structure.
I am using PHP codeigniter to upload that file and insert its data into SQL.
Here date_data is Defined as UNIQUE So that i dont get any duplicate records for same day as it contains information of daily user's login and logout data.
Now in a case where i have uploaded a data of 1st Nov to 10th December. and then again i upload data from 1st December to 1St January i will get an error because it will give error for the duplicate data from 1st Dec and other consecutive days. Is it possible that i can skip the duplicate data and insert the remaining unique data??
As for my current code it stops the execution when it finds duplicate data. i want to insert only that data which is unique.
Below is my code to insert into SQL table:
Controller
public function upload()
{
$file = rand(1000, 100000) . "-" . $_FILES['file']['name'];
$file_loc = $_FILES['file']['tmp_name'];
$file_size = $_FILES['file']['size'];
$file_type = $_FILES['file']['type'];
$folder = "uploads/";
$location = $_FILES['file'];
$new_size = $file_size / 1024; // new file size in KB
$new_file_name = strtolower($file);
$final_file = str_replace(' ', '-', $new_file_name); // make file name in lower case
if (move_uploaded_file($file_loc, $folder . $final_file))
{
$handle = fopen($folder.$final_file, "r") or die("file cannot open");
if ($handle) {
while (($line = fgets($handle)) !== false)
{
$lineArr = explode("\t", "$line");
$result = $this->attendance_m->insert_file_content($lineArr) ;
}
if (fclose($handle)) {
$this->alert('successfully uploaded', 'admin/attendance.php?success');
redirect('admin/attendance');
}
}
else{
echo "file cannot open";
}
}
}
The Model:
public function insert_file_content($lineArr)
{
$data = array(
'emp_id' => $lineArr[0],
'date_data' => $lineArr[1],
'abc' => $lineArr[2],
'def' => $lineArr[3],
'entry' => $lineArr[4],
'ghi' => $lineArr[5],
);
$this->db->insert('daily_data2', $data);
}
To ignore a duplicate record on insert
$data = array(
'emp_id' => $lineArr[0],
'date_data' => $lineArr[1],
'abc' => $lineArr[2],
'def' => $lineArr[3],
'entry' => $lineArr[4],
'ghi' => $lineArr[5],
);
$sql = "INSERT IGNORE INTO `daily_data2`
(`emp_id`,`date_data`,`abc`,`def`,`entry`,`ghi`)
VALUES
(?,?,?,?,?,?)";
$this->db->query($sql, $data);
You could also try out this way
$result = $this->db->get_where('daily_data2', array('date_data' => $data['date_data'));
if(count( $result->result_array() ) < 1)
{
//insert a new record
}

Only the first IF statement out of 3 is executed within a PHP loop

The following code uploads multiple images no problem. However, I'm trying to get it to update a field in a table based on what iteration the loop is in. PROBLEM: The IF Statement seems to not work when looped. I.e. it only adds the first file_name to the database.
Anyone see what I'm doing wrong here? Much appreciated if so!!!
for ($i = 1; $i < 4; $i++)
{
/* Handle the file upload */
$upload = $this->upload->do_upload('image' . $i);
/* File failed to upload - continue */
if ($upload === FALSE)
continue;
/* Get the data about the file */
$data = $this->upload->data();
$uploadedFiles[$i] = $data;
if ($i == 1)
{
$filenames1 = array(
'product_image_front' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames1);
}
if ($i == 2)
{
$filenames2 = array(
'product_image_back' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames2);
}
if ($i == 3)
{
$filenames3 = array(
'product_image_back' => $data['file_name'],
);
$this->db->where('id', $this->db->insert_id());
$this->db->update('products', $filenames3);
}
}
insert_id - Get the ID generated in the last query.
Store it in a variable before the loop.

Categories