So I have weird problem, using Laravel Excel and importing some data. I want to split import by chunks, but every time I define chunk size it skip 1 record from every chunk.
Here is peace of code:
Excel::selectSheets('Sheet1')->load($tmp_path)->chunk(3,function($result) use ($product)
foreach ($result as $row ) {
$row->dump();
}
});
So I just splitting collection by 3 records to demonstrate problem, screen bellow
Update:
'import' => [
'heading' => false,
'startRow' => 3
]
So if I define startRow I will see desired number of items per chunk, but unnecessary data at the beginning...
Well it's seams that https://github.com/Maatwebsite/Laravel-Excel have some problems with chunk method so I used Laravel chunk instead like this:
$tmp_path = $request->file('import_data')->getRealPath();
$results = Excel::load($tmp_path)->get();
$chunks = $results->chunk(3);
$chunks->toArray();
foreach ($chunks as $rows )
{
foreach ($rows as $row)
{
$row->dump();
}
}
Related
I have about 2000 records in my database and i want to fetch them using view into my datatable.
When i run my code, i noticed my database is only showing 166 records. Why is that happening ?
Controller
$chunks = Items::
where('is_active', 1)
->pluck('name','code','barcode','is_active')
->chunk(10);
foreach ($chunks as $key => $chunk) {
$chunks[$key] = array_values($chunk->toArray());
}
Log::info($chunks);
return $chunks;
Your problem is related to the way you retrieve your data:
$chunks = Items::
where('is_active', 1)
->pluck('name','code','barcode','is_active');
Here, you are misusing pluck(). This method returns an array (a Collection in this case but it's the same here) where the first argument is the value and the second argument is the key.
You are passing 4 parameters, the last two are useless.
Here is what's happening:
$chunks = Items::
where('is_active', 1)
->pluck('name','code'); //removed the last two parameters, they are not used
$chunks is a Collection, where the key is the code and the value is the name.
A Collection, like an array, cannot have the same key multiple times. Then, if two users share the same code, the second user will "overwrite" the first one. The third will overwrite the second and so on.
At the end, you'll have a Collection where the number of items will be equal to the number of unique code that are active (where('is_active', 1).
This why it fails.
How to fix this? Use get(), it does exactly what you expected pluck to do:
$chunks = Items::
where('is_active', 1)
->get(['name','code','barcode','is_active']) //be careful, array here
->chunk(10);
foreach ($chunks as $key => $chunk) {
$chunks[$key] = array_values($chunk->toArray());
}
Log::info($chunks);
return $chunks;
the file i am importing has thousands of records which is causing my network to get slow. i want to read only those columns that i need, before inserting it into database. When file is processed , it should first search those columns not the whole file, and fetch the rows of those columns
Excel::load($path, function($reader) {
//Getting headers using this
$headers = $reader->first()->keys()->toArray();
//This is the array of required columns
$headings = array('registrant_name','registrant_address','registrant_phone','registrant_zip','registrant_email','registrant_country','registrant_state','registrant_city');
});
Data insertion after file read.
if($data->count() > 0)
{
foreach($data->toArray() as $value)
{
$insert[] = array(
'registrant_name' => $value['registrant_name'],
'registrant_address' => $value['registrant_address'],
'registrant_phone' => $value['registrant_phone'],
'registrant_zip' => $value['registrant_zip'],
'registrant_email' => $value['registrant_email'],
'registrant_country' => $value['registrant_country'],
'registrant_state' => $value['registrant_state'],
'registrant_city' => $value['registrant_city']
);
}
}
if(!empty($insert))
{
DB::table('customers')->insert($insert);
}
In collections, you can check the method only documentation
i.e.
$headers = $reader->first()->only(['registrant_name','registrant_address','registrant_phone','registrant_zip','registrant_email','registrant_country','registrant_state','registrant_city']);
You might also use chunk, to insert the data in smaller collections than in one big.
Could you update the post with the App\Import class in order to help you more.
I have loop like following and its run for more than 6000 records,
foreach ($csv as $value) {
$research = ResearchData::create(array('company_id' => Input::get('company'), 'date' => Input::get('date')));
}
in here i used 2 values company_id and date.
i want to know what is the most good way to use this from follow codes
................1....................
$company_id=Input::get('company_id');
$date=Input::get('date');
foreach($csv as value){
$research=ResearchData::create(array('company_id'=>$company_id,'date'=>$date));
}
................2...................
foreach ($csv as $value) {
$research = ResearchData::create(array('company_id' => Input::get('company'), 'date' => Input::get('date')));
}
From a performance point of view, number 1 will be faster, but only because Input::get will take a tiny little bit longer as it does some checks, an array concatenation and eventually grabs something from an array. This will take a completely negligible amount of time, but option 1 does this once whereas option 2 will do this for every iteration of the loop
From any other point of view (code clarity, documentation etc) it's completely opinion based.
You can do a bulk insert. I didn't do a performance check, but I expect a better performance. Check below:
$company_id=Input::get('company_id');
$date=Input::get('date');
$data = array_fill(0, count($csv) - 1, ['company_id' => $company_id, 'date' => $date]); // skip the large foreach
ResearchData::insert(array_values($data)); // skip the numeric keys
Documentation:
http://php.net/array_filter
http://laravel.com/docs/4.2/queries#inserts
I want to import an ExcelSheet to my DB using Symfony/Doctrine (imported ddeboer data-import bundle)
What is best practice to import the data and first check if the data is already imported?
I was thinking of two possibilities:
1)
$numarray = $repo->findAllAccounts();
$import = true;
foreach ($reader as $readerobjectkey => $readervalue) {
foreach ($numarray as $numkey){
if (($numkey->getNum() == $readervalue['number'])){
$import = false;
}
}
if($import){
$doctrineWriter ->disableTruncate()
->prepare()
->writeItem(
array(
'num' => $readervalue['number'],
'name' => $readervalue['name'],
'company' => $companyid
)
)
->finish();
2)
foreach ($reader as $row =>$value ) {
// check if already imported
$check = $this->checkIfExists($repo,'num', $value['number']);
if ($check){
echo $value['number']." Exists <br>";
}else{echo $value['number']." new Imported <br>";
$doctrineWriter ->disableTruncate()
->prepare()
->writeItem(
array(
'num' => $value['number'],
'name' => $value['name'],
'company' => $companyid
)
)
->finish();
public function checkIfExists($repo, $field, $value){
$check = $repo->findOneBy(array($field => $value));
return $check;
Problem is with big exceldatasheets (3000 rows +) with both solutions i get a timeout....
Error: Maximum execution time of 30 seconds exceeded
in general: for performance issues: is it prefered to generate 1000 queries to check if value exists (findOneBy) or to use two foreach loops to compare values?
Any help would be awesome!
Thx in advance...
You can try to check the filemtime of the file : http://php.net/manual/en/function.filemtime.php
I'm not sure if it would work properly but it worth a shot to try it and see if the modified date works as expected.
Otherwise you should think of another way that checking the data like this, it would take lot of resources to do so. maybe adding some metadata to the excel file :
http://docs.typo3.org/typo3cms/extensions/phpexcel_library/1.7.4/manual.html#_Toc237519906
Any other way than looping or querying database for large data is better.
I have a PHP application which requires large 2D arrays of hard-coded data. I currently just have this defined in a script which is included at the start of every script execution.
I would like to know if anyone has a better idea for how to do this.
My script looks something like the following. Except that I have many more rows and each row has many more fields.
function getData() {
return array(
1 => array('name'=>'a something', 'price'=>123, 'field1'=>1, 'field2'=>3, 'field3'=>2),
2 => array('name'=>'b something', 'price'=>123, 'field1'=>3, 'field2'=>3, 'field3'=>2),
3 => array('name'=>'c something', 'price'=>234, 'field1'=>2, 'field2'=>3, 'field3'=>2),
4 => array('name'=>'d something', 'price'=>345, 'field1'=>8, 'field2'=>3, 'field3'=>2),
5 => array('name'=>'e something', 'price'=>655, 'field1'=>12, 'field2'=>3, 'field3'=>2),
6 => array('name'=>'f something', 'price'=>124, 'field1'=>11, 'field2'=>3, 'field3'=>2),
);
}
Each row has the same fields. So it is very much like a DB table result set. I suppose I could put it in a DB table, but I find this script easier to edit and I would think it's much faster to run than querying a DB.
The problem with my current solution is that it can be hard to read because there are so many fields. What I would like is a system that is easy to read and edit, but it must also be very fast.
Would a DB table be better? Or perhaps reading in a CSV file from a spreadsheet?
As a general idea, write your data is any way that's convenient, possibly CSV or something similar. Create a function that can process that data into the format you need. Cache the processed data so this doesn't need to be done every time.
function getData() {
$cache = 'data.dat';
if (file_exists($cache)) {
return unserialize(file_get_contents($cache));
}
$data = <<<DATA
a something, 123, 1, 3, 2
b something, 123, 3, 3, 2
...
DATA;
$data = array_map(function ($row) {
$row = array_map('trim', explode(',', $row));
return array_combine(array('name', 'price', 'field1', 'field2', 'field3'), $row);
}, explode("\n", $data));
file_put_contents($cache, serialize($data));
return $data;
}
In the above example I'm just using a HEREDOC string to store the data. You may want to put that into an external CSV file, or an XML file that you can generate automatically from somewhere or whatever fits your needs.
If it's very large, storing it in a database may be a better idea. Something lightweight like SQLite will do.