Use faker in laravel when inserting thousand of records - php

Question

I have one issue with laravel faker, I was looking for a tutorial to insert thousands of records using seeders
This was my PostSeeder.php:
<?php
namespace Database\Seeders;
use Illuminate\Database\Seeder;
use App\Models\Post;
use App\Models\User;
class PostSeeder extends Seeder
{
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
Post::factory(10)->create();
}
}
Here I was inserting 10 posts, but I need to test thousands or millions of records, so I saw a tutorial and modified the seeder
<?php
namespace Database\Seeders;
use Illuminate\Database\Seeder;
use App\Models\Post;
use App\Models\User;
use Illuminate\Support\Str;
class PostSeeder extends Seeder
{
/**
* Run the database seeds.
*
* #return void
*/
public function run()
{
$users= collect(User::all()->modelKeys());
$data = [];
for ($i = 0; $i < 100000; $i++) {
$data[] = [
'body' => Str::random(50),
'image' => 'https://via.placeholder.com/640x480.png/0077dd?text=inventore',
'user_id' => $users->random(),
'created_at' => now()->toDateTimeString(),
'updated_at' => now()->toDateTimeString(),
];
}
$chunks = array_chunk($data, 10000);
foreach ($chunks as $chunk) {
Post::insert($chunk);
}
}
}
With this approach I can insert thousand of records faster, but the problem is that I am not inserting correctly the body and image field
I wanted to try something with faker, in my factory I have this:
PostFactory.php
<?php
namespace Database\Factories;
use App\Models\Post;
use App\Models\User;
use Illuminate\Database\Eloquent\Factories\Factory;
class PostFactory extends Factory
{
/**
* The name of the factory's corresponding model.
*
* #var string
*/
protected $model = Post::class;
/**
* Define the model's default state.
*
* #return array
*/
public function definition()
{
return [
'body' => $this->faker->text,
'image' => $this->faker->imageUrl(),
'user_id' => function() {
return User::factory()->create()->id;
}
];
}
}
I would like to use faker methods like these ones in the PostSeeder but I can't, what can I do? thank you.
EDIT:
I tried this:
public function run(Faker $faker)
{
$users= collect(User::all()->modelKeys());
$data = [];
for ($i = 0; $i < 50000; $i++) {
$data[] = [
'content' => $faker->text,
'image_path' => $faker->imageUrl(),
'user_id' => $users->random(),
'created_at' => now()->toDateTimeString(),
'updated_at' => now()->toDateTimeString(),
];
}
$chunks = array_chunk($data, 5000);
foreach ($chunks as $chunk) {
Post::insert($chunk);
}
}
And I got this message:
PDOException::("SQLSTATE[HY000]: General error: 2006 MySQL server has gone away")
But when I try with fewer records it works, so, I changed the seeder like this:
$users= collect(User::all()->modelKeys());
$posts = Post::factory(10)->create();
$posts = collect($posts->only(['content','image_path']));
...
...
'content' => $posts->random()->content,
'image_path' => $posts->random()->image_path
...
and this doesn't work, it got this error:
You requested 1 items, but there are only 0 items available.
It looks like $posts->only(['content','image_path']) is not working properly. So I tried this:
Post::factory(10)->create();
$tweets = Tweet::select(['content','image_path'])->get();
...
'content' => $posts->random()->content,
'image_path' => $posts->random()->image_path
...
And again it works with a few records, but when I try with thousands, I get this error again:
PDOException::("SQLSTATE[HY000]: General error: 2006 MySQL server has gone away")
What can I do? thank you

Answer 1

Since model factories create in-memory objects, because of big memory usages it is not suitable for large seeds.
But you can use Faker for data generation:
use Faker\Generator as Faker;
class PostSeeder extends Seeder
{
public function run(Faker $faker)
{
$users= collect(User::all()->modelKeys());
$data = [];
for ($i = 0; $i < 100000; $i++) {
$data[] = [
'body' => $faker->text,
'image' => $faker->imageUrl(),
'user_id' => $users->random(),
'created_at' => now()->toDateTimeString(),
'updated_at' => now()->toDateTimeString(),
];
}
$chunks = array_chunk($data, 10000);
foreach ($chunks as $chunk) {
Post::insert($chunk);
}
}
}