Prevent seeding unique key violations - php

I have a unique constraint in my database for two columns.
$table->unique(array("table_a_id", "table_b_id"));
Now I use Faker in a seeder class to fill in this pivot table:
$tableAIds = TableA::all()->lists("id")->toArray();
$tableBIds = TableB::all()->lists("id")->toArray();
foreach(range(1, 20) as $index) {
$tableAId = $faker->randomElement($tableAIds);
$tableBId = $faker->randomElement($tableBIds);
DB::table("table_a_table_b_pivot")->insert([
"table_a_id" => $tableAId,
"table_b_id" => $tableBId
]);
}
However this creates duplicates and the seeding fails due to SQL error.
How do I make sure I don't try to insert duplicates?

You can use this:
$tableAIds = TableA::all()->lists("id")->toArray();
$tableBIds = TableB::all()->lists("id")->toArray();
foreach (range(1, 20) as $index) {
repeat:
$tableAId = $faker->randomElement($tableAIds);
$tableBId = $faker->randomElement($tableBIds);
try {
DB::table("table_a_table_b_pivot")->insert([
"table_a_id" => $tableAId,
"table_b_id" => $tableBId,
]);
} catch (\Illuminate\Database\QueryException $e) {
//look for integrity violation exception (23000)
if($e->errorInfo[0]==23000)
goto repeat;
}
}
Ps: This may not work if from the begining TableA and TableB datas are inconsistents (meaning it is not possible to form 20 uniques couples from those data )

$unique = $faker->unique()->regexify('[0-4][0-9][0-4][0-9]');
The code above will result in a string composed of two numbers between 0 and 49 that never repeats. Now you can break into two and seed your database securely.
$firstId = ltrim(substr($unique, 0, 2), '0') + 1;
$secondId = ltrim(substr($unique, 2, 2), '0') + 1;
The two numbers can be the same in a string, but a string will always be unique. Don't forget to add one for each of the parts because we don't want (and we can't) to seed our database with zeros.

Instead of using faker, use the collection's random method:
$tableAIds = TableA::all()->lists('id')->random(20);
$tableBIds = TableB::all()->lists('id')->random(20);
foreach(range(0, 19) as $index) {
DB::table("table_a_table_b_pivot")->insert([
"table_a_id" => $tableAIds[$index],
"table_b_id" => $tableBIds[$index],
]);
}
Adjust this according to your needs.
For example, this might be what you really want:
$tableAIds = TableA::all()->lists('id');
$tableBIds = TableB::all()->lists('id');
foreach ($tableAIds->random(5) as $tableAId) {
foreach ($tableBIds->random(2) as $tableBId) {
DB::table("table_a_table_b_pivot")->insert([
"table_a_id" => $tableAId,
"table_b_id" => $tableBId,
]);
}
}
Whatever it is, play with it yourself. No need for faker here.

Related

Why laravel eloquent insert function missing values after 50k entries while using chunks?

I have an excel sheet that has more than 1 million rows in it. I want to insert and update these rows of data to a table. I have used "Maatwebsite" excel library and imported excel to a temporary table. After this, I want to insert temporary table data into the actual table.
Following is the code for fetching data from the temp table.
$temp_products = ProductTemp::limit(100000)->get()->map(function(ProductTemp $temp_product) {
return [
'msrp' => $temp_product->msrp,
'price' => $temp_product->unit_price,
'product_overview' => $temp_product->part_description,
'manufacturer_id' => $temp_product->manufacturer_id,
'upc' => $temp_product->upc,
'manufacturer_part_number' => $temp_product->part_no
];
});
Then I tried both the chunk() and array_chunk() functions to divide data into parts.
//$chunks = $temp_products->chunk(5000);
$arr_chunks = array_chunk($temp_products->toArray(), 5000);
Then I loop through them to insert rows in a actual table like this with the array_chunk() function.
foreach($arr_chunks as $chunk)
{
Product::insert($chunk);
}
and with chunk() function like this.
foreach($arr_chunks as $chunk)
{
Product::insert($chunk->toArray());
}
I have also tried the sleep() function with this but not helping. After 50, 000 entries it's missing values or adding more. Also, I tried chunks of 100, and 1000 but the result is the same and the page never finishes loading.
Here is the full function code, this might help with what I have tried more and what I am missing.
public function insertFromTempTable()
{
// dd("HERE 1111111 22222222222 4444444444");
$temp_products = ProductTemp::all()->map(function (ProductTemp $temp_product) {
return [
'msrp' => $temp_product->msrp,
'price' => $temp_product->unit_price,
'product_overview' => $temp_product->part_description,
'manufacturer_id' => $temp_product->manufacturer_id,
'upc' => $temp_product->upc,
'manufacturer_part_number' => $temp_product->part_no,
];
});
// dd($temp_products);
$inc = 0;
dump("Started on: " . Carbon::now()->toDateTimeString());
// sleep(10);
// foreach ($temp_products as $value) {
// Product::insert($temp_products[$inc]);
// $inc++;
// }
// dd($temp_products->toArray());
// Product::insert($temp_products->toArray());
// $chunks = $temp_products->chunk(100);
$chunk_sub = 5000;
$arr_chunks = array_chunk($temp_products->toArray(), $chunk_sub);
// dd($arr_chunks[0]);
// dd(gettype($chunks[0]->toArray()));
// dd($chunks[0]->toArray());
$chunk_temp = 0;
foreach ($arr_chunks as $chunk) {
// dd($chunk->toArray());
// dump(count($chunk->toArray()));
// Product::insert($chunk->toArray());
$chunk_temp = $chunk_temp + $chunk_sub;
Product::insert($chunk);
}
// $temp_products->chunk(2, function ($subset) {
// dd("HHHHHHH");
// $subset->each(function ($item) {
// dd($item);
// });
// });
dump("Ended on: " . Carbon::now()->toDateTimeString());
}
After inserting I want to compare an excel sheet on daily basis, that might have more than 1 million rows. That will check for MSRP and price columns and update them. Please let me know which will be the best solution for all of this.
UPDATE 01:
I have tried using a raw query. There were 104, 8575 rows in the temp table but after the query run, there are 103, 9229 rows in the products table. Following is the code for the raw query.
$results = DB::table('products')->insertUsing(
['msrp', 'price', 'product_overview', 'manufacturer_id', 'upc', 'manufacturer_part_number',],
function ($query) {
$query
->select(['msrp', 'unit_price', 'part_description', 'manufacturer_id', 'upc', 'part_no',])
->from('products_temp');
}
);
When I dump and die the $results variable it output 104, 8575.

Using PHP Faker in Laravel to generate "unique with" entry when seeding a database using a factory

So Similar to the unique with validation rule (See: https://github.com/felixkiss/uniquewith-validator), I want to know how to generate a entry, where one column is unique with another one. I want to seed my database as follows.
Example:
There are 12 steps in the "steps" table. Each step should have 5 categories associated with each one that are stored in the "step_categories" table. Each of those categories are assigned a unique order number 1 through 5 that is unique with each "step_id".
See this image here for an example of what the database should look like: https://imgur.com/a/XYA5yyn
I had to manually to make the entries in the database for the above image example. I don't want to have to generate this manually every time, say I make a mistake and have to rollback the migrations for example.
I am using a factory to generate this data. So the factory name is StepCategoriesFactory.php and clearly I'm calling the factory with the create() method from the DatabaseSeeder.php file.
I thought about doing this in a for loop, then i got as far as realizing when i called the 'step_id' => App\Model::all()->random()->id to grab a new id, that I wouldn't be able to ensure I wasn't grabbing the id that i just generated 5 entries for. I'm really new with Laravel, and I'm not sure where to even start on this. There's no real information on SO where faker can use the unique with another column. How would I Go about this?
NOTE: The step id is not always going to be 1-12. The step ID might be different depending on whether a step gets deleted and remade. So just assigning the step_id to equal 1-12 wont work.
UPDATE: Here's some code I just wrote, and I think I'm on the right track. Maybe. I've grabbed the step_id by it's number field as that will always be 1-12, and I've grabbed the IID out of the entry. But now I'm stuck on how to generate the order 1-5 without repeating itself. I still haven't run this yet as its incomplete and I know it'll throw an error without the correct order number.
UPDATE 2: I think I'm on the right track here. However I'm getting an undefined variable error. When I put the first line from within the anonymous function, it's resetting the order to "1" for every entry. How do i make the $autoIncrement variable available to the anonymous function? The Seeder has stayed the same between updates.
Image of the error: https://imgur.com/a/ywkd0Lb
Second image with the Die/Dump error in terminal: https://imgur.com/a/rOGRv32
Reference this article here: https://laracasts.com/discuss/channels/laravel/model-factory-increment-value-faker?page=1
UPDATE 3: I forgot the use ($autoIncrement) line of code for the anonymous function. Code below has been updated, but now I'm getting a different error saying that the order column has a null value and can't be inserted. clearly it should be '1'. Even after I call my $autoIncrement->next(); which should increment it to '1' it's still returning null according to the terminal. However, when I do a diedump on $autoIncrement->current() it's returning 1. Weird.
Update 3 error: https://imgur.com/a/STOmIjF
StepCategoriesFactory.php
use Faker\Generator as Faker;
$autoIncrement = autoIncrement();
$factory->define(App\StepCategory::class, function (Faker $faker) use ($autoIncrement) {
// Generate Created At and Updated at DATETIME
$DateTime = $faker->dateTime($max = 'now');
$autoIncrement->next();
$order = (int) $autoIncrement->current();
return [
// Generate Dummy Data
'order' => $order,
'name' => $faker->words(4, true),
'created_at' => $DateTime,
'updated_at' => $DateTime,
];
});
function autoIncrement()
{
for ($i = 0; $i < 5; $i++) {
yield $i;
}
}
Edit: Put a bounty on this question, as I think it would be helpful for the community to get a detailed answer. I'm looking for help to explain how to go about making sure I'm grabbing the same entry through each loop.
FINALLY SOLVED!
So I took in everyone's answers, and thought long and hard about using a for loop to create the order number. 1-5. The problem that I was running into at the end was that the $i variable was not resetting. So after the yield I had to check if the $i variable equalled 5 and then reset it back to zero.
Heres the code!
StepCategories.php
use Faker\Generator as Faker;
$autoIncrement = autoIncrement();
$factory->define(App\StepCategory::class, function (Faker $faker) use ($autoIncrement) {
// Generate Created At and Updated at DATETIME
$DateTime = $faker->dateTime($max = 'now');
// Get the next iteration of the autoIncrement Function
$autoIncrement->next();
// Assign the current $i value to a typecast variable.
$order = (int) $autoIncrement->current();
return [
// Generate Dummy Data
'order' => $order,
'name' => $faker->words(4, true),
'created_at' => $DateTime,
'updated_at' => $DateTime,
];
});
function autoIncrement()
{
// Start a loop
for ($i = 0; $i <= 5; $i++) {
// Yield the current value of $i
yield $i;
// If $i is equal to 5, that must mean the start of a new loop
if($i == 5) {
// Reset $i to 0 to start over.
$i = 0;
}
}
}
DatabaseSeeder.php
// Generate Dummy Categories
// Run the factory 12 times
foreach(range(1, 12) as $i) {
// Generate 5 entries each time
factory(App\StepCategory::class, 5)->create([
// Since all steps have a number 1-12 grab the step by the number column and get it's ID
'step_id' => App\Step::where('number', '=', $i)->first()->id,
]);
}
Thanks to all who helped!
Sorry if you don't understand my point so I'll try to explain it in code
use Illuminate\Database\Seeder;
$factory->define(App\StepCategory::class, function (Faker $faker) {
// Generate Created At and Updated at DATETIME
$DateTime = $faker->dateTime($max = 'now');
$step_id = function () {
return factory('App\Step')->create()->id;
};
return [
// Generate Dummy Data
'step_id' => $step_id,
'order' => uniqueOrder($step_id),
'name' => $faker->words(4, true),
'created_at' => $DateTime,
'updated_at' => $DateTime,
];
});
function uniqueOrder($step_id)
{
$unique = rand(1,5);
do {
$unique = rand(1,5);
}
while(StepCategory::where('step_id', $step_id)->andWhere( 'order', $unique)->exists())
return $unique;
}
for example if your Step model name is Steps :
$allSteps = Steps::all();
foreach($allSteps as $step){
for($i=1;$i<6;$i++){
//insert to table $step->id , $i for example
DB::table('yourTableName')->insert([
'name'=>'Step '.$step->id.'- Category '.$i ,
'order'=>$i ,
'step_id'=>$step->id
]);
}
}

Laravel - generate unique order number

I'm currently trying to generate a unique order number when the user reaches the create method. The order numbers are generated like this in the seed and need to look like this as well
Seed
foreach(range(1,25) as $index)
{
DB::table('orders')->insert([
'user_id' => rand(1,25),
'order_nr' => '#' . sprintf("%08d", $index),
'price_sum' => $faker->randomNumber($nbDigits = 4, $strict = false) . '.' . $faker->randomNumber($nbDigits = 2, $strict = false),
'status' => $faker->randomElement(['paid', 'pending', 'failed']),
'created_at' => Carbon::now(),
'updated_at' => Carbon::now(),
]);
}
The order numbers look like this #00000001 or #00000002. Now when the user reaches the create method in my controller a new unique order number in this sequence needs to be created. How can I achieve that? The controller currently looks like this:
public function create()
{
$order = new Order;
$order->user_id = Auth()->id();
$order->order_nr =
dd($order);
return view('steps.order');
}
It needs to check the latest order number and create one with +1 on that order number. Say for instance there are 25 orders and the last one is #00000025 the one that needs to be created next needs to be #00000026. How can I achieve that?
Try doing as following
$order = new Order;
$order->user_id = Auth()->id();
$latestOrder = App\Order::orderBy('created_at','DESC')->first();
$order->order_nr = '#'.str_pad($latestOrder->id + 1, 8, "0", STR_PAD_LEFT);
$order->save();
Here I am assuming that the id is auto-incrementing. See the str_pad method for more details
Use the auto column id value of your rows to generate the order number. However, do not create an extra column for your order number, because this would give you an unnormalized DB, since the order-column is completely dependent of the id column.
Instead, add this method to you order model
public function get_order_number()
{
return '#' . str_pad($this->id, 8, "0", STR_PAD_LEFT);
}
If your last order had the id 5 and you would delete it, then the next order would have the id 6.
The only possible exception is if you create orders within an transaction. If a transaction is rolled back, the associated order id would be skipped.
You can use the Laravel ID generator.
First Install it:
composer require haruncpi/laravel-id-generator
Import the class in your controller.
use Haruncpi\LaravelIdGenerator\IdGenerator;
Now simply use it
$prefix = "#";
$id = IdGenerator::generate(['table' => 'your_table_name', 'length' => 9, 'prefix' =>$prefix]);
Output
#00000001
#00000002
#00000003
...
There is a problem in reading the maximum id and increment it.
In heavy servers or parallel requests, two requests may read the same max(id) from MySQL.
The best way to create unique sequence number is use a sequence table with just one auto increment field.
First insert new record in the sequence table.
Then read the last inserted id from db by LAST_INSERT_ID() MySQL function.
At last you can remove old records lower than your id.
Laravel way:
$id = DB::table('seq_foo')->insertGetId(['id' => null]);
DB::table('seq_foo')->where('id', '<', $id)->delete();
$id is your unique sequence number.
You may try with this:
public function generateOrderNR()
{
$orderObj = \DB::table('orders')->select('order_nr')->latest('id')->first();
if ($orderObj) {
$orderNr = $orderObj->order_nr;
$removed1char = substr($orderNr, 1);
$generateOrder_nr = $stpad = '#' . str_pad($removed1char + 1, 8, "0", STR_PAD_LEFT);
} else {
$generateOrder_nr = '#' . str_pad(1, 8, "0", STR_PAD_LEFT);
}
return $generateOrder_nr;
}
You can generate order_nr by using this: $this->generateOrderNR(); in your create() function.
In addition mt_rand() is 4 times faster than rand() you may use this for better user experience.
I have found a better solution:
$paynowbillprefix1="ADDON-";
$paynowbillprefix1=strlen($paynowbillprefix1);
$query2 = "select gen_id from user_cart_addon order by id desc limit 0, 1";
$exec2 = mysqli_query($conn,$query2) or die ("Error in Query2".mysqli_error());
$res2 = mysqli_fetch_array($exec2);
$num2 = mysqli_num_rows($exec2);
$billnumber = $res2["gen_id"];
$billdigit=strlen($billnumber);
if ($billnumber == '')
{
$billnumbercode ='ADDON-'.'1';
}
else
{
$billnumber = $res2["gen_id"];
$billnumbercode = substr($billnumber,$paynowbillprefix1, $billdigit);
$billnumbercode = intval($billnumbercode);
$billnumbercode = $billnumbercode + 1;
$maxanum = $billnumbercode;
$billnumbercode = 'ADDON-'.$maxanum;
}

Laravel "Trying to get property of non-object" Setting Array with Database Values

I'm Trying to solve this error i'm having with PHP, i'm not completely familiar with the Language, so it would be nice if you would help me out, I can't figure out this error.
I have this Code Here:
public function index() {
$counterino = ClientsJobs::all()->count();
$MasterArray = array();
/* Go Through All of the Records in the Client-Jobs Table and Resolve their columns to Desired Names */
for ($i = 1; $i <= $counterino; $i++ ) {
//Temporary Array for one Name-Resolved-Row of the Table.
$tempArray = array(
'id' => ClientsJobs::find( $i )->id, // id
'client_name' => ClientsJobs::find( $i )->clients->fname , // get the first name ( based on fk )
'job_name' => ClientsJobs::find( $i )->jobs->name, // get the name of the job ( based on fk )
'wage' => ClientsJobs::find( $i )->wage, // wage for the job
'productivity'=> ClientsJobs::find( $i )->producivity // productivity level for the job
);
$MasterArray[] = $tempArray; //add the row
}
return $MasterArray;
}
This code changes the names of the of the Columns in the ClientsJobs Junction Table.
public function up()
{
Schema::create('clients-jobs', function(Blueprint $table)
{
$table->increments('id')->unsigned();
$table->integer('client_id')->unsigned();
$table->foreign('client_id')->references('id')->on('clients');
$table->integer('job_id')->unsigned();
$table->foreign('job_id')->references('id')->on('jobs');
$table->decimal('wage', 4, 2);
$table->decimal('productivity', 5, 2); // 0.00 - 100.00 (PERCENT)
$table->timestamps();
});
}
The Jobs and Clients Table are very simple.
I am having the Error in the index() function I posted above, it says
'Trying to get property of non-object'
Starting on the Line
'client_name' => ClientsJobs::find( $i )->clients->fname,
It's also mad at me for the other parts of setting the array.
I have tested the individual functions I am using to set the array and they all work, fname should also return a string, I used dd() to get the value.
I have tried:
-Using FindorFail
-Setting the Array without the for loop and setting each element manually
-Dumping out multiple parts of the function to make sure it works( counterino, all of the functions for the array, .. )
My guess is that it has to do with the type-deduction of PHP, I actually only need a string array, but would still like to use the name mappings because I am going to be passing this a View I am using for some of my other stuff. The Code was actually working earlier, but I broke it somehow (adding a new record or running a composer update?) anyway, there's some serious voodoo going on.
Thanks in Advance for the help, I am working on this project for a Non-Profit Organization for free.
P.S. I am using Laravel 4.2, and Platform 2.0
First off, this is a horrible practice:
$tempArray = array(
'id' => ClientsJobs::find( $i )->id, // id
'client_name' => ClientsJobs::find( $i )->clients->fname , // get the first name ( based on fk )
'job_name' => ClientsJobs::find( $i )->jobs->name, // get the name of the job ( based on fk )
'wage' => ClientsJobs::find( $i )->wage, // wage for the job
'productivity'=> ClientsJobs::find( $i )->producivity // productivity level for the job
);
By calling ClientJobs::find($i) multiple times, you are doing multiple times the same lookup - either to your DB, or to your cache layer if you have one configured.
Secondly, the answer to your question depends on your ClientJobs model. For your example to work, it needs:
A valid clients relations, defined as follows:
public function clients()
{
return $this->hasOne(...);
}
clients also needs to be a valid 1:1 always existing relation. i.e. there must always be one client. If there isn't, you are susceptible to the error you just got (as the `clients̀ magic would end up being null)
The same applies to jobs.
In every case, it is better to make sure everything is set first. Check using the following:
$clientJob = ClientJobs::find($i);
if (!$clientJob->clients || $clientJob->jobs) throw new \RangeException("No client or job defined for ClientJob $i");
And then catch the exception at whichever level you prefer.
Best approach
public function index() {
$masterArray = array();
ClientsJobs::with('clients', 'jobs')->chunk(200, function($records) use (&$masterArray) {
foreach ($records as $record) {
$masterArray[] = array(
'id' => $record->id, // id
'client_name' => !empty($record->clients) ? $record->clients->fname : null,
'job_name' => !empty($record->jobs) ? $record->jobs->name : null,
'wage' => $record->wage,
'productivity'=> $record->productivity,
);
}
});
return $MasterArray;
}
Your Approach is very wrong
If you want to return an array you can do like this
$counterino = ClientsJobs::all()->toArray();
This will fetch all rows from the table and the toArray will convert the object into an array

post an array and iterate throught it in PHP codeigniter

This is the first time i create my own webservice (someone always did it for me before), so please bear with me.
I post this array :
$data = array(
'user_id' => $this->post('user_id'),
'group_id' => $this->post('group_id'),
'child_id' => $this->post('child_id'), //will be nested array
'custom' => $this->post('custom'),
'time' => $this->post('time'),
'date' => $this->post('date')
);
I tried to create a nested array with this : $this->post('child_id'), because user can post multiple child_id at once.
Then i tried to iterate through the child_id, because i need to insert them to the mysql :
for($i = 0; $i < sizeof($data['child_id']); $i++)
{
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data, $result_id[0]['id']);
}
What should i do, so i can have an array of child_id in my $data array? (nested array)
And how to iterate through it?
UPDATE :
I have updated the codes above.
I use advanced rest client for testing, and i tried to post something like this in the form content type :
child_id=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id=2
Notice that theres two child_id (left most and right most), but only the last one (right most) is inserted.
And this is the add_trans in the model :
function add_trans($table, $data, $schedule_id) {
$query = $this->db->insert($table, array('child_id' => $data['child_id'], 'schedule_id' => $schedule_id));
return $query;
}
Thanks a lot for your time.
Even thought you set the name attribute as child[] on the markup,
You still need to call it as:
'child_id' => $this->post('child_id')
It will still return an array.
for($i = 0; $i < sizeof($data['child_id']); $i++) {
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data, $result_id[0]['id']);
}
EDIT:
Looking upon you query string, that seems to be the culprit:
child_id=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id=2
^ same index , same index, same index, it will overwrite and you will get only `2`
If you want to get them all into an array format, you need to set them like this
child_id[]=1&user_id=1&group_id=1&custom=&time=17%3A17%3A00&date=&child_id[]=2
^ it needs to be set like this
UPDATE:
And in your model, if you want each id per row, well you can also loop in this case:
function add_trans($table, $data, $schedule_id) {
foreach($data['child_id'] as $child_id) {
$query = $this->db->insert($table, array('child_id' => $child_id, 'schedule_id' => $schedule_id));
}
// return $this->db->insert_id();
return $query;
}
ofcourse that won't work, it has to be
for($i = 0; $i < sizeof($data['child_id']); $i++)
{
$result2 = $this->schedule_m->add_trans('transaction_schedule', $data['child_id'][$i], $result_id[0]['id']);
}
because you've not set $data['child_id[]'] so it doesn't exist, the key is just a string or number, it does not validate or parse anything
you don't need to give child[] in post method. just give only child, it will get complete array what are you sending from views
replace
'child_id' => $this->post('child_id[]')
with
'child_id' => $this->post('child_id')

Categories