Dears,
Actually, I would like to know the best solution to manipulate BIG DATA in LARAVEL/MYSQL.
In my system, am uploading daily excel file (5K rows) into my DB, in case I find the same row in my table so I don't insert it, and if I find the same row, I changed the uploaded date in my DB.
Every time I upload the excel, am checking each row if exist in my table (table contain > 50K) with a normal array like the below
$res = policies::where('phone', '=', $row['phone'])
->where('draft_no', '=', $row['draftno'])
->where('due_date', '=', $duedate)
->where('status', '=', $stat)
->where('bord_date', '=', $borddate)
->where('amount', '=', $row['amnt'])
->where('remarks', '=', $row['remarks'])
->exists();
if(!$res) {
// insert row ($table->save())
}
else {
//update uploaded date to this row.
}
this process takes lots of time bcz each time is checking the table. I tried to use array_chunk in order to insert, but still, the load is big (15 min to 20 min) to finish
Your advice are highly appreciated.
Thanks
You can create an hash of each row and store along with the row. Then check only the row with given hash.
For instance try this stub of code
foreach ($rows as $row) {
$hash = md5($row['phone'] . $row['draft_no'] . $row['due_date'] ...);
$res = Policiess::where('hash', $hash);
if (!$res) {
// create a new row and store the `$hash` in `hash` column
} else {
//update uploaded date to this row
}
}
if you not add new record same excel then not need check in database. but you add same excel file new record then insert all record after update this excel file
Why don't use laravel default eloquent method updateOrCreate.
Hope you already read about it if you don't you can read it from documentation other-creation-methods.
Let me explain what actually it does.
It Accepts an array of values and checks value is already in the database if it's already in the database it will update these values and also update column updated_at or if it's not already in the database, it will create a new entry in the table.
See eg below :-
policies::updateOrCreate(['value'=>1,'value'=>2,'so on...']);
and dont forget to add protected $fillable = [your column] , Because it's use $fillable for this.
Related
I've set up a database and want to update the column status for each row in my UsersController:
I started with this:
User::where('id', '=', 1)->update(['status' => $status]);
This is working, but I need some loop to change all the rows, something like this:
foreach $id from the table
run some code to change individual $status variable
set individual $status value in the 'status' column in each row:
User::where('id', '=', $id)->update(['status' => $status])
end foreach
So for me its unclear how to go through the table via the foreach. Then save the calculated status from my code to each individual id?
#Serge solution is fine for few records but you should be able to use chuck as #ceejayoz suggested
User::chunk(100, function ($users) {
$users->each(function ($user) {
$user->status = getStatus($user);
$user->save();
});
});
Unless the table contains millions of rows... a simple procedural way of doing it is...
$users = Users::get(); // Gets a collection of all users...
foreach ( $users as $user ) {
//compute your status
$user->status = get_your_user_status($user->id);
$user->save();
}
You could also consider using a more functional approach with map for example...
I need to pass some records from one table to another but I also need to delete those records that have been copied.
The thing that makes this complex is that I need to grab a random bunch of records!
This is what I have been trying the past 2 hours:
$temps = DB::table('temps')->inRandomOrder()->take(3)->get();
foreach ($temps as $temp) {
DB::table('videos')->insert(
[
'idd' => $temp->idd,
'title' => '',
]
);
}
foreach($temps as $temp){
DB::table('temps')->where('idd', '=', $temp->idd)->get()->delete();
}
return 'success';
If I only want to insert, no problem.
If I want to delete specific records or the whole table, no problem.
However, it's not working if I try to delete the randomly inserted records. What am I doing wrong?
Executing a delete() after the get() method is incorrect:
foreach($temps as $temp){
DB::table('temps')->where('idd', '=', $temp->idd)->delete();
}
or a single query without the loop:
DB::table('temps')->whereIn('idd', $temps->pluck('idd'))->delete();
I'm new to Laravel and here's my issue.
I have a table currentpercentage.
This is the structure of the table currentpercentage
currentpercentage(**id**, name, total_cap, current_usage, created_at, updated_at)
I'm trying to calculate percentage of current usage; based on total_cap and current usage.
total_cap = 1000,
current_usage = 237,
name = User Name
In my controller i've setup a query to get the value of total_cap and the value of current_usage then calculate that the percentage would be.
When i call my query, it returns an array with the column name (total_cap) and value (1000). Same as when i query for current_usage.
{
"currentpercentage": [
{
"total_cap": 1000
}
]
}
I just want the query to return just the number (1000) without the array.
This is my query
$totalcap = CurrentPercentageModel::select('total_cap')->where('name', '=', 'User Name')->get();
How do I just get the value. Or is there an easier way to calculate the percentage from one query?
CurrentPercentageModel //What I use to connect to the database. DB Model
The problem is that you are using the get method which returns a collection even when you only have one row.
$totalcap = CurrentPercentageModel::select('total_cap')->where('name', '=', 'User Name')->get();
If you just want one record and one column value, then use the value method to get just the value of the column, more info here (you might have to scroll a little bit)
$totalcap = CurrentPercentageModel::select('total_cap')->where('name', '=', 'User Name')->value('total_cap');
I'm trying to retrieve single column from my table grades.
For that I have used following code in my controller:
public function verify($id,$sid)
{
$grade=Grade::all('annual')->whereLoose('id',$id);
return $grade;
}
Where, annual is column name. But it is returning empty set of array [].
all() takes a list of columns to load from the database. In your case, you're fetching only one column called annual, therefore filtering on id later on does not return results. Replace your code with the following and it should work:
$grade = Grade::all('id', 'annual')->whereLoose('id', $id);
Keep in mind that it will return a collection of objects, not a single object.
NOTE: you're always loading all Grade objects from the database which is not efficient and not necessary. You can simply fetch object with given id with the following code:
$grade = Grade::find($id); // fetch all columns
$grade = Grade::find($id, ['id', 'annual']); // fetch only selected columns
The code you are using is loading all rows from the grades table and filtering them in code. It is better to let your query do the filter work.
For the columns part, you can add the columns you need to the first() function of the query, like so:
public function verify($id,$sid)
{
$grade = Grade::where('id', $id)->first(['annual']);
return $grade->annual;
}
I have populate a form of which every text field generated is based on the database result. I simply name every text field using the id. Now when the form is filled, I use controller to save it. But prior to insert the database, I loop the Request::input() to check every item whether such entry is exist or not. I just wonder if there is efficient way to check every item in the loop to insert it into db. Here is my code
public function store(Request $request, $id, $inid)
{
$startOfDay = Carbon::now()->startOfDay();
$endOfDay = Carbon::now()->endOfDay();
$instruments = InstrumentReading::whereBetween('created_at', [$startOfDay, $endOfDay])
->where('iv_inid', '=', $inid)
->get();
foreach ($request->input() as $k => $v) {
$read = new InstrumentReading;
$read->iv_inid = $inid;
$read->iv_ipid = $k;
$read->iv_usid = Auth::user()->id;
$read->iv_reading = $v;
$read->save();
}
if ($instruments->count() > 0) {
//to filter the iv_ipid...
foreach($instruments as $instrument)
{
$instrument->iv_status = "VOID";
$instrument->save();
}
}
}
In words of efficent approach what you can do is to simple check / fetch ONLY all posible rows from the database, and the check in the loop if the row was already inserted. Also fetch only iv_ipid column, as we do not need all columns from the table to do our check. It will be faster to select only the column we need. You can use directly Fluent (Query Builder) over Eloquent to pull the data from database as it greatly increase the performance for a simple query like this.
public function store(Request $request, $id, $inid)
{
// Search only records with submitted iv_ipid, iv_inid and created today
$alreadyInserted = DB::table('instrument_readings')
->whereBetween('created_at', [
Carbon::now()->startOfDay(),
Carbon::now()->endOfDay()
])
// Get only records with submitted iv_ipid
->whereIn('iv_ipid', array_keys($request->input()))
// Get records with given iv_inid only
->where('iv_inid', $inid)
// For our check we need only one column,
// no need to select all of them, it will be fast
->select('iv_ipid')
// Get the records from DB
->lists('iv_ipid');
foreach ($request->input() as $k => $v) {
// Very simple check if iv_ipid is not in the array
// it does not exists in the database
if (!in_array($k, $alreadyInserted)) {
$read = new InstrumentReading;
$read->iv_inid = $inid;
$read->iv_ipid = $k;
$read->iv_usid = Auth::user()->id;
$read->iv_reading = $v;
$read->save();
} else {
//todo
}
}
This is the most efficent way suggested until now, because you fetch at once only the records you are interested in, not all records from today. Also you fetch only one column, the one that we need for out check. Eloquent ususlally give a lot of overheat on the perfomance, so in the suggested code I use directly Fluent, which will boost the speed this part of code is executed by ~ 20%.
Your mistake in the original code is that you are doing database call each time in a loop. When you need such a simple task as a check, never put database calls, queries etc. in a loop. It is an overkill. Instead select all needed data before the loop and then do your checks.
Now this is in case you only need to save new records to database. In case you want to manipulate each record in the loop, let's say you need to loop through each submited entry, get get the model or create it if it does not exists and then do something else with this model, the most efficent way then will be this one:
public function store(Request $request, $id, $inid)
{
foreach ($request->input() as $k => $v) {
// Here you search for match with given attributes
// If object in DB with this attributes exists
// It will be returned, otherwise new one will be constructed
// But yet not saved in DB
$model = InstrumentReading::firstOrNew([
'iv_inid' => $inid,
'iv_ipid' => $k,
'iv_usid' => Auth::user()->id
]);
// Check if it is existing DB row or a new instance
if (!$model->exists()) {
// If it is a new one set $v and save
$model->iv_reading = $v;
$model->save();
}
// Do something with the model here
.....
}
This way Laravel will check if model with the passed parameters already exist in database, and if so it will return it for you. If it does not exist, it will create new instance of it, so you can then set the $v and save to db. So you are good to go to do anything else with this model and you can be sure it exists in database after this point.
First approach (efficiency first)
Consider using a simple SQL INSERT IGNORE Query and use Fluent, i.e.:
Make a composite unique key containing:
iv_inid
iv_ipid
created_time, up to an hour granularity, this is important, because created_at might have a far greater granularity than your intended purpose, and might slow things down a bit.
Use DB, i.e.:
DB::query(
"INSERT IGNORE INTO $yourTable VALUES ( ... )"
);
Pros:
- Extremely fast, all the necessary checking is done on the DB Server
Cons:
- You cannot know which values triggered a duplicate value / unique key violation, as related errors are treated as warnings.
Second approach (convenience first)
Use firstOrFail, i.e.:
$startOfDay = Carbon::now()->startOfDay();
$endOfDay = Carbon::now()->endOfDay();
// ... for
try {
InstrumentReading::where('iv_inid', $inid)
->where('iv_ipid', $k)
->whereBetween('created_at', [$startOfDay, $endOfDay])
->firstOrFail();
continue;
} catch (ModelNotFoundException $e) {
$instrumentReading = InstrumentReading::create([
// your values
]);
}
// ... endfor
Pros:
- Easy to implement
Cons:
- Somewhat slower than simple queries
Your code will send request to database every time you need to check the value. Instead, search all value of this day then check the value. This approach will send request to database only one time.
$startOfDay = Carbon::now()->startOfDay();
$endOfDay = Carbon::now()->endOfDay();
// Search only this day
$instruments = InstrumentReading::whereBetween('created_at', [$startOfDay, $endOfDay])->get();
foreach($instruments as $instrument)
{
// Check the value
}