Validating a large CSV file format in Laravel - php

I would like to validate a large (up to millions of rows) CSV file content without reading it twice : once in the validation request file, once when storing the content of the CSV file to the database. What's the best, clean and efficient approach?
Here is what I have so far:
CreateSymbolRequest.php:
public function rules()
{
return [
'prices' => [
'bail',
'required',
'file',
'max:8192',
new PriceFileExtensionsRule(['csv']),
new PriceFileHeadersRule,
new PriceFileDataFormatRule
],
'pair' => 'required|array',
'timeframe' => 'required|in:1S,1,1H,4H,1D,1W,1M',
'type' => 'required|in:spot,future',
'exchange' => 'required|in:FTX,binance',
'tx_fee' => 'required|numeric',
'm_fee' => 'required|numeric'
];
}
public function persist(): void
{
// checks if the Symbol already exists in the DB
if ($this->isSymbolDuplicate()) {
throw new Exception('Symbol already exists in the database.');
}
// Stores the Symbol in the DB
$this->persistSymbol();
// Stores the CSV content in the DB
$this->persistPrices();
}
SymbolsController:
public function store(CreateSymbolRequest $request)
{
$request->persist();
}
Of course, I could split the basic validation in CreateSymbolRequest.php and the more advanced validation in SymbolsController.php, but that would kind of defeat the purpose of having a separate Request file with the validation rules in it.
I was also thinking of passing a $content variable by reference to the Rule files but I don't think I can return anything from there and that feels very ugly anyway.

Related

How to pass an error message without a bag in Laravel/Inertia?

I'm trying to validate some data this way:
public function update($user, array $input)
{
$v = Validator::make($input, [
'name' => ['required', 'string', 'max:255'],
'email' => ['required', 'email', 'max:255', Rule::unique('users')->ignore($user->id)],
'photo' => ['nullable', 'mimes:jpg,jpeg,png', 'max:1024'],
]);
//MORE CODE
}
Then, in place of "//MORE CODE", I tried to do this:
$messagesArray = [];
foreach ($v->messages()->all() as $message) {
$messagesArray[] = $message;
}
if($v->fails()){
return redirect()->back()->withErrors($messagesArray, 'errors');
}
I tried other possibilities too (without success) like using the Request $request as a parameter but for some reason that I don't know, I'm not allowed to.
The Problem
My problem is that I'm working with a huge application where I can't work with the bag at the client-side. I need to deliver the error props as an single Object (like: {'errors' : {0:"error1", 1:"error2"...}}), but I'm delivering something like this at the moment: {'errors': {pagename:{0:"error1, 1:"error2"}}}, that is I can't pass the errors in bags.
Last Comment
I know that the Validator class has the validateWithBag() method and I wondered if there wasn't some option like validateWithoutBag()...
I already read the Laravel documentation and other questions, but... nothing
Thank you for reading it!
I solved it. the errorBags are set at the Vue Components.
For example:
this.form.post(route("user-profile-information.update"), {
errorBag: "updateProfileInformation",
preserveScroll: true,
onSuccess: () => this.clearPhotoFileInput(),
});
At this snippet, the client set the errorBag, not the server-side.
Validate Without Bag
To validate without the bag, you must change ->validateWithBag("BagName"); for ->validate(); and remove the errorBag: "BagName" at your Vue Component.

Laravel validation issue

I'm experiencing an issue when validator rules
return [
'features' => 'required|array',
'features.*' => 'required|string',
'link' => 'required|url',
'image' => 'nullable|file|image|mimes:jpeg,png,gif,webp|max:2048',
];
Return me an error that fields are required even if they are present.
I can't understand what causes the problem. I use identical validation for storing and it works perfectly.
Here is my controller's code
public function update(UpdateSite $request, Site $site)
{
$validatedData = $request->validated();
if ($validatedData['image']) {
Storage::delete($site->path);
$imagePath = $validatedData['image']->store('thumbnails');
$interventedImage = Image::make(Storage::url($imagePath));
$interventedImage->resize(500, null, function ($constraint) {
$constraint->aspectRatio();
});
$interventedImage->save('storage/'.$imagePath);
$site->update([
'path' => $imagePath,
]);
}
$site->update([
'site_link' => $validatedData['link'],
]);
$site->features()->delete();
if ($validatedData['features']) {
foreach ($validatedData['features'] as $feature) {
$site->features()->save(new SiteFeature(["feature" => $feature]));
}
}
return $this->response->item($site, new SiteTransformer);
}
Update #1
My route
$api->put('sites/{id}', 'SiteController#update')->where(['id' => '\d+']);
The problem lies in PHP which cannot work with multipart/form-data in PUT, PATCH request. Very curious that this problem is still present since there are in the Internet topics from about 2014.
Soulution
There is a solution in docs https://laravel.com/docs/5.6/routing#form-method-spoofing
So to update a record all I need is to use method post instead of put/patch and send an input field _method = PUT.
Just tried myself the put route was invoked.
I see the api method is PUT, but you are using Postman by form-data to request. Try to use x-www-form-urlencoded request you api.
This is about my test. Sorry for my english.
if features is an array so line two is correct but if you pass features as a string line two should be remove, this rules say validation that you have two parameter called features one of them is string and is required other also is array and required
'features' => 'required|array',
'features.*' => 'required|string',

Laravel 5: Validate if field value exists if data source is not MYSQL but Repository

I am working on REST API with add/update/delete methods.
In Controller, In Update action i have to check if id passed exists, I know i can use Laravel validator exists:table,column , but i am using Repository inside controller
public function __construct(BookRepositoryInterface $bookRepository, Validator $validator)
{
$this->bookRepository = $bookRepository;
$this->validator = $validator;
}
Repository source can be different than mysql database. Laravel validator 'exists' will not work in every case.
This is the code for updating book by an id.
public function update($id)
{
$params = Input::only('title', 'author');
$params['id'] = $id;
$rules = array(
'id' => 'required',
'author' => 'required',
'title' => 'required'
);
$validator = $this->validator->make($params, $rules);
...........
}
What is best way to check if id exists in book repository during update?
The best way for that, is to have the create method on the Repository, and run the validation inside the repository.
In that case, the validation implementation will be Repository dependent.

Unit Test Laravel's FormRequest

I am trying to unit test various custom FormRequest inputs. I found solutions that:
Suggest using the $this->call(…) method and assert the response with the expected value (link to answer). This is overkill, because it creates a direct dependency on Routing and Controllers.
Taylor’s test, from the Laravel Framework found in tests/Foundation/FoundationFormRequestTest.php. There is a lot of mocking and overhead done there.
I am looking for a solution where I can unit test individual field inputs against the rules (independent of other fields in the same request).
Sample FormRequest:
public function rules()
{
return [
'first_name' => 'required|between:2,50|alpha',
'last_name' => 'required|between:2,50|alpha',
'email' => 'required|email|unique:users,email',
'username' => 'required|between:6,50|alpha_num|unique:users,username',
'password' => 'required|between:8,50|alpha_num|confirmed',
];
}
Desired Test:
public function testFirstNameField()
{
// assertFalse, required
// ...
// assertTrue, required
// ...
// assertFalse, between
// ...
}
public function testLastNameField()
{
// ...
}
How can I unit test (assert) each validation rule of every field in isolation and individually?
I found a good solution on Laracast and added some customization to the mix.
The Code
/**
* Test first_name validation rules
*
* #return void
*/
public function test_valid_first_name()
{
$this->assertTrue($this->validateField('first_name', 'jon'));
$this->assertTrue($this->validateField('first_name', 'jo'));
$this->assertFalse($this->validateField('first_name', 'j'));
$this->assertFalse($this->validateField('first_name', ''));
$this->assertFalse($this->validateField('first_name', '1'));
$this->assertFalse($this->validateField('first_name', 'jon1'));
}
/**
* Check a field and value against validation rule
*
* #param string $field
* #param mixed $value
* #return bool
*/
protected function validateField(string $field, $value): bool
{
return $this->validator->make(
[$field => $value],
[$field => $this->rules[$field]]
)->passes();
}
/**
* Set up operations
*
* #return void
*/
public function setUp(): void
{
parent::setUp();
$this->rules = (new UserStoreRequest())->rules();
$this->validator = $this->app['validator'];
}
Update
There is an e2e approach to the same problem. You can POST the data to be checked to the route in question and then see if the response contains session errors.
$response = $this->json('POST',
'/route_in_question',
['first_name' => 'S']
);
$response->assertSessionHasErrors(['first_name']);
I see this question has a lot of views and misconceptions, so I will add my grain of sand to help anyone who still has doubts.
First of all, remember to never test the framework, if you end up doing something similar to the other answers (building or binding a framework core's mock (disregard Facades), then you are doing something wrong related to testing).
So, if you want to test a controller, the always way to go is: Feature test it. NEVER unit test it, not only is cumbersome to unit test it (create a request with data, maybe special requirements) but also instantiate the controller (sometimes it is not new HomeController and done...).
They way to solve the author's problem is to feature test like this (remember, is an example, there are plenty of ways):
Let's say we have this rules:
public function rules()
{
return [
'name' => ['required', 'min:3'],
'username' => ['required', 'min:3', 'unique:users'],
];
}
namespace Tests\Feature;
use App\Models\User;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Tests\TestCase;
class HomeControllerTest extends TestCase
{
use RefreshDatabase;
/*
* #dataProvider invalid_fields
*/
public function test_fields_rules($field, $value, $error)
{
// Create fake user already existing for 'unique' rule
User::factory()->create(['username' => 'known_username']);
$response = $this->post('/test', [$field => $value]);
$response->assertSessionHasErrors([$field => $error]);
}
public function invalid_fields()
{
return [
'Null name' => ['name', null, 'The name field is required.'],
'Empty name' => ['name', '', 'The name field is required.'],
'Short name' => ['name', 'ab', 'The name must be at least 3 characters.'],
'Null username' => ['username', null, 'The username field is required.'],
'Empty username' => ['username', '', 'The username field is required.'],
'Short username' => ['username', 'ab', 'The username must be at least 3 characters.'],
'Unique username' => ['username', 'known_username', 'The username has already been taken.'],
];
}
}
And that's it... that is the way of doing this sort of tests... No need to instantiate/mock and bind any framework (Illuminate namespace) class.
I am taking advantage of PHPUnit too, I am using data providers so I don't need to copy paste a test or create a protected/private method that a test will call to "setup" anything... I reuse the test, I just change the input (field, value and expected error).
If you need to test if a view is being displayed, just do $response->assertViewIs('whatever.your.view');, you can also pass a second attribute (but use assertViewHas) to test if the view has a variable in it (and a desired value). Again, no need to instantiate/mock any core class...
Have in consideration this is just a simple example, it can be done a little better (avoid copy pasting some errors messages).
One last important thing: If you unit test this type of things, then, if you change how this is done in the back, you will have to change your unit test (if you have mocked/instantiated core classes). For example, maybe you are now using a FormRequest, but later you switch to other validation method, like a Validator directly, or an API call to other service, so you are not even validating directly in your code. If you do a Feature Test, you will not have to change your unit test code, as it will still receive the same input and give the same output, but if it is a Unit Test, then you are going to change how it works... That is the NO-NO part I am saying about this...
Always look at test as:
Setup minimum stuff (context) for it to begin with:
What is your context to begin with so it has logic ?
Should a user with X username already exist ?
Should I have 3 models created ?
Etc.
Call/execute your desired code:
Send data to your URL (POST/PUT/PATCH/DELETE)
Access a URL (GET)
Execute your Artisan Command
If it is a Unit Test, instantiate your class, and call the desired method.
Assert the result:
Assert the database for changes if you expected them
Assert if the returned value matches what you expected/wanted
Assert if a file changed in any desired way (deletion, update, etc)
Assert whatever you expected to happen
So, you should see tests as a black box. Input -> Output, no need to replicate the middle of it... You could setup some fakes, but not fake everything or the core of it... You could mock it, but I hope you understood what I meant to say, at this point...
Friends, please, make the unit-test properly, after all, it is not only rules you are testing here, the validationData and withValidator functions may be there too.
This is how it should be done:
<?php
namespace Tests\Unit;
use App\Http\Requests\AddressesRequest;
use App\Models\Country;
use Faker\Factory as FakerFactory;
use Illuminate\Routing\Redirector;
use Illuminate\Validation\ValidationException;
use Tests\TestCase;
use function app;
use function str_random;
class AddressesRequestTest extends TestCase
{
public function test_AddressesRequest_empty()
{
try {
//app(AddressesRequest::class);
$request = new AddressesRequest([]);
$request
->setContainer(app())
->setRedirector(app(Redirector::class))
->validateResolved();
} catch (ValidationException $ex) {
}
//\Log::debug(print_r($ex->errors(), true));
$this->assertTrue(isset($ex));
$this->assertTrue(array_key_exists('the_address', $ex->errors()));
$this->assertTrue(array_key_exists('the_address.billing', $ex->errors()));
}
public function test_AddressesRequest_success_billing_only()
{
$faker = FakerFactory::create();
$param = [
'the_address' => [
'billing' => [
'zip' => $faker->postcode,
'phone' => $faker->phoneNumber,
'country_id' => $faker->numberBetween(1, Country::count()),
'state' => $faker->state,
'state_code' => str_random(2),
'city' => $faker->city,
'address' => $faker->buildingNumber . ' ' . $faker->streetName,
'suite' => $faker->secondaryAddress,
]
]
];
try {
//app(AddressesRequest::class);
$request = new AddressesRequest($param);
$request
->setContainer(app())
->setRedirector(app(Redirector::class))
->validateResolved();
} catch (ValidationException $ex) {
}
$this->assertFalse(isset($ex));
}
}

Laravel 5 - Validate Multiple Request - do all Requests at the same time

Continuing discussion from here
If we have two Requests like:
public function store(FirstRequest $request, SecondRequest $request)
{ ... }
is it possible to run both Requests and not one after another. This way if validation doesn't pass for FirstRequest, SecondRequest won't start and it will create error messages only after FirstRequest passes without any errors.
I think that you can "manually creating validators"
http://laravel.com/docs/5.1/validation#other-validation-approaches
Basically in your method instead of use a a Request injection, use the rules directly in the method and call the $validator->fails() method for every set of rules.
Something like this:
public function store(Request $request){
$rulesFirstRequest = ['field1' => 'required', 'field2' => 'required'];
$rulesSecondRequest = ['field12' => 'required', 'field22' => 'required'];
$validator1 = Validator::make($request->all(), $rulesFirstRequest);
$validator2 = Validator::make($request->all(), $rulesSecondRequest);
if ($validator1->fails() && $validator2->fails()) {
//Do stuff and return with errors
}
// return with success
}
Hope it helps

Categories