I am quite new to unit testing and testing in general. I am developing with phpUnit, but as my question is more general / a design question, the actual environment shouldn't be of too much importance.
I assume, that it is a good practice, to write your testcases as specific as possible. For example (the later the better):
assertNotEmpty($myObject); // myObject is not Null
assertInternalType('array', $myObject); // myObject is an array
assertGreaterThan(0, count($myObject)); // myObject actually has entries
If that is correct, here is my question:
Is it a accepted practice to write some flow control inside a testcase, if the state of the object one is testing against depends on external sources (i.e. DB) - or even in general?
Like:
if (myObject !== null) {
if (count(myObject) > 0) {
// assert some Business Logic
}
else {
// assert some different Business Logic
}
}
Is this kind of flow control inside a testcase admissible or is a "code smell" and should be circumvented? If it is ok, are there any tips or practices, which should be kept in mind here?
Paul's answer addresses test method scope and assertions, but one thing your question's code implied is that you would test A if the returned object had value X but test B if it had value Y. In other words, your test would expect multiple values and test different things based on what it actually got.
In general, you will be a happier tester if each of your tests has a single, known, correct value. You achieve this by using fixed, known test data, often by setting it up inside the test itself. Here are three common paths:
Fill a database with a set of fixed data used by all tests. This will evolve as you add more tests and functionality, and it can become cumbersome to keep it up-to-date as things move. If you have tests that modify the data, you either need to reset the data after each test or rollback the changes.
Create a streamlined data set for each test. During setUp() the test drops and recreates the database using its specific data set. It can make writing tests easier initially, but you still must update the datasets as the objects evolve. This can also make running the tests take longer.
Use mocks for your data access objects when not testing those DAOs directly. This allows you to specify in the test exactly what data should be returned and when. Since you're not testing the DAO code, it's okay to mock it out. This makes the tests run quickly and means you don't need to manage data sets. You still have to manage the mock data, however, and write the mocking code. There are many mocking frameworks depending on your preference, including PHPUnit's own built-in one.
It's okay to have SOME control flow within your testcases, but in general, understand that your unit tests will work out best if they are disjoint, that is, they each test for different things. The reason this works out well is that when your test cases fail, you can see precisely what the failure is from the testcase that failed, as opposed to needing to go deeper inside a larger test case to see what went wrong. The usual metric is a single assertion per unit test case. That said, there are exceptions to every rule, and that's certainly one of them; there's nothing necessarily wrong with having a couple of assertions in your test case, particularly when the setup / teardown of the test case scenario is particularly difficult. However, the real code smell you want to avoid is the situation where you have one "test" which tests all the code paths.
Related
I know that me asking this question means that i probably didn't understand mocking fully. I know why using mocks (isolation) , i know how to use mocks (each FW and it's way) - but i don't understand - if the purpose of mocking is to return expected value to isolate my unit test, why should i mock an object and not just create the value by myself?
Why this:
$searchEngine = Mockery::mock('elasticSearch');
$searchEngine->shouldReceive('setupDataConnection')->once()->andReturn("data connected");
$insertData = new DataInserter;
$this->assertEquals("data connected",$insertData->insertData($searchEngine));
And not this:
$knownResult = "data connected";
$insertData = new DataInserter;
$this->assertEquals($insertData->insertData($searchEngine) ,$knownResult);
Edit
Sorry for the mistake, but the accidently didn't include the insertData in the second example.
by using a mock you have access to additional information, from a object that behaves directly like a C style struct ( only performs data storage ) then there is little difference (with the exception about making assertions about calls to your mocks that are usually rather useful since you can then make sure that your value is for example set 3 times, and not 7 (maybe with some intermediate value that could cause potential problems. )
Mocks are useful for testing the interactions between objects.
Now if your class does something more complex ( such as access a resource like a database, or read/write data from a file. ) then mocks become lifesaving because they pretty much allow you to abstract away the internal behavior of classes that are not under test (for example lets imagine that the first version of your class simply stores values in memory, the next version stores them in a specific place in a database, this way you can first of all save yourself the resource hit of a database access and secondly be able to effectively prove that your class under test works correctly as opposed to also proving that your data access class works. if you have a failed test then you can hone in on the issue immediately as opposed to trying to have to figure out if the issue is in your data provider or data accessor. )
Because tests can be ran in parallel certain resources can cause false failures.
A few examples of interactions which are extremely difficult to test without mocks would be:
Code which talks to a Database Access layer ( ensuring that queries are correct, close() is called on the connection, and appropriate SQL is sent, plus you don't modify a real database. )
File/Socket IO (again ensuring that data is consistent)
System calls ( e.g. calls to php exec)
anything that relies on threads / timing ( not as much of an issue in PHP)
GUI Code ( again, almost unheard of in PHP) but if I shift the language to lets say java its significantly easier to call :
when(myDialogBoxMock).showOptionInputDialog().thenReturn("mockInput")
then to try to fake it with subclasses or temporary implementations/subclasses.
Calls to specific methods which must be called with a specific value only.
verify(myMock.setValue(7), times(2));
verifyNoOtherInteractions(myMock);
A large part of this is also speed, if you are reading a file off the disk lets say 300/400 times in a massive codebase then you will definitely notice a speed increase by using mocks.
If you have a tool like EMMA/jacoco for java in PHP you will be able to have a effective code coverage report to show you where your tests are not covering code. And on any non-trivial application you will find yourself trying to figure out how the hell to get your object under test into a specific state to test specific behavior, Mocks with DI are really your tool to perform these tasks.
In the first example you are providing a fake collaborator ($searchEngine) for your DataInserter. In the assert statement you are calling insertData, which will execute the logic of your DataInserter that will in term interact with the fake $searchEngine. By providing the fake, you can verify that the logic of DataInserter works correctly in isolation.
In your second example, you are not testing the DataInserter logic. You are just checking whether the constant string $knownResult is equal to "data connected", i.e. you are testing string equality of php. As far as I can see from the snippet you are constructing a new DataInserter object but you aren't even exercising its code in the test.
I'm writing a PHPUnit test case for an API (so not exactly a unit test) and I'm thinking about having a test that all other tests will depend on.
The tests in the test case make API requests. Most of these requests require a user. The test in question will create that user that the other tests will use.
Would that be a horrible idea?
I think that the best way for unit tests is to eliminate the dependencies first.
You can abstract the end point with your own local version that will return predictible results. This way you can test that your requests are correct.
You can abstract the data providers (database, filesitem, etc...) with your stubs that will also return predictible data (username, etc..).
After that you just test your request and see they are correct..
The second part is to actualy test the data providers, with different tests, so you know that the good username will be given.
And then you can test the API connectivity, etc..
EDIT. If you have dependencies in your code, and it's difficult to abstract the providers or the end point web service, you may need to adjust your code so that it will accept references to those objects as parameters. Than in your tests you change the objects passed with your own stub objects. In production you pass the correct references, so that you will not need to change your code for testing.
I hope i have been clear. If not, ask me and i can explain better, maybe i did not understand your question well
I'm trying to understand how unit testing works. So far i understand you're testing the output of functions based on what you put in. Ok. Does that mean if your function has the only possibility of returning one datatype you only have to write one test for it? So say i write a function that only has the possibility of returning TRUE or FALSE, does that mean i would just be checking if the response is boolean?
And then also say i have a function that is pulling posts from a blog from a database. Say have the function set to: if number of rows = 0 return FALSE, else return the results. So i now i have the possibility of a function that could return a boolean value or an array. How the heck do you test that? The function now doesn't rely solely on the input, it relies on what's in the database.
A useful function does not return true or false randomly. It returns true under certain circumstances and false under other circumstances.
A unit test would therefore do three things (in that order):
Create an environment from which you know whether the function should return true or false (this environment is called the test fixture).
Run the function that is to be tested.
Compare the result of the function with you expectations.
Obviously this cannot be done with all possible inputs (if it could be done, this would be a proof, not a test), so the art of unit testing is choosing appropriate test fixtures. There are two ways to go about this:
Black box testing looks at the specification of the function to choose the fixtures: what are the valid input values and how should they influence the output?
White box testing looks into the function and chooses fixtures such that every statement of the function is executed by at least one fixture.
... if number of rows = 0 return FALSE, else return the results.
If such a function directly sends a query to a database server by calling an API of the DBMS (e.g. mysql_query), the creation of the test fixture would include
Setting up a database.
Inserting rows into the database (or not, depending on whether the test should check that the results are returned properly or FALSE is returned properly).
If, on the other hand, that function calls a custom function (e.g. provided by an object-relational mapper), then the test fixture should provide a mock implementation of that custom function. Dependency injection helps to ensure lose coupling of caller and callee, so the real implementation can be replaced with the mock implementation at runtime.
You have the gist of it about right. If there are external depecnencies for the unit there are two approaches. creating a Mock class or using some kind of fixture.
Using your DB example if youre testing a class that has to call on a data access class to pull data from the database then you would typically Mock that data access class, making the methods you plan on calling from it return the response you expect.
If on the other hand you were actually testing the data access class itself then you would use a fixture - a test db filled with data you know. And then test that the data access class returned the right data.
I think you need to do a bit of reading about unit testing. To answer your first question about the cases you should test, I suggest you read up on Boundary-Value testing and equivalence partitioning. in your case, yes you assert that the output is true or false, but are there multiple pre-conditions that can lead to the same result? If so, these need testing.
Regarding your second question where the return value depends on the database. This is strictly not a unit test (i.e. it involves multiple units). I suggest you spend some time reading about mock objects.
Unit testing just means that you're getting the outputs you expect. Write a test for every conceivable, unique state your program could be in and ensure the output of your function is what you expect.
For example, you could test in the case that your database is empty, a few rows, or full.
Then for each state, give different inputs (a couple invalid, a couple valid).
Unit testing just breaks up your programs into units and makes sure each unit works as expected.
EDIT:
Wikipedia has a great article on unit testing.
For unit testing you are trying to isolate and test certain pieces of code. You do this by providing the inputs such that you know what the answer should be if the code is working properly. If you are testing whether a function returns a boolean correctly then you need to provide inputs that you know will evaluate to say "false", then check to ensure that the function did indeed return "false". You will also want to write a test that should evaluate to "true". In fact, write as many tests with as many different inputs as you like until you are convinced that the function is working as expected.
There seems to be mixed opinions when it comes to unit testing with a database. Some argue that if you are using a database that you are performing integration testing. Regardless, for a unit test you should know what the input is and know what the expected answer should be. Your explanation of what you want to test doesn't align with what unit testing is really for. Once suggestion would be to create a test database that always contains the same data each time you run the unit test. This way you always know what the input is and know what the output should be. You could accomplish this by creating an in-memory database such as Hypersonic.
Hope this helps!
If you allow me a suggestion, take this book: THE ART OF UNIT TESTING, by Roy Osherove. http://amzn.com/1933988274. It's not a huge weapon-massive-like book.
I had never read anything about UnitTesting prior to this book, but now it's very clear how UnitTest works.
Hope it helps!
I should point out that unit testing is not just about what results your functions give out; it's also about how they mutate / impact other objects, as well. In a sense, they test what messages are passed between objects.
I'm in the process of testing a factory class.
One of the methods must load the data in an array for an object that another method will instantiate.
That method contains the SQL query that holds a critical condition that must be tested. ( in this case only ask for the records that are "published". Ex.: WHERE published=1 ). That distinction in the SQL Query is the only detail that makes that method differ from another one, and I want to test the query execution behaviour.
Now, I can't really mock my PDO object and ask it to return a fixed result as I would not test the execution of the query by mySQL. That would make a useless test.
That leads me to think that I'll need to set up a static database with fixed test data inside it. Am I right on this or have I missed something?
Should I separate the test requiring the "test database" from the tests that are autonomous ?
I strongly agree on the part of not mocking out the PDO. At some point i want to make sure my queries work against a real database. While this might not be a unit test anymore, technically. For me it offers so much more to know that my code that handles data storage really works against the db.
What i tend to do is creating a sort of Data Access class for each Class that needs to talk to the DB and so separating out most of the business logic from the db access code.
That way i can mock out the data access when testing the classes and after that set up a "Test Database" for each "data access class" and see if those work.
#zerkms Answer (+1) already linked http://phpunit.de/manual/current/en/database.html and the only other resource i have found to be of value when it comes to DB-Testing is the Book Real-World Solutions for Developing High-Quality PHP Frameworks and Applications that has a big chapter covering this subject.
Should I separate test requiring the "test database" from the tests that are autonomous ?
Only if your testsuite gets really really big and you have runtime issues that force you to say "hitting even a testing database just takes to long for all my tests so i run those only on a continuous integration server and not while developing.
Yes, it is the common practice.
It is also the special methods for loading fixtures: http://phpunit.de/manual/current/en/database.html
What about second question: no, you should not separate them. Your tests should test the behaviour, not the details of implementation. Join test cases into test classes by logic.
Given the following methods:
public function setFoo($foo) {
$this->_foo = $foo;
return $this;
}
public function getFoo() {
return $this->_foo;
}
Assuming, they may be changed to be more complex in the future:
How would you write unit tests for those methods?
Just one test method?
Should I skip those tests?
What about code coverage?
How about #covers annotation?
Maybe some universal test method to implement in the abstract test case?
(I use Netbeans 7)
This seems like a waste of time, but I wouldn't mind if IDE would generate those test methods automatically.
To qoute from the comment of Sebastian Bergman's blog:
(it's like testing getters and setters -- fail!). In any case, if they were to fail; wouldn't the methods that depend on on them fail?
So, what about the code coverage?
If you do TDD you should write a test for getter and setter. too. Do not
write a single line of code without a test for it - even if your code is
very simple.
Its a kind of religious war to use a tandem of getter and setter for
your test or to isolate each by accessing protected class members using
your unit test framework capabilities. As a black box tester i prefer to
tie my unit test code to the public api instead of tie it to the
concrete implementation details. I expect change. I want to encourage
the developers to refactor existing code. And class internals should not
effect "external code" (unit tests in this case). I don want to break
unit tests when internals change, i want them to break when the public
api changes or when behavior changes. Ok, ok, in case of a failing unit
test do not pin-point to the one-and-only source of problem. I do have
to look in the getter AND the setter to figure out what caused the
problem. Most of the time your getter is very simple (less then 5 lines
of code: e.g. a return and an optional null-check with an exception). So
checking this first is no big deal and not time consuming. And checking
the happy path of the setter is most of the time only a little more
complex (even if you have some validation checks).
Try to isolate your test cases - write a test for a SUT (Subject under
test) that validates its correctness without reley on other methods
(except my example above). The more you isolate the test, the more your
tests spot the problem.
Depending on your test strategy you may be want to cover happy path only
(pragmatic programmer). Or sad pathes, too. I prefer to cover all
execution pathes. When i think i discovered all execution pathes i check
code coverage to identify dead code (not to identify if there are
uncovered execution pathes - 100% code coverage is a missleading indicator).
It is best practice for black box testers to use phpunit in strict mode
and to use #covers to hide collateral coverage.
When you write unit test your test on class A should be executed independent from class B. So your unit tests for class A should not call / cover method of class B.
If you want to identify obsolete getter/setter and other "dead" methods (which are not used by production code) use static code analysis for that. The metric you are interested in is called "Afferent coupling at method level (MethodCa)". Unfortunately this metric (ca) is not available at method-level in PHP Depend (see: http://pdepend.org/documentation/software-metrics/index.html and http://pdepend.org/documentation/software-metrics/afferent-coupling.html). If you realy need it, feel free to contribute it to PHP Depend. An option to exclude calls from the same class would be helpful to get a result without "collateral" calls. If you identify a "dead method" try to figure out if it is meant to be used in near future (the counterpart for an other method that has a #depricated annotation) else remove it. In case it is used in the same class only, make it privat / protected. Do not apply this rule to library code.
Plan B:
If you have acceptance tests (integration test, regression test, etc.) you can run that test without running unit tests at the same time and without phpunits strict mode. This can result in a very similar code coverage result as if you had analysed your production code. But in most cases your non-unit tests are not as strong as your production code is. It depends on your discipline if this plan B is "equal enought" to production code to get a meaningful result.
Further reading:
- Book: Pragmatic Programmer
- Book: Clean Code
Good Question,
i usually try not to test getters&setters directly since i see a greater benefit in testing only the methods that actually do something.
Especially when not using TDD this has the added benefit of showing me setters that i don't use in my unittests showing me that ether my tests are incomplete or that the setter is not used/needed. "If i can execute all the "real" code without using that setter why is it there."
When using fluent setter i sometimes write a test checking the 'fluent' part of the setters but usually that is covered in other tests.
To answer your list:
Just one test method?
That is my least favorite option. All or none. Testing only one is not easy for other people to understand and looks 'random' or needs to be documented in a way.
Edit after comment:
Yes, for "trivial" get/set testing I'd only use one method per property maybe depending on the case even only one method for the whole class (for value objects with many getters and setters I don't want to write/maintain many tests)
How would you write unit tests for those methods?
Should I skip those tests?
I wouldn't skip them. Maybe the getters depending on how many you have (i tend to write only getters i actually need) but the task of having a class completely covered shouldn't fail because of getters.
What about code coverage?
How about #covers annotation?
With #covers my take is always "use it everywhere or don't use it at all". Mixing the two 'styles' of testing takes away some of the benefits of the annotation and looks 'unfinished' to me.
Maybe some universal test method to implement in the abstract test case?
For something like value objects that could work nicely. It might break (or gets more complicated) once you pass in objects / array with type hinting but I'd presonally prefer it over writing manual tests for 500 getters and setters.
This is a common question but strangely can't find a dupe on SO.
You could write unit tests for accessors but the majority of practioners do not. i.e. if the accessors do not have any custom logic, I would not write unit tests to verify if field access works.
Instead I would rely on the consumers of these accessors to ensure that the accessors work. e.g. If getFoo and setFoo don't work, the callers of these method should break. So by writing unit tests for the calling methods, the accessors get verified.
This also means that code coverage should not be a problem. If you find accessors that are not covered after all test suites are run, maybe they are redundant / unused. Delete them.
Try to write a test that illustrates a scenario where a client will use that accessor. e.g. The below snippet shows how the Tooltip (property) for the Pause Button toggles based on its current mode.
[Test]
public void UpdatesTogglePauseTooltipBasedOnState()
{
Assert.That(_mainViewModel.TogglePauseTooltip, Is.EqualTo(Strings.Main_PauseAllBeacons));
_mainViewModel.TogglePauseCommand.Execute(null);
Assert.That(_mainViewModel.TogglePauseTooltip, Is.EqualTo(Strings.Main_ResumeAllBeacons));
_mainViewModel.TogglePauseCommand.Execute(null);
Assert.That(_mainViewModel.TogglePauseTooltip, Is.EqualTo(Strings.Main_PauseAllBeacons));
}