I understand that 100% code coverage is just a goal to shoot for, but it's annoying to have a line containing a closing brace counted as not covered because it follows a method call whose sole purpose is to throw an exception. Here's a simple example from my base test case class to demonstrate:
function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped(); // [1] always throws an exception
} // [2] shown as executable but not covered
}
Since [1] always exits the method, line [2] is not actually reachable. Is there any way to tell Xdebug this by annotating the markTestSkipped() method itself?
Your pull request got merged so starting with php-code-coverage 1.1.2, which should come around rather soon (with PHPUnit 3.6.3 or 3.6.4) one will be able to write:
private static function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped();
} // #codeCoverageIgnore
}
Also in the further away future when xDebug will be able to provide 'Conditionals' coverage i think i remember discussion about making the whole issue going away with that refactoring as the closing brace will just count as 'covered' when the last statement in a function terminates the function... But I might be wrong on that
You can surround the line with stard/end comments to have PHP_CodeCoverage ignore it, but that means doing it everywhere the method is called.
function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped();
// #codeCoverageIgnoreStart
}
// #codeCoverageIgnoreEnd
}
This is a maintenance nightmare and prone to error. I would really like to avoid this solution.
I understand that 100% code coverage is just a goal to shoot for, but it's annoying to have a line containing a closing brace counted as not covered because it follows a method call whose sole purpose is to throw an exception. Here's a simple example from my base test case class to demonstrate:
Indeed, 100% code coverage is not a goal, but it's nice to have, especially if it takes you zero time to make it so. I do wonder though; your tests are not the files that are to be tested. I never test my tests, nor am I interested in their code coverage. I already know which tests are done, which succeeded, which failed and which are skipped. This is what PHPUnit brings to the table for me; .....S...F is enough feedback.
My tests are in a separate directory, which isn't included in code coverage; it just seems useless to do so, in my eyes. Anyway, if you're sold on having code coverage reports on your testcases, you might want to simply get rid of the }, like so:
function checkForSkipAllTests() {
if (self::$_skipAllTests)
self::markTestSkipped();
}
Yeah, I know that having an if without curly brackets will make me the least cool person answering your question, but it seems like a much easier solution than having some annotations which magically work.
Related
I have following method:
MyClass {
public function __construct(array $data, ?SomeObject $object): void
}
I have following test:
public function testWrongDataTypeThrowsErrorException(): void
{
$this->expectException(TypeError::class);
//works as expected
new MyClass(null, new SomeObject);
//this is never called
new MyClass('string', new SomeObject);
}
And what happens is it only runs 1 test/assertion with the null parameter. The second time when I try to create MyClass with string, it never gets there. Why is that?
The same logic works if the code throws regular Exception.
I can write 2 separate methods instead, but thats very inconvenient.
Or is it stupid to test for this? My idea is that I want to test that the method accept array and only array. So if somebody would remove it from the method the test would fail.
Place the expectExeception... or exepectError... only directly before the action you want to assert it for.
If you need multiple, write multiple tests methods - for each exception and error individually.
The suggestion above is how it works with Phpunit out of the box and normally keeps your tests clean.
There is however also the option to just make use of PHP here and handle it like the following:
try {
new MyClass(null, new SomeObject);
$this->fail('an expected exception was not thrown');
} catch (TypeError $e) {
$this->addToAssertionCount(1);
}
Then the test-method is not interrupted and you can continue (you have to remove the earlier TypeError expectation as Phpunit will not detect it any longer).
If you feel this is more convenient than separated tests, go for it.
You can even add a helper function and pass a closure in so you can wrap it. Normally it's better to stick with clean tests IMHO, if you really must test for that.
From what you write in your question, I'd say the problem is solved in PHP since the array type-hint back in PHP 5.??? and I would not need to test for that (or you would also need to write tests for your tests, because also the tests can be changed).
If someone changes the method signature, they should have a valid reason for that. If they make a mistake, they perhaps didn't cover their refactoring with tests so the problem lies not in the code of the unit and is not to test for with unit tests.
Keep in mind, that a test should only have one reason to fail. So it's good to aim for that when writing a (unit) test.
Is it OK to have multiple asserts in a single unit test?
Summary
I have a class that uses debug_backtrace() for reporting where a method was called. The method may be called in third party code either directly or via some helper method that's in the same package as the class in question. In the latter case the method will report the line from the third party code and not from the helper class, thus the index at which the actual caller is in the results varies, so I'd like to include a test in my PHPUnit suite which would ensure that the correct index is found in each use case.
Is there a way I could automatically and reliably figure out the correct line in the test file that my method should return?
tl;dr, code plz
Here's some code to demonstrate. This is just the most bare bones example of the situation, please disregard that it doesn't really make sense. The actual use case is more complex and it's very much possible, likely even, that at some point in time I will mess something up and the traces end up giving wrong results. I'd like my tests to catch that.
Classes.php:
<?php
class Tracer
{
public function send()
{
$trace = debug_backtrace();
if ($trace[1]['class'] === 'Helper') {
$calledOnLine = $trace[2]['line'];
} else {
$calledOnLine = $trace[1]['line'];
}
return $calledOnLine;
}
}
class Helper
{
public static function send()
{
$Tracer = new Tracer;
return $tracer->send();
}
}
TracerTest.php:
<?php
class TracerTest extends PHPUnit_Framework_TestCase
{
public function testGetProperCallerInfo()
{
$Tracer = new Tracer;
$result1 = $Tracer->send(); // $TargetLine1
$result2 = Helper::send(); // $TargetLine2
$this->assertEquals($TargetLine1, $result1);
$this->assertEquals($TargetLine2, $result2);
}
}
Ideas
I could hardcode the line numbers into the test, and then update them each and every time something higher up in the test file changes. Even if I separate this particular test to its own file it's still going to break at some point, and besides it's such an ugly solution I wouldn't be able to look myself in the eye anymore.
Another method might be to tag the target lines with comments, load the test file to an array with file() and find the tags in the array. This is still somewhat fragile, and not really all that elegant.
Ask yourself, what is the behavior you are trying to test?
If you want to ensure you are generating exact strings, then maybe test fixtures is a good idea.
Probably more easier to maintain is to assert parts of the string which can be easily inferred.
Alternatively, you can mock the function call itself with libraries such as
https://github.com/instaclick/ICBaseTestBundle/blob/master/Test/Helper/Unit/FunctionHelper.php
http://www.workhabit.com/labs/mock-function-testing-drupal
Since I haven't been able to find a better solution, I ended up putting together a small package that let's me find lines reliably from the test source.
Accepted answer is still up for grabs if you can come up with a better solution!
In case someone is interested, the package I made is here.
I am writing a Zend Framework application and unit testing it with PHPUnit. In general, things are going swimmingly however I have a small, but annoying issue with PHPUnit and code coverage - it sometimes tells me that a particular line is not tested, and I don't know how to force it to be tested.
In the following code, for example, I launch two tests: one with a GET request, one with a POST request. The tests pass, and that's all fine. When I look at the code coverage, however, it shows me that the 'else' line is not executed.
public function editAction()
{
if ($request->isPost()) {
// do actions related to POST
} else {
// do action related to GET
}
}
Any ideas? As a side issue, do you usually persevere with unit tests until you get 100% code coverage? Or is this not really practical?
Thanks muchly...
I was the project lead on the Zend Framework a few years ago, through the release of ZF 1.0. I worked pretty hard on raising the coverage of testing for all components, and we had a policy that a component must have a certain minimum code coverage to be adopted into ZF from the incubator.
However, you're right, trying to get 100% code coverage from tests for all your classes isn't really practical. Some of the classes in ZF have 100% coverage, but for these, one or more of the following was true:
The class was trivially simple.
The tests took extraordinary work to write. E.g. complex setup code to create conditions to exercise all the obscure corner cases. Look at the unit tests for Zend_Db that I wrote! Though it's beneficial to force yourself to test these corner cases, because I guarantee that it'll lead you to code that you need to fix.
The class had to be refactored to be more "testable". This is often a good thing anyway, because you end up with better OO code, with less coupling, fewer static elements, etc. See the classes and tests for Zend_Log.
But we also realized that 100% code coverage is sometimes an artificial goal. A test suite that achieves less tan 100% coverage may nevertheless be adequate. And a test suite that does get to 100% coverage doesn't necessarily assure quality.
It would be very easy to get 100% code coverage for the following function. But did we test for division by zero?
function div($numerator, $denominator) {
return $numerator / $denominator;
}
So you should use code coverage as one metric of testing, but not the ultimate goal.
The code you have only comments for is what matters. The closing brace of a block will be shown as executable in the code coverage report if it's possible to fall through to the end. Look for a branch that you aren't actually testing.
if ($request->isPost()) {
if ($x < 5) {
return '<';
}
elseif ($x > 5) {
return '>';
}
// Do you have a test for $x == 5?
}
As for the 100% code coverage goal, I agree wholeheartedly with Bill. For some framework classes that I write I will strive to make 100%, but I know that doesn't mean I've truly tested every possibility. Often, when I find myself working overly hard to achieve 100% coverage, it's probably OCD kicking in. :)
Just . . . one . . . more . . . test . . .
If that is all there is to your test than i would assume your tests looks like Matthew described:
class UserControllerTest extends Zend_Test_PHPUnit_ControllerTestCase {
// [...]
public function testSomething()
{
$this->request
->setMethod('POST')
->setPost(array(
'username' => 'foobar',
'password' => 'foobar'
));
$this->editAction();
// assertThatTheRightThingsHappend
}
}
and in that case i don't see any reason why it shouldn't be easy to get 100% Code Coverage.
But yes: It is pretty hard to test Zend Framework Controllers and at some point you either have to try really hard to get all your application logic out of your controllers or just live with it.
The same thing doesn't apply for your models though. Those should be really easy to test, even in a ZF Application.
The purpose that code coverage serves is that it tells you what parts of your code base are not even getting executed. It doesn't tell you what is really tested and can only serve as a "minimum" to get an idea about the quality of your test suite (if you don't use #covers even that might lie to you).
On short: If you have big controllers and an not so easy to change architecture just setting with so and so tested controllers but don't apply the same logic to your models. Nothing in ZF prevents you to properly test those
I would like to use PHP's assert function in my unit testing framework. It has the advantage of being able to see the expression being evaluated (including comments) within the error message.
The problem is that each method containing tests may have more than one assert statement, and I would like to keep track of how many actual assert statements have been run. assert does not give me a way to count how many times it has been run, only how many times it has failed (within the failure callback).
I tried to abstract the assert statement into a function so that I can add a counting mechanism.
private function assertTrue($expression) {
$this->testCount++;
assert($expression);
}
This does not work however because any variables within the expression are now out of scope.
$var = true;
$this->assertTrue('$var == true'); // fails
Any advice on how I can use assert in my unit testing while being able to count the number of actual tests?
The two ideas I have come up with are to make users count themselves
$this->testCount++;
assert('$foo');
$this->testCount++;
assert('$bar');
or make users put only one assert in each test method (I could then count the number of methods run). but neither of these solutions is very enforcable, and make coding more difficult. Any ideas on how to accomplish this? Or should I just strip assert() from my testing framework?
In PHPUnit, all of the assert*() methods take an additional $message parameter, which you can take advantage of:
$this->assertTrue($var, 'Expected $var to be true.');
If the assertion fails, the message is output with the failure in the post-test report.
This is more useful generally than outputting the actual expression because then you can comment on the significance of the failure:
$this->assertTrue($var, 'Expected result of x to be true when y and z.');
A bit of a cheeky answer here, but open vim and type:
:%s/assert(\(.+\));/assert(\1) ? $assertSuccesses++ : $assertFailures++;/g
(In principle, replace all assert() calls with assert() ? $success++ : $fail++;)
More seriously, providing a mechanism to count tests is really a responsibility a bit beyond the scope of the assert() function. Presumably you want this for an "X/Y tests succeeded" type indicator. You should be doing this in a testing framework, recording what each test is, its outcome and any other debug information.
You are restricted by the fact assert() must be called in the same scope the variables you are testing lie. That leaves -- as far as I can tell -- solutions that require extra code, modify the source before runtime (preprocessing), or a solution that extends PHP at the C-level. This is my proposed solution that involves extra code.
class UnitTest {
// controller that runs the tests
public function runTests() {
// the unit test is called, creating a new variable holder
// and passing it to the unit test.
$this->testAbc($this->newVarScope());
}
// keeps an active reference to the variable holder
private $var_scope;
// refreshes and returns the variable holder
private function newVarScope() {
$this->var_scope = new stdClass;
return $this->var_scope;
}
// number of times $this->assert was called
public $assert_count = 0;
// our assert wrapper
private function assert($__expr) {
++$this->assert_count;
extract(get_object_vars($this->var_scope));
assert($__expr);
}
// an example unit test
private function testAbc($v) {
$v->foo = true;
$this->assert('$foo == true');
}
}
Downfalls to this approach: all variables used in unit testing must be declared as $v->* rather than $*, whereas variables written in the assert statement are still written as $*. Secondly, the warning emitted by assert() will not report the line number at which $this->assert() was called.
For more consistency you could move the assert() method to the variable holder class, as that way you could think about each unit test operating on a test bed, rather than having some sort of magical assert call.
That's not something which unit-testing is intended to do (remember it originated in compiled langs).
And PHPs semantics do not help much with what you are trying to do either.
But you could accomplish it with some syntactic overhead still:
assert('$what == "ever"') and $your->assertCount();
Or even:
$this->assertCount(assert('...'));
To get the assertion string for succeeded conditions still, you could only utilize debug_backtrace and some heuristic string extraction.
This is not enforced much either (short of running a precompiler/regex over the test scripts). But I would look at this from the upside: not every check might be significant enough to warrant recording. A wrapper method thus allows opting out.
It's hard to give an answer without knowing how your framework has been built, but I'll give it a shot.
Instead of directly call the methods of your unit testing class ( methods like assertTrue() ), you could use the magic method of PHP __call(). Using this, you could increase an internal counter everytime assertTrue() method is called. Actually, you can do whatever you want, every time any method is called.
Remember that __call() is invoked if you try to call a method that does not exist. So you would've to change all your methods names, and call them internally from __call(). For instance, you'd have a method called fAssertTrue(), but the unit testing class would use assertTrue(). So since assertTrue() is not defined, __call() method would be invoked, and there you would call fAssertTrue().
Since you're passing the expression already (which might lead, correct me if I'm wrong, to quoting hell):
$this->assertTrue('$var == true'); // fails with asset($expression);
Why not add a tiny extra layer of complexity, and avoid the quoting hell, by using a closure instead?
$this->assertTrue(function() use ($var) {
return $var == true;
}); // succeeds with asset($expression());
Simple:
$this->assertTrue($var == true);
(without quotes!)
It will be evaluated in caller space, so assertTrue() will be passed just false or true.
As others have pointed out, this might not be the best way of testing, but that's another question entirely... ;)
I'm currently working on unit testing of a custom class I made, which is based on the singleton design pattern. Based on the code coverage report I have 95.45% of it covered. I am using PHPUnit to do the unit testing and I have been through this article
by Sebastian Bergmann.
The only problem I am left with is testing against class cloning throught the magic method __clone(). I have set that method as private to avoid instantiation
private final function __clone()
{}
What would be the best way to write a test to make sure that the singleton isn't "clonable". (The same test could eventually be used to test the __constructor())
Not really a question but is it just me or the tests runs awfully slow on a windows box compared to a *nix box?
Keep in mind that code coverage is not a measure of how correct your program is, nor does 100% coverage mean you've executed every code path. For example, the ternary operator
a ? b : c
and compound boolean expressions
if (a < 1 || b > 6)
are counted as single statements even though you may execute only a portion of them due to short-circuiting. Also, omitting the braces around single-statement if, while, etc. blocks turns the whole thing into a single statement.
The following will appear as a single statement in the code coverage report so you can't tell if you've executed both cases (true and false).
if (...)
foo();
I feel that
private final function __clone() { }
is too simple to fail. Testing that the method throws an exception (using reflection no less which your clients won't do) is testing the PHP interpreter--out of scope in my book.
[For the record, I too get a little OC when it comes to reaching 100% code coverage, but keeping the above facts in mind helps to alleviate it so I can move on to writing better code.]
Call clone or constructor and check if excpetion has been thrown.