I'm currently working on unit testing of a custom class I made, which is based on the singleton design pattern. Based on the code coverage report I have 95.45% of it covered. I am using PHPUnit to do the unit testing and I have been through this article
by Sebastian Bergmann.
The only problem I am left with is testing against class cloning throught the magic method __clone(). I have set that method as private to avoid instantiation
private final function __clone()
{}
What would be the best way to write a test to make sure that the singleton isn't "clonable". (The same test could eventually be used to test the __constructor())
Not really a question but is it just me or the tests runs awfully slow on a windows box compared to a *nix box?
Keep in mind that code coverage is not a measure of how correct your program is, nor does 100% coverage mean you've executed every code path. For example, the ternary operator
a ? b : c
and compound boolean expressions
if (a < 1 || b > 6)
are counted as single statements even though you may execute only a portion of them due to short-circuiting. Also, omitting the braces around single-statement if, while, etc. blocks turns the whole thing into a single statement.
The following will appear as a single statement in the code coverage report so you can't tell if you've executed both cases (true and false).
if (...)
foo();
I feel that
private final function __clone() { }
is too simple to fail. Testing that the method throws an exception (using reflection no less which your clients won't do) is testing the PHP interpreter--out of scope in my book.
[For the record, I too get a little OC when it comes to reaching 100% code coverage, but keeping the above facts in mind helps to alleviate it so I can move on to writing better code.]
Call clone or constructor and check if excpetion has been thrown.
Related
Given this code:
function testMe($a)
{
if ($a)
{
return 1;
}
else
{
return $this->testMe(true);
}
}
testMe() cannot be mocked, because then I cant call it. On the other hand, it must be mocked…
I'd say your question has some philosophical potential. I'll try to answer it the way you ask it but before, allow me to comment on your comprehension:
testMe() cannot be mocked, because then I cant call it. On the other hand, it must be mocked… [italics by me]
The unfinished sentence is wrong. You probably sensed it already because you didn't finish it. In a unit test you don't mock the unit. So you put the unit under test, and the unit is the method. That is the smallest part (unit) that can be tested.
What perhaps actually creates a bit of confusion is the recursion within the method.
So you actually ask how to unit-test recursion or a single method call within that recursion. But do you really need to test it?
I would say no. And that is because the recursion is an implementation detail of the method. From the outside it should not make any difference if you exchange the internal algorithm from recursion to a stack based loop for example.
But despite the fact I say you don't need that (and I hope you already have understood the argumentation I outlined), it technically is possible for your very specific scenario to test such a method-call without re-writing the code under test by re-binding $this. Then you can replace the subject under test ($this) with a mock when called. So that you have two methods: The one to test and the mocked one that is accessible via $this->testMe().
This could be done by instantiating the subject under test, create a mock, use PHP's reflection to obtain the closure of testMe() then re-bind $this on the closure to the mock and then call the closure for your test assertions.
I would not call this a unit-test any longer because as I outlined earlier, you're testing internals / privates, so you can use it more to actually test fragments of the recursion under certain circumstances and other similar detailed things that will more actually proof / debug fragments of code. You normally only need that under very specific circumstances when code is really high valuable.
Don't use your little confusion about the recursion to think this is an entirely important place to test on it's own. But if you want to get your fingers dirty, it's perhaps something worth to play with to learn about PHP reflection, closures and re-binding $this.
Description
I have a TestSuite which I need to mark as skipped (the entire test suite - not the specific test cases within the suite).
class AllTests
{
public static function suite()
{
// this does not work same as within TestCase:
// throw new \PHPUnit_Framework_SkippedTestError("Out of order");
$Suite = new \PHPUnit_Framework_TestSuite(__NAMESPACE__);
$Suite->addTestSuite(translators\AllTests::cls());
$Suite->addTestSuite(TlScopeTest::cls());
$Suite->addTestSuite(TlNsTest::cls());
$Suite->addTestSuite(TlElementTest::cls());
$Suite->addTestSuite(TlItemTest::cls());
$Suite->addTestSuite(LangCodeTest::cls());
$Suite->addTestSuite(TlElemClassTagTest::cls());
return $Suite;
}
}
As you can see throwing the PHPUnit_Framework_SkippedTestError exception does not work. It is not caught by the PHPUnit, and is breaks the execution as any uncaught exception (which is understandable, as the suite() method is invoked while building tests hierarchy, before actually running the tests).
I've seen an exception class named PHPUnit_Framework_SkippedTestSuiteError, but have no clue how to take advantage of it. Any ideas?
Motivation
I have a TestSuite, which aggregates many test cases as well as other test suites. Almost every test in this fails, becouse of a change which I made in the core of my code.
The problem is that this package is not crutial, and is scheduled to be fixed later. Until then I have to run tests for every other package, but when I do the PHPUnit output becomes flooded with the errors coming from the package in question. This forces me to check every time if any of the failures is coming from any other package.
This, as you might suspect, is very susceptible to human error, i.e. I could miss a failure, which actually is important.
I could run only the test suite on which I am currently working, but I lose control of whether or not my changes in one package causes a failure in other package.
I do not want to comment out that test suite, because I'm afraid that I (or someone who will take over the code after me) could forget about it entirely.
Ok, so I'll put it together:
The AllTests class has to be refactored to extend PHPUnit_Framework_TestSuite.
This makes the class a fully valuable TestSuite and allows to implement a setUp method on the suite level.
The setUp method is called by the test runner (not by the builder), so it is safe to throw a SkippedTestError exception.
The corresponding method to do just that within a test suite is called markTestSuiteSkipped (notice the Suite in the method name).
The entire class would look like this:
class AllTests extends \PHPUnit_Framework_TestSuite
{
protected function setUp()
{
$this->markTestSuiteSkipped("zzz");
}
public static function suite()
{
$Suite = new self(__NAMESPACE__);
$Suite->addTestSuite(translators\AllTests::cls());
$Suite->addTestSuite(TlScopeTest::cls());
$Suite->addTestSuite(TlNsTest::cls());
$Suite->addTestSuite(TlElementTest::cls());
$Suite->addTestSuite(TlItemTest::cls());
$Suite->addTestSuite(LangCodeTest::cls());
$Suite->addTestSuite(TlElemClassTagTest::cls());
return $Suite;
}
}
The output is a pretty block of S letters, which definetly indicate, that we skipped a lot of tests. This cannot escape our attention and yet allows our tests to pass.
You could mark test as skipped.
I am new to PHPUnit, infact I started today. And, as far as I have been reading, I came to understand only what this script does.
class UserTest extends PHPUnit_Framework_TestCase
{
protected $user;
// test the talk method
protected function setUp() {
$this->user = new User();
$this->user->setName("Tom");
}
protected function tearDown() {
unset($this->user);
}
public function testTalk() {
$expected = "Hello world!";
$actual = $this->user->talk();
$this->assertEquals($expected, $actual);
}
}
For this class:
<?php
class User {
protected $name;
public function getName() {
return $this->name;
}
public function setName($name) {
$this->name = $name;
}
public function talk() {
return "Hello world!";
}
}
Ok, so I have established that the test returns a Ok/Fail statement based on the equality of the test, but what I am looking for is more. I need an actually ways to test a more complex class that, its outcome, un-like in this example can not be guessed easily.
Say, I write a script that does polling. How would, or in what ways can I test if the methods/classes can work? The above code only shows if a methods outcome will only be 'Hello World' But, that is too easy thing to test, because I need to test complex things and there aren't much tutorials for that.
Testing a class in a unit test is supposed to NOT be complex.
And the reason for this to be true is that in a well designed system, you can test that class in isolation, and do not add the complexity of the system behind that class - because that system does not exist during the test, it is mocked.
The classic example for this is that a User class usually asks a database for info, but in the test case, that database is really hard to set up, prepare with data, and destroy afterwards. And using a real database also slows things down. So you design the User class to accept a database object from the outside, and in the test case you give to it a mock object of the database.
That way, it is really simple to simulate all kinds of return values from the database, and additionally you can check whether the database object gets the right parameters without the complexity of dealing with a real database.
You can only do proper mock object injection if your classes are designed to allow dependency injection. That is a principle to NOT create objects inside of other objects, but require the outside world to supply them. Just have a quick look at some video for explanation:
Dependency Injection - Programming With Anthony (Jan 2013)
And remember that creating good tests needs some experience. Good for you to start experimenting.
But, that is too easy thing to test, because I need to test complex things and there aren't much tutorials for that.
The more complex, the more complex the tests are. It's a bit like a snake biting in its own end. You normally want to prevent that, so to make it golden: Write simple tests to ensure that a complex software is tested and runs.
This does not always work 100% but it works better than without tests. PHPUnit has been designed for Unit-Tests (Xunit test patterns) but you can also use it to run different tests. For that you group tests. This is done differently by different people depending on different things. For example:
Small tests
Medium tests
Large tests
or (not equivalent):
Unit tests
Integration tests
Acceptance tests
or (perhaps equivalent TestPyramid):
Unit tests
Service tests
UI tests
and what not. When you start with testing, it's probably good to start with the unit-tests and as Sven already answered, keep them simple. Get comfortable with TDD, read some slides and books. And welcome to the world of automating tests.
What is Unit test, Integration Test, Smoke test, Regression Test?
P.S. Yes, simple getter setters are too easy to test. If all a setter does is storing to a private member and the getter gets it back, you can trust that PHP is working, writing a test for that is a waste of time and will only lead to cruft. It clearly shows that somebody wrote the test after the code has been written. Instead write first the test and see it fail (red), than hack together the code as quick as possible to get the test to pass (green). You can improve the code later on as the test already shows it's working.
I understand that 100% code coverage is just a goal to shoot for, but it's annoying to have a line containing a closing brace counted as not covered because it follows a method call whose sole purpose is to throw an exception. Here's a simple example from my base test case class to demonstrate:
function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped(); // [1] always throws an exception
} // [2] shown as executable but not covered
}
Since [1] always exits the method, line [2] is not actually reachable. Is there any way to tell Xdebug this by annotating the markTestSkipped() method itself?
Your pull request got merged so starting with php-code-coverage 1.1.2, which should come around rather soon (with PHPUnit 3.6.3 or 3.6.4) one will be able to write:
private static function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped();
} // #codeCoverageIgnore
}
Also in the further away future when xDebug will be able to provide 'Conditionals' coverage i think i remember discussion about making the whole issue going away with that refactoring as the closing brace will just count as 'covered' when the last statement in a function terminates the function... But I might be wrong on that
You can surround the line with stard/end comments to have PHP_CodeCoverage ignore it, but that means doing it everywhere the method is called.
function checkForSkipAllTests() {
if (self::$_skipAllTests) {
self::markTestSkipped();
// #codeCoverageIgnoreStart
}
// #codeCoverageIgnoreEnd
}
This is a maintenance nightmare and prone to error. I would really like to avoid this solution.
I understand that 100% code coverage is just a goal to shoot for, but it's annoying to have a line containing a closing brace counted as not covered because it follows a method call whose sole purpose is to throw an exception. Here's a simple example from my base test case class to demonstrate:
Indeed, 100% code coverage is not a goal, but it's nice to have, especially if it takes you zero time to make it so. I do wonder though; your tests are not the files that are to be tested. I never test my tests, nor am I interested in their code coverage. I already know which tests are done, which succeeded, which failed and which are skipped. This is what PHPUnit brings to the table for me; .....S...F is enough feedback.
My tests are in a separate directory, which isn't included in code coverage; it just seems useless to do so, in my eyes. Anyway, if you're sold on having code coverage reports on your testcases, you might want to simply get rid of the }, like so:
function checkForSkipAllTests() {
if (self::$_skipAllTests)
self::markTestSkipped();
}
Yeah, I know that having an if without curly brackets will make me the least cool person answering your question, but it seems like a much easier solution than having some annotations which magically work.
I would like to use PHP's assert function in my unit testing framework. It has the advantage of being able to see the expression being evaluated (including comments) within the error message.
The problem is that each method containing tests may have more than one assert statement, and I would like to keep track of how many actual assert statements have been run. assert does not give me a way to count how many times it has been run, only how many times it has failed (within the failure callback).
I tried to abstract the assert statement into a function so that I can add a counting mechanism.
private function assertTrue($expression) {
$this->testCount++;
assert($expression);
}
This does not work however because any variables within the expression are now out of scope.
$var = true;
$this->assertTrue('$var == true'); // fails
Any advice on how I can use assert in my unit testing while being able to count the number of actual tests?
The two ideas I have come up with are to make users count themselves
$this->testCount++;
assert('$foo');
$this->testCount++;
assert('$bar');
or make users put only one assert in each test method (I could then count the number of methods run). but neither of these solutions is very enforcable, and make coding more difficult. Any ideas on how to accomplish this? Or should I just strip assert() from my testing framework?
In PHPUnit, all of the assert*() methods take an additional $message parameter, which you can take advantage of:
$this->assertTrue($var, 'Expected $var to be true.');
If the assertion fails, the message is output with the failure in the post-test report.
This is more useful generally than outputting the actual expression because then you can comment on the significance of the failure:
$this->assertTrue($var, 'Expected result of x to be true when y and z.');
A bit of a cheeky answer here, but open vim and type:
:%s/assert(\(.+\));/assert(\1) ? $assertSuccesses++ : $assertFailures++;/g
(In principle, replace all assert() calls with assert() ? $success++ : $fail++;)
More seriously, providing a mechanism to count tests is really a responsibility a bit beyond the scope of the assert() function. Presumably you want this for an "X/Y tests succeeded" type indicator. You should be doing this in a testing framework, recording what each test is, its outcome and any other debug information.
You are restricted by the fact assert() must be called in the same scope the variables you are testing lie. That leaves -- as far as I can tell -- solutions that require extra code, modify the source before runtime (preprocessing), or a solution that extends PHP at the C-level. This is my proposed solution that involves extra code.
class UnitTest {
// controller that runs the tests
public function runTests() {
// the unit test is called, creating a new variable holder
// and passing it to the unit test.
$this->testAbc($this->newVarScope());
}
// keeps an active reference to the variable holder
private $var_scope;
// refreshes and returns the variable holder
private function newVarScope() {
$this->var_scope = new stdClass;
return $this->var_scope;
}
// number of times $this->assert was called
public $assert_count = 0;
// our assert wrapper
private function assert($__expr) {
++$this->assert_count;
extract(get_object_vars($this->var_scope));
assert($__expr);
}
// an example unit test
private function testAbc($v) {
$v->foo = true;
$this->assert('$foo == true');
}
}
Downfalls to this approach: all variables used in unit testing must be declared as $v->* rather than $*, whereas variables written in the assert statement are still written as $*. Secondly, the warning emitted by assert() will not report the line number at which $this->assert() was called.
For more consistency you could move the assert() method to the variable holder class, as that way you could think about each unit test operating on a test bed, rather than having some sort of magical assert call.
That's not something which unit-testing is intended to do (remember it originated in compiled langs).
And PHPs semantics do not help much with what you are trying to do either.
But you could accomplish it with some syntactic overhead still:
assert('$what == "ever"') and $your->assertCount();
Or even:
$this->assertCount(assert('...'));
To get the assertion string for succeeded conditions still, you could only utilize debug_backtrace and some heuristic string extraction.
This is not enforced much either (short of running a precompiler/regex over the test scripts). But I would look at this from the upside: not every check might be significant enough to warrant recording. A wrapper method thus allows opting out.
It's hard to give an answer without knowing how your framework has been built, but I'll give it a shot.
Instead of directly call the methods of your unit testing class ( methods like assertTrue() ), you could use the magic method of PHP __call(). Using this, you could increase an internal counter everytime assertTrue() method is called. Actually, you can do whatever you want, every time any method is called.
Remember that __call() is invoked if you try to call a method that does not exist. So you would've to change all your methods names, and call them internally from __call(). For instance, you'd have a method called fAssertTrue(), but the unit testing class would use assertTrue(). So since assertTrue() is not defined, __call() method would be invoked, and there you would call fAssertTrue().
Since you're passing the expression already (which might lead, correct me if I'm wrong, to quoting hell):
$this->assertTrue('$var == true'); // fails with asset($expression);
Why not add a tiny extra layer of complexity, and avoid the quoting hell, by using a closure instead?
$this->assertTrue(function() use ($var) {
return $var == true;
}); // succeeds with asset($expression());
Simple:
$this->assertTrue($var == true);
(without quotes!)
It will be evaluated in caller space, so assertTrue() will be passed just false or true.
As others have pointed out, this might not be the best way of testing, but that's another question entirely... ;)