I need to skip all the tests in the following cest class and currently I am skipping individual tests with #skip annotation. Is there any way I can skip execution at the class level itself, instead of individually skipping each test?
Currently I am skipping individual tests like this:
Class MyTests{
/**
* #skip Skip message
*/
public funtion test1(){
// Test steps
}
/**
* #skip Skip message
*/
public function test2(){
// Test steps
}
}
I tested it with Codeception 5 and #skip message works above class too.
I discovered a number of related issues and raised https://github.com/Codeception/Codeception/issues/6615
Related
I am trying to set test expectations on a mock object that is created in a data provider and passed to my test method. This is useful because I can reused my data provider across different test cases and have the tests define what to expect on the mock. However, phpunit marks this test as risky when the case passes, but correctly fails the test when it does not pass. Is this a known thing that cannot be done?
I am using phpunit v9.3
Here is a contrived example to show the problem:
<?php
use PHPUnit\Framework\MockObject\MockObject;
use PHPUnit\Framework\TestCase;
class Test extends TestCase
{
public function provideMock(): array
{
return [
[$this->createMock(\DateTime::class)],
];
}
/** #dataProvider provideMock */
public function testMockPasses(MockObject $mock): void
{
$mock->expects($this->once())->method('format')->with('Y-m-d');
$mock->format('Y-m-d');
}
/** #dataProvider provideMock */
public function testMockFails(MockObject $mock): void
{
$mock->expects($this->once())->method('format')->with('Y-m-d');
$mock->format('Y-m-');
}
}
I would expect this to work fine as I am just passing the object to the method - all internal php stuff.
Try running PHPUnit with --verbose key - it may tell more about the reason of marking the test as risky.
We are using phpunit to unit-test our classes like this
namespace VendorName/SystemName {
final class TestableClass { /** ... */ }
}
namespace VendorName/SystemName/Tests {
/**
* #group unit
* #covers /VendorName/SystemName/TestableClass
*/
final class TestableClassTest extends TestCase { /** ... */ }
}
When we run these test the TestableClass get covered correctly, but we also want to see, whether every line of code in TestableClassTest was actually exectued, as we deal with dead code sometimes like
/**
* #expectedException /InvalidArgumentException
*/
public function testSomeMethodThrowsException(): void
{
// do some arrangement
// call the tested method
$testable->someMethod();
// do some post-call assertions
self::assertBar($baz);
self::assertFoo($baz);
// lines above are actually dead code
// if test goes right and throws exception
}
So we wan't to grab coverage statistics for our tests also.
The straightforward way is to add #covers /VendorName/SystemName/Tests/TestableClassTest to each test class, but this requires a lot of manual work.
Is there a correct way to automatically get test cases coverage (maybe listener or config or other phpunit hacks)
I am using php for testing my code. Here is an example:
/**
* #covers Calculator::
*/
class CalculatorTest extends PHPUnit_Framework_TestCase {
protected function setUp() { /* ... */ }
/**
* #covers Calculator::add
*/
public function testAddTwoIntegers() { /* ... */ }
/**
* #covers Calculator::multiply
*/
public function testMultiplyTwoIntegers() { /* ... */ }
}
However, my code is complicated and I want to get rid of #covers over individual test methods. How does php treat the following class when generating coverage reports:
/**
* #covers Calculator
*/
class CalculatorTest extends PHPUnit_Framework_TestCase {
protected function setUp() { /* ... */ }
public function testAddTwoIntegers() { /* ... */ }
public function testMultiplyTwoIntegers() { /* ... */ }
}
Notice that the #covers annotation over a class is still there but I have removed it from every method.
I am able to get coverage reports using this approach too but I have not seen an example of it anywhere, so I want to know if this is not a correct use.
Code Coverage Report:
2016-01-18 08:57:50
Summary:
Classes: 17.67% (56/317)
Methods: 0.33% (5/1520)
Lines: 0.60% (109/18094)
Class1:
Methods: 66.67% ( 2/ 3) Lines: 95.45% ( 21/ 22)
Class2:
Methods: 50.00% ( 3/ 6) Lines: 96.70% ( 88/ 91)
Indeed, setting the classname without the method name is the recommended #covers syntax.
Documentation here:
https://phpunit.readthedocs.io/en/8.5/annotations.html#covers
Annotation
#covers ClassName::methodName (not recommended)
#covers ClassName (recommended)
#covers ClassName<extended> (not recommended)
#covers ClassName::<public> (not recommended)
#covers ClassName::<protected> (not recommended)
#covers ClassName::<private> (not recommended)
#covers ClassName::<!public> (not recommended)
#covers ClassName::<!protected> (not recommended)
#covers ClassName::<!private> (not recommended)
#covers ::functionName (recommended)
When you ask:
How does php treat the following class when generating coverage reports
The answer is: Every line of the class mentioned in #covers executed during "any" of the tests in that TestCase will compute a +1 in the execution.
If you do real TDD and start first by writing the test, then writing the class, this is a perfect approach. Some of your lines in your public methods with branches will count 2 or more executions, and probably the lines inside each of the branches will count 1 execution.
Private methods will also count +n executions, while you won't write named tests for those methods, they will be called as a result of being helpers to the public methods.
When you say:
but I have not seen an example of it anywhere
I can tell that 90% of our tests use a #covers MyNiceClass annotation without specifying any method.
If you start by the test first, you normally won't mess up things. If you "need" that "MyNiceClass" is able to get something, you start by doing testGetSomething() in the test and set a basic expectation, then you write the public function getSomething() in the class and you end up having a one-to-one test for each method in 80% of the cases. The rest of the cases normally are multiple tests for the same method like for example testGetSomething() (happy path) and testGetSomethingThrowsExceptionWithInvalidParameter() (sad path).
In short
Yes, what you propose is correct, normal, usual and it is the recommended way to go according to the documentation.
Another reason (which was the reason in my case) is not using the complete class-name including the namespace.
// Should be like
#covers \App\Module\MyClass::doSomething
Why not simply run the test case and see the coverage with your changes? You risk getting down-voted as you are asking a question without trying anything.
From the PHPUnit manual:
The #covers annotation (see Table B.1) can be used in the test code to specify
which method(s) a test method wants to test. If provided, only the code
coverage information for the specified method(s) will be considered. Example
11.2 shows an example.
Table B1 Referenced above
If I need to run only one test from class like
phpunit --filter someTest tests/ExampleTest.php
PHPUnit still initializes dataProviders for all tests in that class, which, in my case, takes plenty of time.
Is there a way to initialize dataProvider only for executed method?
If you use --group instead of --filter, only the tests of the selected group are initialized. That should also apply to data providers.
To do so, you have to add the #group annotation (fortunately this is possible per method and not only per class). If you still want to be able to execute all tests separately, put each one in its own group:
/**
* #test
* #group someTest
* #dataProvider someData
*/
public function someTest(...)
I have a unit tested application which we have updated from symfony 2.3 to 2.6. We followed all upgrading docs and had to change only some minor stuff.
Everything is working perfectly, except for the PHPUnit tests.
We have 2 seperate runs, one for only testing the entity classes, which is fired on a pre-commit hook. and a second one which runs the full suite, with database setups and the whole nine yards.
Now since the upgrade to 2.6, the PHPUnit_Framework_Error thrown in the unit tests have been replaced by Symfony's Symfony\Component\Debug\Exception\ContextErrorException, this failing all tests like this:
/**
* #dataProvider objectTestDataProvider
* #expectedException \PHPUnit_Framework_Error
*/
public function testCanNotSetClientToArbitraryValue($value)
Now I do not want to change this into the new Exception since running the entity-only test suite does not depend on symfony components, thus symfony is not loaded, thus the errors are the regular PHPUnit_Framework_Error so changing it makes these tests fail.
In other words, when I run one test class it works, once a symfony dependent test is run, it fails:
# runs perfectly
phpunit -c app/phpunit.xml --debug src/My/Bundle/Tests/Entity
# fails when reaching the tests that ran perfectly in previous command
phpunit -c app/phpunit.xml --debug
This new ErrorHandler seems undocumented, I couldnt find much about it in google except for the pull request and this small article
I've tried:
setting the SYMFONY_DEBUG=0 environment variable, but this doesnt seem to make any difference.
adding the debug.error_handler.throw_at: 0 parameter to my test_config.yml
edit:
On request by #cerad I've tried to isolate the tests to try and reproduce the code with as little as possible, Ive managed to reproduce with 4 tests:
class MyControllerTest extends WebTestCase
{
public function testRoutesLoaded_1()
{
$client = self::createClient();
/** #var Router $router */
$router = $client->getKernel()->getContainer()->get('router');
$this->assertEquals('/menu', $router->generate('front_menu'));
}
/**
* #expectedException \PHPUnit_Framework_Error
*/
public function testCreateOrder_1()
{
new Order(); // required parameter missing
}
public function testRoutesLoaded_2()
{
$client = $this->createNewFrontClient();
/** #var Router $router */
$router = $client->getKernel()->getContainer()->get('router');
$this->assertEquals('/menu', $router->generate('front_menu'));
}
/**
* #expectedException \PHPUnit_Framework_Error
*/
public function testCreateOrder_2()
{
new Order(); // required parameter missing
}
}
As you can see, I just run the same exact test 2 times, but still the last one results in an error:
MyControllerTest::testCreateOrder_2
Failed asserting that exception of type "Symfony\Component\Debug\Exception\ContextErrorException" matches expected exception "\PHPUnit_Framework_Error"
Since I did not get any replies here, I posted an issue on Symfony's github and they confirmed this was incorrect behavior.
The issue was resolved and is merged in 2.6-dev.