Moving to Codeception from Behat and still getting used to it's concepts & where things go.
In the hypothetical that my tests are 100% driven from .feature files, does this mean that all of the test code could be in Contexts? That there wouldn't be anything in any functional tests that extend PHPUnit_Framework_TestCase? (Assuming that all my functional tests would extend that)
Codeception is not driven by Gherkin as Behat is. If you are moving away from Behat you will write functions in classes in Codeception directly and you are not going to start from a Gherkin script to then derive the executable specs (in your contexts files, page objects).
In brief the two flows
Behat
Write the BDD scripts/Gherkin - features. These are totally abstract and should usually be logic descriptions of the use cases your system implements. A product owner can start writing this for instance when a user story is created. Requires no programming logic
For each line in the feature implement an executable specification (a function in a Context class) that handles that action
In Behat usually you also use Page objects (unsure if this can also be done in Codeception but I dont see why not if you can import the Page Object library)
Codeception
You write executable specifications as a first step for instance in a Cept class. A developer is needed here as this is actual PHP Code/Classes
When you run codeception then it prints out a list of all the statements it has run, just like a report.
The above is a very simplified description as your question is also very generic. I hope it answers your question
Related
how to let behat know which context to use?, because by default,there is this FeatureContext.php inside the features folder, whereby, when the behat runs, it will use that file while the .feature file is running (scenarios)...the problem is, if I have alot of modules to test, ofcourse, I cannot just lump all of those functions inside one FeatureContext.php file right?, I would want to name different contexes and different .feature files, so how to let behat know which context and .feature file to run?
this is the line of command that i use to fire behat
/export/home/project/htdocs/bdd/bin/behat --profile phantomjs-via-webdriver
Behat uses all of the contexts you specify, the only way out of this is to specify only the context(s) you want to use. Depending on the Behat version the syntax also differs. For Behat 2 this answer explains one strategy. For Behat 3 you can do the following:
default:
suites:
module1:
contexts:
- 'FeatureContext1'
- 'FeatureContext2'
module2:
contexts:
- 'FeatureContext1'
- 'FeatureContext3'
My two cents: I don't have a tiniest idea about your app, but don't confuse functional and unit testing (if you do). There can be different situations when you benefit from per module contexts, like "authorisation", "admin", or "comments", but their only purpose is to separate code from the overall context into a more manageable chunks. They still work together and cover the entire application (or a very distinct part of it). Testing module-specific functionality would be a lot easier with a unit test framework, which ensures that the modules works on their own.
I am new to testing. All i knew was we PHPunit for testing various functions within class and then i know selenium for browser testing.
I know we can write php to link with selenium web driver to do headless testing of browser.
I am not able to get how does behat and mink come in there. Are these seperate from selenium and they are the alternatives of selenium.
Can i do aweb application tetsing without beaht, mink and only with selenium and php
PHPUnit and Behat are similar, both being testing frameworks. They allow you to test your code, by using different approaches:
PHPUnit tests are based on code you write to check how your classes behave under the required circunstances. A lot of people use this type of framework to practice TDD, but you can of course write tests after the code, or for code written a long time ago.
Behat tests are written in a human readable way, and they are supposed to allow everyone involved in the project to read them. This type of testing is called BDD. You can write tests that explain in (nearly) plain english how your system is supposed to behave.
IMO PHPUnit is more general and is the preferred way of writing most tests. I use Behat for testing my systems general behavior, and PHPUnit for unit testing every class and method independently of others.
On the other side, Mink is a library that allows you to browse programatically, using PHP, and access the contents. It can be used to control in a unified way a lot of browsing systems like Selenium, Zombie, etc. each of them based on different technologies.
You can use Mink outside Behat, but they are usually used together because this way you can write tests that show how a website behaves: Given I enter my credentials in the login form, And press the submit button, I should see my profile page...
And yes, you can use PHPUnit and Selenium together as explained in the docs...
My Yii app getting phpunit coverage in controllers, models, components, modules but not any views. The problem is that code coverage loader in phpunit includes the view file during preparing a coverage report. Views with forms have calls $this->beginWidget which causes a crash since there is no $this context.
Views dont really have important code or logic but still they have some conditions and even loops to call renderPartial so it would be good to get a view code also covered.
Is there a solution to this problem?
Have you tried extending CWebTestCase? Generally when writing unit tests, you have fixtures and things to provide the necessary data - but with tests on views and 'functional' tests, for web-apps, it's generally easiest to mimic a browser and have it perform actions on the web app as if a user was actually using it. Currently, this mimicking is most easily done with Selenium (in my opinion).
The Yii Guide on Functional Testing is a good place to start as well as the Selenium Documentation. There's also this book that goes over using selenium (I'm not sure if the newest edition does, but I know the previous release with Publication Date: August 11, 2010 did), and Larry Ullman's Yii Book will have chapters on testing and the usage of Selenium in functional tests when he's completed that chapter.
Hope this helps!
Update to further explain CWebTestCase
CWebTestCase extends PHPUnit_Extensions_SeleniumTestCase which directly implements a functional testing foundation that you can use within phpunit in order to test views, test widget creation inside of views, assert that text exists, 'click' on links, etc. These tests are still run from the command line though they require that Selenium-RC server be started upon the test being run and they require a valid browser being configured. A valid browser can be configured with as little code as the following placed inside of the setUp() function:
$this->setBrowser('*firefox /usr/lib/firefox/firefox-bin');
Stating that code coverage cannot be provided by CWebTestCase is not true, as CWebTestCase extends PHPUnit_Extensions_SeleniumTestCase, which provides the following as quoted from the documentation:
PHPUnit_Extensions_SeleniumTestCase can collect code coverage
information for tests run through Selenium:
Copy PHPUnit/Extensions/SeleniumTestCase/phpunit_coverage.php into
your webserver's document root directory. In your webserver's php.ini
configuration file, configure
PHPUnit/Extensions/SeleniumTestCase/prepend.php and
PHPUnit/Extensions/SeleniumTestCase/append.php
as the auto_prepend_file and auto_append_file, respectively. In your test
case class that extends PHPUnit_Extensions_SeleniumTestCase, use
protected $coverageScriptUrl = 'http://host/phpunit_coverage.php';
to configure the URL for the phpunit_coverage.php script.
I am currently going to start from scratch with the phpunit tests for a project. So I was looking into some projects (like Zend) to see how they are doing things and how they organizing their tests.
Most things are pretty clear, only thing I have some problems with is how to organize the test suites properly.
Zend has an AllTests.php from which loads others test suites.
Tough looking at the class it is useing PHPUnit_Framework_TestSuite to create a suite object and then add the other suites to it, but if I look in the PHPUnit docs for organizing tests in PHPUnit versions after 3.4 there is only a description for XML or FileHierarchy. The one using classes to organize the tests was removed.
I haven't found anything that this method is deprecated and projects like Zend are still using it.
But if it is deprecated, how would I be able to organize tests in the same structure with the xml configuration? Executing all tests is no problem, but how would I organize the tests (in the xml) if I only wanted to execute a few tests. Maybe creating several xmls where I only specify a few tests/test suites to be run?
So if I would want to only test module1 and module2 of the application, would I have an extra xml for each and defining test suites only for those modules (classes used by the module) in it. And also one that defines a test suite for all tests?
Or would it be better to use the #group annotation on the specific tests to mark them to be for module1 or module2?
Thanks in advance for pointing me to some best practices.
I'll start of by linking to the manual and then going into what I've seen and heard in the field.
Organizing phpunit test suites
Module / Test folder organization in the file system
My recommended approach is combining the file system with an xml config.
tests/
\ unit/
| - module1
| - module2
- integration/
- functional/
with a phpunit.xml with a simple:
<testsuites>
<testsuite name="My whole project">
<directory>tests</directory>
</testsuite>
</testsuites>
you can split the testsuites if you want to but thats a project to project choice.
Running phpunit will then execute ALL tests and running phpunit tests/unit/module1 will run all tests of module1.
Organization of the "unit" folder
The most common approach here is to mirror your source/ directory structure in your tests/unit/ folder structure.
You have one TestClass per ProductionClass anyways so it's a good approach in my book.
In file organization
One class per file.
It's not going to work anyways if you have more than one test class in one file so avoid that pitfall.
Don't have a test namespace
It just makes writing the test more verbose as you need an additional use statement so I'd say the testClass should go in the same namespace as the production class but that is nothing PHPUnit forces you to do. I've just found it to be easier with no drawbacks.
Executing only a few tests
For example phpunit --filter Factory executes all FactoryTests while phpunit tests/unit/logger/ executes everything logging related.
You can use #group tags for something like issue numbers, stories or something but for "modules" I'd use the folder layout.
Multiple xml files
It can be useful to create multiple xml files if you want to have:
one without code coverage
one just for the unit tests (but not for the functional or integration or long running tests)
other common "filter" cases
PHPBB3 for example does that for their phpunit.xmls
Code coverage for your tests
As it is related to starting a new project with tests:
My suggestion is to use #covers tags like described in my blog (Only for unit tests, always cover all non public functions, always use covers tags.
Don't generate coverage for your integration tests. It gives you a false sense of security.
Always use whitelisting to include all of your production code so the numbers don't lie to you!
Autoloading and bootstrapping your tests
You don't need any sort of auto loading for your tests. PHPUnit will take care of that.
Use the <phpunit bootstrap="file"> attribute to specify your test bootstrap. tests/bootstrap.php is a nice place to put it. There you can set up your applications autoloader and so on (or call your applications bootstrap for that matter).
Summary
Use the xml configuration for pretty much everything
Seperate unit and integration tests
Your unit test folders should mirror your applications folder structure
To only execute specif tests use phpunit --filter or phpunit tests/unit/module1
Use the strict mode from the get go and never turn it off.
Sample projects to look at
Sebastian Bergmanns "Bank Account" example project
phpBB3 Even so they have to fight some with their legacy ;)
Symfony2
Doctrine2
Basic Directory Structure:
I have been experimenting with keeping the test code right next to the code being tested, literally in the same directory with a slightly different file name from the file with the code it is testing. So far I am liking this approach. The idea is you don't have to spend time and energy keeping the directory structure in sync between your code and your test code. So if you change the name of the directory the code is in, you don't then also need to go and find and change the directory name for the test code. This also causes you to spend less time looking for the test code that goes with some code as it is right there next to it. This even makes it less of a hassle to create the file with the test code to begin with because you don't have to first find the directory with the tests, possibly create a new directory to match the one you are creating tests for, and then create the test file. You just create the test file right there.
One huge advantage of this is it means the other employees (not you because you would never do this) will be less likely to avoid writing test code to begin with because it is just too much work. Even as they add methods to existing classes they will be less likely to not feel like adding tests to the existing test code, because of the low friction of finding the test code.
One disadvantage is this makes it harder to release your production code without the tests accompanying it. Although if you use strict naming conventions it still might be possible. For example, I have been using ClassName.php, ClassNameUnitTest.php, and ClassNameIntegrationTest.php. When I want to run all the unit tests, there is a suite that looks for files ending in UnitTest.php. The integration test suite works similarly. If I wanted to, I could use a similar technique to prevent the tests from getting released to production.
Another disadvantage of this approach is when you are just looking for actual code, not test code, it takes a little more effort to differentiate between the two. But I feel this is actually a good thing as it forces us to feel the pain of the reality that test code is code too, it adds its' own maintenance costs, and is just as vitally a part of the code as anything else, not just something off to the side somewhere.
One test class per class:
This is far from experimental for most programmers, but it is for me. I am experimenting with only having one test class per class being tested. In the past I had an entire directory for each class being tested and then I had several classes inside that directory. Each test class setup the class being tested in a certain way, and then had a bunch of methods each one with a different assertion made. But then I started noticing certain conditions I would get these objects into had stuff in common with other conditions it got into from other test classes. The duplication become too much to handle, so I started creating abstractions to remove it. The test code became very difficult to understand and maintain. I realized this, but I couldn't see an alternative that made sense to me. Just having one test class per class seemed like it would not be able to test nearly enough situations without becoming overwhelming to have all that test code inside one test class. Now I have a different perspective on it. Even if I was right, this is a huge dampener on other programmers, and myself, wanting to write and maintain the tests. Now I am experimenting with forcing myself to have one test class per class being tested. If I run into too many things to test in that one test class, I am experimenting with seeing this as an indication that the class being tested is doing too much, and should be broken up into multiple classes. For removing duplication I am trying to stick to simpler abstractions as much as possible that allows everything to exist in one readable test class.
UPDATE
I am still using and liking this approach, but I have found a very good technique for reducing the amount of test code and the amount of duplication. It is important to write reusable assertion methods inside the test class itself that gets heavily used by the test methods in that class. It helps me to come up with the right types of assertion methods if I think of them as internal DSLs (something Uncle Bob promotes, well actually he promotes actually making internal DSLs). Sometimes you can take this DSL concept even further (actually make a DSL) by accepting a string parameter that has a simple value that refers to what kind of test you are trying to perform. For example, one time I made a reusable asssertion method that accepted a $left, $comparesAs, and a $right parameter. This made the tests very short and readable as the code read something like $this->assertCmp('a', '<', 'b').
Honestly, I can't emphasize that point enough, it is the entire foundation of making writing tests something that is sustainable (that you and the other programmers want to keep doing). It makes it possible for the value that tests add to outweigh what they take away. The point is not that you need to use that exact technique, the point is you need to use some kind of reusable abstractions that allow you to write short and readable tests. It might seem like I'm getting off topic from the question, but I'm really not. If you don't do this, you will eventually fall into the trap of needing to create multiple test classes per class being tested, and things really break down from there.
I am looking for some sort of dynamic testing tool. Let me explain what I mean. I have a modular application and part of my business strategy is to license each module independently. I would like to test things such as the fact that one module's tests all still pass even when the other module is not present on the file system.
Another thing I would like to be able to verify during my test suite, is that un-related settings should not change behavior they are not supposed to be coupled to.
To write an ad-hoc script for each test case and inject the defect would be trivial, but sounds like a violation of DRY. Is there any tools designed for this kind of defect injection, specifically in PHP?
So in an ideal world I would like to be able to write a set of "environment modifying" scripts, each of which will alter the environment in some way. For each "environment modification" phpunit should run all my tests. I should be able to view in some kind of report which tests failed under which "environment extremes"
So with N "environment modifications" and M tests, The sum of passes & failures should equal N times M, similar to how GUI test tools such as selenium run the suite on each different environment (browser)
This might work using phpUnit and some magic inside the fixtures. But I'd look into some build tool (like Ant or Phing) which creates a checkout from your source repository, set's up the required modules and then runs the phpUnit tests.
Phing or Ant can be called from an continous integration system like phpUnderControl or Hudson on a regular basis so you get regular automated feedback.