I am looking for some sort of dynamic testing tool. Let me explain what I mean. I have a modular application and part of my business strategy is to license each module independently. I would like to test things such as the fact that one module's tests all still pass even when the other module is not present on the file system.
Another thing I would like to be able to verify during my test suite, is that un-related settings should not change behavior they are not supposed to be coupled to.
To write an ad-hoc script for each test case and inject the defect would be trivial, but sounds like a violation of DRY. Is there any tools designed for this kind of defect injection, specifically in PHP?
So in an ideal world I would like to be able to write a set of "environment modifying" scripts, each of which will alter the environment in some way. For each "environment modification" phpunit should run all my tests. I should be able to view in some kind of report which tests failed under which "environment extremes"
So with N "environment modifications" and M tests, The sum of passes & failures should equal N times M, similar to how GUI test tools such as selenium run the suite on each different environment (browser)
This might work using phpUnit and some magic inside the fixtures. But I'd look into some build tool (like Ant or Phing) which creates a checkout from your source repository, set's up the required modules and then runs the phpUnit tests.
Phing or Ant can be called from an continous integration system like phpUnderControl or Hudson on a regular basis so you get regular automated feedback.
Related
I am doing TDD for a project using PHP. Until now, I write unit tests, make them fail and then write the least amount of code to fulfill the test. After the project has been completed, I write acceptance testing using CasperJS.
Of late I have been looking into Codeception and Behat and some other test frameworks and have been reading about different types of tests like Unit Testing, Integration Testing et al.
Nowhere could I find the correct order of testing.
What I want to know is when I sit down to design the project, I do:
Requirement Analysis
Technology Stack Selection
Enumerate the Resources/Business Entities
Then decide what goes into Models, what stays as Services etc.
Database Design
Do the list of Models, Controllers, Services necessary
Write tests before writing the individual classes using phpUnit
Once API is ready, write CasperJS tests to verify behavior.
While this is not exact, but a good indication of how I run my shop. So, where do integration testing and behavior testing fit in?
This really feels like an opinion based question, so don't be surprised if it gets closed for being such. There really isn't a perfect answer, and deciding how and when to write tests really depends on the project and you.
You could try to work out all the user stories and behaviors and write the acceptance tests before your step 3. This could help illuminate dark corners in the plan.
Or, you could write acceptance tests before starting a feature. This could help to get you in the mindset of what needs to be done with a given feature, its scope, and edge cases.
Or, you could write acceptance tests after the project is finished. This could serve as a final check list of expected behaviors before handing off to the customer for whatever acceptance testing they want to do.
I'm sure there're other points in your workflow where writing acceptance tests might be appropriate, but these are three points where I've found myself writing such tests. IMO, the best place is right before starting a feature. At that point, I have a user story, I'm familiar with the code I've already written, and I have an idea of what the new code is expected to do.
The acceptance tests can be organized to guide coding in the same way unit tests do, but at a broader level. Still iterate through "write failing test, write code to make test pass, write failing test," but also have a larger loop driven by the acceptance tests. Once you get to a point in the inner cycle where you think you'll have a passing acceptance test, check by running the whole suite.
There is another way in which you can ask "where integration and behavior testing fit in," and that's in the sense of "where does that testing fit in with the rest of my testing and code?" This is a little less grey. Unit testing should be run often. The entire unit test suite. Often. So it needs to be incredibly fast. You should be able to know if you broke something internal to your project immediately.
Integration tests are there to verify the ins and outs are working as expected. Outside of your app, your dependencies aren't going to change, and if they do, it should be a big deal that you should be aware of. So there's a clear demarcation between your code and their code. Your unit tests can carry you all the way to that interface. Integration tests verify that the interface you coded for, really is the interface they're providing. You don't need to run these with every little code change. You do need to run them, but maybe only every commit. They can be slower.
Acceptance tests are similar to integration tests, only rather than enlisting an external dependency to verify the interfaces match, they define the interface. You could hold of on running them until near release, but the more often you can run them, the more value they actually provide.
YMMV.
I am new to testing. All i knew was we PHPunit for testing various functions within class and then i know selenium for browser testing.
I know we can write php to link with selenium web driver to do headless testing of browser.
I am not able to get how does behat and mink come in there. Are these seperate from selenium and they are the alternatives of selenium.
Can i do aweb application tetsing without beaht, mink and only with selenium and php
PHPUnit and Behat are similar, both being testing frameworks. They allow you to test your code, by using different approaches:
PHPUnit tests are based on code you write to check how your classes behave under the required circunstances. A lot of people use this type of framework to practice TDD, but you can of course write tests after the code, or for code written a long time ago.
Behat tests are written in a human readable way, and they are supposed to allow everyone involved in the project to read them. This type of testing is called BDD. You can write tests that explain in (nearly) plain english how your system is supposed to behave.
IMO PHPUnit is more general and is the preferred way of writing most tests. I use Behat for testing my systems general behavior, and PHPUnit for unit testing every class and method independently of others.
On the other side, Mink is a library that allows you to browse programatically, using PHP, and access the contents. It can be used to control in a unified way a lot of browsing systems like Selenium, Zombie, etc. each of them based on different technologies.
You can use Mink outside Behat, but they are usually used together because this way you can write tests that show how a website behaves: Given I enter my credentials in the login form, And press the submit button, I should see my profile page...
And yes, you can use PHPUnit and Selenium together as explained in the docs...
We're considering using a CI server soon.
From my reading, I've found that Sismo and Hudson were available for PHP project.
Considering that we're actually using GIT and PHPUnit, what are the big difference between Hudson and Sismo that we should know in order to make the best choice for our situation ?
Thanks
The language match is not key in your hunt for the best CI server; it's all the features around:
source control
concurrent build
trigger build
notification
Even for simple project, Jenkins (the new name for Hudson) is easy to use and quick to install. Then it is really easy to scale Jenkins up by adding more nodes (satellite machine that can execute build) when you need to. Also Jenkins has hundreds of plugin for numerous task.
Have a look at Bamboo, Jenkins, TeamCity, and CruiseControl Features to compare some of the features of the big names (you might actually want to consider Bamboo, TeamCity or Cruise Control over Jenkins)
I would lean towards Sismo since it matches the language of the project you are developing (PHP) and can be ran from just a single PHP and config file. Then you don't have to deal with having a java environment just for Hudson.
There is a really good php-integration for Jenkins by the phpunit inventor Sebastian Bergmann. You should really have a look at it.
As far as I see the biggest downside of Sismo is, that is not a "real" CI server, but more a build-and-report-environment, because you need to trigger the builds yourself (or let something trigger it).
I'll preface this by saying that I haven't used sismo.
We use Hudson with applications being built & tested in both Java and PHP. It has a nice plugin system, and getting it up and running on a centOS box took about 15 minutes yesterday. (We had to move it from one box to another).
For PHP Hudson integrates with both PHPUnit and Selenium so we run both unit tests and functional tests against the same codebase. Hudson has a great 'one-click' plugin system that really lets you customize your installation.
One thing we had to get a plugin for was sending an email on every build whether successful or not. Hudson by default will only email when your build goes from good (tests pass) to bad, from bad to good, or repeatedly bad. This means it will not send an email for every build if 2 builds in a row were successful. The email plugin solves this but it was confusing to uncover that.
Which unit testing framework do you use for Symfony?
Lime or PHPUnit? What are the pros and cons of using them?
In my opinion, here are a few things that come to my mind :
PHPUnit is more integrated with other tools, like, for instance,
Selenium (PHPUnit can use it to open true real browsers to test your site)
phpUnderControl for continuous-integration
PHPUnit works well with Xdebug, to generate code-coverage reports
PHPUnit is more widely used ; which probably means more support
But note I don't work with symfony, nor lime...
Still, I've never heards anyone speak about it, except for those working with symfony -- that not a good thing, for the day you'll have to work with another framework (yes, this happens ^^ )
One thing that's not in PHPUnit :
"false" browser (being able to do HTTP Requests to the application, without using Selenium to open a real browser)
But some frameworks (Zend Framework does, with it's Zend_Test component) integrate with PHPUnit (or use it), while allowing injection of data into the MVC and fetching of the response, without having to issue any HTTP Request.
I don't know if symfony allows that, but that's a nice thing with ZF/PHPUnit ^^
(Yes, not a symfony-specific answer ; but of the things I said must still be valid with that framework)
Lime is a much more simple testing framework, which can be a good or a bad thing depending on how you want to use it.
The symfony library itself uses its own testing framework, Lime, to test its code base. From the symfony book:
It is based on the Test::More Perl
library, and is TAP compliant, which
means that the result of tests is
displayed as specified in the Test
Anything Protocol, designed for better
readability of test output.
I cannot vouch for the statement that the lime framework is "more lightweight" than other PHP testing frameworks as the symfony docs claim, but I do really like that it's built right into your symfony project and works well with the symfony command line tools without any additional configuration. One thing that is especially cool is that the lime tests within symfony are set to run within your "test" environment which has it's own database, symfony cache (which gets cleared out during each test session), and environment variables. This comes in handy when you want to do functional testing (checking server response and your html output in your modules/actions, versus basic unit testing). I also like that lime is super easy to pick up and understand since it's so simple. You also have the ability to put your tests into YAML configuration file rather than write the tests by hand.
Pascal is entirely right that PHPUnit is much more widely used and you'd be able to use it in non-symfony projects. There is even a plugin for it, PHPUnit symfony plugin. My best advice would be to use lime if you just wanted to jump right into writing simple tests while you develop your symfony app. But, if you have the time and hope to use these testing skills outside of the symfony world, or bring in pre-existing PHPUnit tests into your symfony code, it'd be worth your time to check out the plugin and give it a spin.
I'm curious to see how other developers go about testing their web sites. PHP specifically in my case, but this probably spans multiple languages. I've been working on a site for over a year now, and I'd really like to automate a lot of the regression testing I do between versions.
This specific site is in CodeIgniter, so I have some tests for my models. I'd like to move beyond just testing those though. However, this is an issue even non-MVC developers have had to tackle I'm sure.
Edit: I think the functionality that would satisfy a lot of my test desires is the ability to assert that paramters have a specific value at the end of the script processing. In my case a lot of logic is in the controller, and that's the main area I'd like to test.
For actual unit testing without testing the UI, you should just test the functions in the model. Most of your functionality should be in there anyways.
You might want to have a look at Selenium for testing the UI of your site. It can record your actions and play them back, or you can edit the scripting directly.
(source: seleniumhq.org)
Have you tried Fitnesse ?
It helps on creating Acceptance tests. They are specially useful for websites, which doing this kind of tests are a pain.
There are a couple of videos from unclebob inside the webpage too. The good thing is that Fitnesse is not restricted for website testing, so your knowledge about using it can be used with other apps too.
The project I'm working on is a Desktop APP written in c++ that uses Fitnesse tests.
But if you meant unit testing the models (which I think you didn't), they can be create using the phpunit lib. I think the ZEND framework has a similar lib for that.
You might want to check out PHPUnit
http://www.phpunit.de/manual/current/en/
I have started using it on my PHP projects and it's very easy to work with and very powerful. In particular, learn and use mocks:
http://www.phpunit.de/manual/3.0/en/mock-objects.html
Mocking is especially important when unit testing applications that do database operations.
Take a look at TOAST. It's build specially for CodeIgniter. It uses CI infrastructure, so you can run all test tests via a browser and results are displayed back as a web page (HTML). It's very simple to use.
I suggest you test your Controllers as well. Testing model is ok, but model is just the DB storage. Controllers contain all the "business logic" and are the place where most things go wrong.
One of the best ideas I've heard of, as far as testing web apps go, was to create a script that would go over all the pages in the site and check them for differences from the previous scan, letting you accept changes and fix regressions.
Generally speaking, automatic testing of GUI applications (websites are GUI apps) is difficult and usually unnecessary. Unit tests work best with simple libraries.
I use Canoo WebTest. It is the best free web site unit test framework out there. It is entirely scriptable with XML and requires no browser so it can run from a build server.
We modified Waiter (Ruby). It plays back "scripts" of URLs and Form Filling to IE and we have added a script "command" to take a Screen Capture; the screen capture image is compared against a Known-Good-Image (i.e. a Master Image) and if that image is different it is logged (basically a Web page of such results is prepared) and "a human" does a review of the Master / Test image. Obviously there are two outcomes at that point - "The difference is intentional" or "There is an incorrect change". In the first instance the Master image is replaced with the New Image; in the second we go fix the bug, and the change will be included in the next test run