I'm curious to see how other developers go about testing their web sites. PHP specifically in my case, but this probably spans multiple languages. I've been working on a site for over a year now, and I'd really like to automate a lot of the regression testing I do between versions.
This specific site is in CodeIgniter, so I have some tests for my models. I'd like to move beyond just testing those though. However, this is an issue even non-MVC developers have had to tackle I'm sure.
Edit: I think the functionality that would satisfy a lot of my test desires is the ability to assert that paramters have a specific value at the end of the script processing. In my case a lot of logic is in the controller, and that's the main area I'd like to test.
For actual unit testing without testing the UI, you should just test the functions in the model. Most of your functionality should be in there anyways.
You might want to have a look at Selenium for testing the UI of your site. It can record your actions and play them back, or you can edit the scripting directly.
(source: seleniumhq.org)
Have you tried Fitnesse ?
It helps on creating Acceptance tests. They are specially useful for websites, which doing this kind of tests are a pain.
There are a couple of videos from unclebob inside the webpage too. The good thing is that Fitnesse is not restricted for website testing, so your knowledge about using it can be used with other apps too.
The project I'm working on is a Desktop APP written in c++ that uses Fitnesse tests.
But if you meant unit testing the models (which I think you didn't), they can be create using the phpunit lib. I think the ZEND framework has a similar lib for that.
You might want to check out PHPUnit
http://www.phpunit.de/manual/current/en/
I have started using it on my PHP projects and it's very easy to work with and very powerful. In particular, learn and use mocks:
http://www.phpunit.de/manual/3.0/en/mock-objects.html
Mocking is especially important when unit testing applications that do database operations.
Take a look at TOAST. It's build specially for CodeIgniter. It uses CI infrastructure, so you can run all test tests via a browser and results are displayed back as a web page (HTML). It's very simple to use.
I suggest you test your Controllers as well. Testing model is ok, but model is just the DB storage. Controllers contain all the "business logic" and are the place where most things go wrong.
One of the best ideas I've heard of, as far as testing web apps go, was to create a script that would go over all the pages in the site and check them for differences from the previous scan, letting you accept changes and fix regressions.
Generally speaking, automatic testing of GUI applications (websites are GUI apps) is difficult and usually unnecessary. Unit tests work best with simple libraries.
I use Canoo WebTest. It is the best free web site unit test framework out there. It is entirely scriptable with XML and requires no browser so it can run from a build server.
We modified Waiter (Ruby). It plays back "scripts" of URLs and Form Filling to IE and we have added a script "command" to take a Screen Capture; the screen capture image is compared against a Known-Good-Image (i.e. a Master Image) and if that image is different it is logged (basically a Web page of such results is prepared) and "a human" does a review of the Master / Test image. Obviously there are two outcomes at that point - "The difference is intentional" or "There is an incorrect change". In the first instance the Master image is replaced with the New Image; in the second we go fix the bug, and the change will be included in the next test run
Related
I've got a PHP project to test. I've already (almost) finished coding the project and I didn't have any testing plan at the start. Being new to PHP, I thought I'd finish the coding and start testing afterwards. As unit testing needs to be done as you go along, what is my other alternative for a test? I want something like a complete system test rather than unit by unit.
I've got around 30 PHP script files, 5 of which are class scripts.
NB It needs to be a formal way of testing, rather than trying to hack it.
I believe your aim behind the test is more "academic" than practical. So testing by friends doesn't really help much. Having said that you have already completed coding the project so if you wanted to unit test using PHPUnit or SimpleTest you will have to go back and start coding the unit test cases.
I think the best option here would be to create a "formal" test document. There are tons of resources out there for how to do this. Basically it will have:
Url
Action
Expected Result
Observed Result
If your expected and observed results match, then that test case is passed.
You can create as many test cases as you want for your project based on these basic guidelines.
Testing has it's different role in SDLC. There are various aspect on which testing could be done. Like:
Smoke Testing
Functionality Testing
System Testing
Retesting
Regration Testing
Preformance Testing
and many more testing technique are there for testing a web application.
Just upload the project and ask someone to test it.You have to test it or get it tested.There is no other way
You can also run a simple scanner(like burp or acunetix) to test whether website can be hacked or not.
You can use a web app scanner,
There are some F/OSS out there such as;
Skipfish - CLI Windows/Linux
Netsparker Community Edition - Windows - Light version of a commercial scanner
List of web app security scanners - commercial and free
I am looking for a (small) library that helps me cleanly implement a front controller for my pet project and dispatches requests to single controller classes. The front controller/dispatcher and controller classes need to be fully unittestable without sending HTTP requests.
Requirements
PSR-0 compatible
installable via its own PEAR channel
support for unit testing:
checking if the correct HTTP headers are sent
catches output to allow inspection in unit tests
perferably PHPUnit helper methods to help inspecting the output (for different output types, i.e. HTML, XML, JSON)
allows setting of incoming HTTP headers, GET and POST parameters and cookies without actually doing HTTP requests
needs to be usable standalone - without the db abstraction, templating and so that the fat frameworks all provide
Background
SemanticScuttle, the application that is bound to get proper "C" support, is an existing, working application. The library needs to blend in it and needs to work with the existing structure and classes. I won't rewrite it to match a framework's specific required directory layout.
The application already has unittests, but based on HTTP requests which make them slow. Also, the current old way of having several dozens of .php files in the www directory isn't the most managable solution, which is why proper controller classes need to be introduced. All in all, there will be about 20-30 controllers.
Previous experience
In general, I was pretty happy with Zend Framework for some previous projects but it has several drawbacks:
not pear-installable, so I cannot use it as dependency in my pear-installble applications
only available as one fat download, so I manually need to extract the required bits from it - for each single ZF update.
while unit test support exists for ZF controllers, it's lacking some advanced utility functionality like assertions for json, HTTP status code and content type checks.
While these points seem to be nit-picking, they are important for me. If I have to implement them myself, I do not need to use an external libary but write my own.
What I don't want
StackOverflow has a million "what's the best PHP framework" questions (1, 2, 3, 4, 5), but I'm not looking for those but for a specific library that helps with controllers. If it's part of a modular framework, fine.
I also know the PHP framework comparison website, but it doesn't help answer my question since my requirements are not listed there.
And I know that I can build this all on my own and invent another microframework. But why? There are so many of them already, and one just has to have all that I need.
Related questions
What's your 'no framework' PHP framework?
How do you convert a page-based PHP application to MVC?
Knowing Symfony2 well, I can assure you it's definitely possible to use it just for the "C" in MVC. The models and templates are completely free and are typically executed from the Controllers anyway, so if you don't call Doctrine or Twig specifically, you can do what you want.
As for functional testing, which is really what you're talking about in your article, what you want to look at is the WebTestCase class, which is well complemented by the LiipFunctionalTestBundle bundle for more advanced cases.
That allows for some things like this example of testing a contact form that sends an email, where the entire HTTP request is done in process, since the framework is written to allow multiple requests per process and has no global state, this works very well and does not require a http server to be running or anything. As you can see I do assertions on the HTTP status code of the response too, and was able to capture the email without sending it since in test configuration sending of emails is disabled in the standard distro of Symfony2.
That being said, you could also just use the Request and Response classes from Symfony2's HttpFoundation component. It should allow you to test your code, but IMO you wouldn't get as many nice features as you could if you'd use the entire framework. Of course that's just my biased opinion ;)
I would recommend downloading the Symfony 2 framework Routing component: https://github.com/symfony/Routing
Documentation is found here: http://symfony.com/doc/current/book/routing.html
Perhaps it does not satisfy all you requirements, but it's the closest.
If you are familiar with symfony (which I think you are) you should check out silex From their website this is what they say about it:
A microframework provides the guts for building simple single-file apps. Silex aims to be:
Concise: Silex exposes an intuitive
and concise API that is fun to use.
Extensible: Silex has an extension
system based around the Pimple micro
service-container that makes it even
easier to tie in third party
libraries.
Testable: Silex uses
Symfony2's HttpKernel which abstracts
request and response. This makes it
very easy to test apps and the
framework itself. It also respects
the HTTP specification and encourages
its proper use.
I'd add Net_URL_Mapper, it doesn't have the assertions though. Is that why you ruled it out?
Another pretty interesting thing is silex. It also comes with controller tests. I'd use that over Symfony2. But that's my personal preference.
Quite a understandable wishlist. I think we all hate it in testing when we run into dependencies that make testing to havoc. Tests should be simple and short, having many things to solve before and after running each test can be a burden.
From the description of your question it looks like that you pretty specifically know what you're looking for.
My first reaction would be that you use PHPUnit for this. It does not qualify all your requirements, but it's a base you can build on. It's highly expendable and flexible, however it does not support PSR-0 but has an autoloader of it's own so probably that does not weight that hard.
From the information you give in your question I'm not sure if the design of your testsuite(s) or the design of your application are hindering in writing and performing the tests you would love to.
I smell sort of probably both. If your application code is not easily testable, then there is not much a testing framework like PHPUnit can do about. So for example, if your controllers do not use a request object with an interface, it's not so easy to inject some request that was not triggered by the HTTP request, but by your tests. As HTTP is most often the entry-point into a webapplication, it pays to abstract here for tests. There exist some suggestions apart from specific frameworks: Fig/Http. However this is just a pointer.
Similar is this with the database scenario you give: If your application code is depending on the database, then your tests will be as well. If you don't want to test against your database all the time, you need to have your controllers being able to work w/o the concrete database. This is comparable with the HTTP requests.
There exists numerous approaches to cope with these circumstances, but as I read you question you don't look uneducated, but it's more you're looking for a better solution than exisiting ones.
As with every own code, it's pretty hard to find something that matches the own design. The best suggestion I can give is to extend PHPUnit to add those suites and constraints you need to for your application while you use the support of automated tests to refactor your application to fit the needs of how you would like to test.
So you can start with the tests and then develop the controller like you need it. This will keep your controller light I assume and help you to find the solutions you need.
If you find something that is missing with PHPUnit, you can first extend it on your own and additionally the author is very helpful in adding missing features.
Keep in mind that if there does not exist what you need, you need to code it your own. However if you're able to share (part) of the work with others, you most often get a benefit than by doing everything alone. That's a point for an existing framework, be it for testing or the application.
So if as of yet there is no such controller / MVC that does support easy unit-testing out of the box that fits your needs, chime in and develop one TDD-wise. If done right it can exactly match your requirements. However I think you're not alone with this problem. So not a very concrete answer, but I can only say that I made very good experiences with PHPUnit and it's extendability. That includes output tests you're mentioning in your question.
And probably a little differentiation at the end: It's one thing to test code-units and another to test if they all work in concert in the application with it's various requests. The last most often requires larger test setups by nature. However, if you can separate units from each other and clearly define with which other units they interact, then you normally only need to test the interaction between those which can reduce the setup. This does not save you from infrastructure problems, but those are normally not tested with unit-tests anyway (albeit you can extend PHPUnit to perform other type of checks).
A popular framework - even with a bad design - has the big plus that components tend to be better tested by use. That normally helps to go over the first years of your application until design issues in a framework make you need to rewrite your whole code base (probably).
As controllers often are sort in the middle of everything, this can lead to the scenario that you tend to test the whole application while you only want to test the controller(s). So you should think about the design and role of the controllers and their place within the overall application, what you really want to test with your controllers, so you can really make them testable according to your needs. If you don't need to test the database, you don't need to test the models. So you could mock a model returning random data to take it to the extreme. But if you want to test if HTTP handling is right, then probably a unit that abstracts HTTP handling is needed at first. Each controller relying on this would not be needed to test (theoretically) as the HTTP processing has been tested already. It's a question of the level of abstraction as well. There is no overall solution, it's only that frameworks can offer something but you're then bound to those paradigms the framework expects. AFAIK testing in php is getting more and more popular but that doesn't mean that the existing frameworks have good support for it. I know from the zend framework that they are working on this to improve the situation since longer. So it's probably worth to look into the more recent developments in the more popular frameworks to what this leads to as well.
And for the very specifics, you need to test on your own always.
Opting to PHPUnit and own testcases however does look as a practically way to me. Code your controllers as you need them for your project in TDD and you should get what you need.
Probably the more component based approach of Symfony 2 is better fitting your needs than what you experienced with Zend Framework. However, I can not suggest you anything specific as needs highly differ within application design. What's a quick and solid solution for one application is a burden for the other. See Page Controller.
You could take a look at the http://ezcomponents.org/ witch is becoming apache zeta
There are three ways how to make eZ components available for your PHP environment, please read the whole of this article before continuing with the practical part:
Use PEAR Installer for convenient installation via command line
Download eZ components packaged in an archive
Get the latest sources from SVN
I haven't got my hands into it yet but looks like a good solution...
Seldaek: WebTestCase isn't quite the right thing - it's for testing a view directly, and a controller or model only indirectly.
A unit test case for a controller would invoke the controller, likely giving it a mock object for the templating engine (e.g. a mock Smarty object), then check the values that were assigned to that object for display: for example, if you invoked the controller for /countries/south-sudan, you could check that the template variable $continent was set to "Africa". This kind of unit testing wouldn't actually involve any template rendering in most cases.
I asked a question here on automated deployment of automatically deploying java code.
Our back-end Java API is accessed by a PHP web app. It would be nice to be able to automatically deploy this web app (along with static files like CSS, JS and images) to our web server while performing automatic testing on PHP code. Is there something similar to this for PHP?
I also wonder if as part of this automated testing you can check that each actual page loads without a fatal error. I am sure I read about a google project which allows you to write page tests e.g. click on link a, go to page b etc etc.
Thanks
You may want to look into using Phing for deployment which has features that allow testing with PHPUnit and/or SimpleTest
Maybe this question handling deployment strategies can help you.
Additionally, but maybe gone too far, tools like cruise control may help you to apply continuous integration, too.
PHPUnit with ant was my way to go for automated testing, which could be a vaild option for you, too since you're obviously using some java already :)
Since you're working with Java, you might consider using Hudson (also mentioned in your other question), which has the ability to execute build tasks consisting of several 'steps' One step step could be unit testing your Java App, another unit testing a PHP app, yet another deployinhg Java app, and another deploying a PHP app (and you could add some more ;P )
It has a Maven plugin, so you could actually use your existing Maven scripts.
EDIT: to clarify - I am asking for advice on both unit testing and user interface testing.
Currently, I don't use TDD. While I am developing an application I am constantly testing what I am working on. Testing iteration could be anything from minor function changes to entire models. I try not to code too much before I test. I like instant feedback.
Of course, with experience I can see potential problems or bugs occurring as I'm coding.
BUT, after an application is complete I will usually go through the entire app on the frontend and ensure all functions are working as expected. This means literally everything. Every add/edit/delete, sort, filter, even broken links and such.
This can take a lot of time sometimes but it does ensure my work hardly ever contains bugs after deployment.
However, I'm looking for a more standard solution. What do experienced developers do? Assume for a moment I am a single developer and so do not have a testing department etc
Do you hire beta testers (no good if app is sensitive to public use)?
Is it viable to build a series of 'general' unit tests which can for example test ALL sorting, filter functions. One for testing ALL add/edit/delete functions.
Love to hear your feedback. Will be changing the way I develop based on suggestions.
Of course as David said : Unit-testing for models and helpers, of course.
and I need to add Selenium
Selenium is a robust set of tools that supports rapid development of
test automation for web-based
applications. Selenium provides a rich
set of testing functions specifically
geared to the needs of testing of a
web application. These operations are
highly flexible, allowing many options
for locating UI elements and comparing
expected test results against actual
application behavior.
which is amazing :
watch the 2 min intro
http://seleniumhq.org/movies/intro.mov
How Selenium Works
Testing with PHPUnit and Selenium
Unit-testing for models and helpers, of course.
But you can do unit-testing on "page-requests", as well. See, for example:
Content with Style - Unit testing controllers with Zend Framework
That's a step towards integration testing. But for issues of layout and visual aesthetics, you're pretty much stuck with walking-through each request with your browser.
Usually what you code works is not that the problem, very nice if you also write tests for it. The problem is that you need to test the integration of your changes every time you make commit.
If you like instant feedback,as I do, probably you should have a look at Continuous Integration.
I've started using Hudson as CI server and I am not regretting it!
Which unit testing framework do you use for Symfony?
Lime or PHPUnit? What are the pros and cons of using them?
In my opinion, here are a few things that come to my mind :
PHPUnit is more integrated with other tools, like, for instance,
Selenium (PHPUnit can use it to open true real browsers to test your site)
phpUnderControl for continuous-integration
PHPUnit works well with Xdebug, to generate code-coverage reports
PHPUnit is more widely used ; which probably means more support
But note I don't work with symfony, nor lime...
Still, I've never heards anyone speak about it, except for those working with symfony -- that not a good thing, for the day you'll have to work with another framework (yes, this happens ^^ )
One thing that's not in PHPUnit :
"false" browser (being able to do HTTP Requests to the application, without using Selenium to open a real browser)
But some frameworks (Zend Framework does, with it's Zend_Test component) integrate with PHPUnit (or use it), while allowing injection of data into the MVC and fetching of the response, without having to issue any HTTP Request.
I don't know if symfony allows that, but that's a nice thing with ZF/PHPUnit ^^
(Yes, not a symfony-specific answer ; but of the things I said must still be valid with that framework)
Lime is a much more simple testing framework, which can be a good or a bad thing depending on how you want to use it.
The symfony library itself uses its own testing framework, Lime, to test its code base. From the symfony book:
It is based on the Test::More Perl
library, and is TAP compliant, which
means that the result of tests is
displayed as specified in the Test
Anything Protocol, designed for better
readability of test output.
I cannot vouch for the statement that the lime framework is "more lightweight" than other PHP testing frameworks as the symfony docs claim, but I do really like that it's built right into your symfony project and works well with the symfony command line tools without any additional configuration. One thing that is especially cool is that the lime tests within symfony are set to run within your "test" environment which has it's own database, symfony cache (which gets cleared out during each test session), and environment variables. This comes in handy when you want to do functional testing (checking server response and your html output in your modules/actions, versus basic unit testing). I also like that lime is super easy to pick up and understand since it's so simple. You also have the ability to put your tests into YAML configuration file rather than write the tests by hand.
Pascal is entirely right that PHPUnit is much more widely used and you'd be able to use it in non-symfony projects. There is even a plugin for it, PHPUnit symfony plugin. My best advice would be to use lime if you just wanted to jump right into writing simple tests while you develop your symfony app. But, if you have the time and hope to use these testing skills outside of the symfony world, or bring in pre-existing PHPUnit tests into your symfony code, it'd be worth your time to check out the plugin and give it a spin.