I am developing some project with CodeIgniter and write unit tests and web tests in SimpleTest. I've noticed that my tests are not deterministic, i.e. they produce different outputs in time. I mean the test cases that should be strictly deterministic, not relying on random variables etc.
The tests look like affecting each other. Quite often, when everything goes okay, I have let's say 100 passed tests, but when I write a new test method that fails, then several other tests also do fail. But often after correcting the problem in my failing test case and re-running whole test suite 2-3 times whole suite gives a pass again.
This happens with WebTestCases generally.
Do you have any idea what could be the problem?
I do not modify any class variables that are shared etc.
I've glance at the code of SimpleTest (more or less, it's big to analyze whole flow quickly) and it looks like the instance of browser is re-created before launching different tests.
The thing that is the strangest is that after re-running, some errors disappear, and finally, all of them. Is there some caching involved in this?
I'll be grateful for hints as there is really not much documentation / blog entries / forum posts about SimpleTest in the web, except its API on the website.
Things it might be:
Caching - are you caching bad results
somewhere in the chain?
Misunderstanding - Are you sure you
are testing the right things?
Bad Data - If you are testing this on
top of a database, and the failure
corrupted the data in the database,
you might see results like you
mention.
(edit: moved the answer as a separate post)
Huh, I made quite thorough investigation and it seems that there is a bug in SimpleTest library.
They use fsockopen for opening connection, then send request via fwrite, and then incorrectly fetch response from socket. What I mean: it can happen that we read 0 bytes from socket, but we're not done, as we falsely assume, cause the server can be busy, and send data later, while we prematurely ended reading. That way, we haven't read whole response and we do tests against only partial response, causing it to fail.
Related
I'm writing some classes for a personal project that handles data and ensures it conforms to my exact specifications. The output is an array (JSON object for API) consisting of true\false values for the respective fields to see if they passed my tests.
Some of the tests are complex with multiple conditions and it is hard determine at face value why a field failed the tests.
For each of the tests I was thinking of putting a DEBUG flag which can be enabled at object instantiation. That leaves me with two choices.
A) Copy the validate function as follows. Within the debug_validate() function I return an additional array with the fields and the reason they failed. This requires me to maintain two copies of the function.
if ($this->debug_mode) {
debug_validate();
}else{
validate();
}
B) Within the actual validate function I can do the following
if (!test1($field)) {
$validation_result[$field]=false;
if($this->debug_mode) {
$debug_result[$field]="Field is too small. Result of strlen: ".strlen($field);
}
}
.
.
.
This means that I have to run the "if" blocks every time the validation function runs even though it will not be used 99.9999999% of the time. It seems inefficient.
Am I overthinking this. I mean with modern hardware who cares if it runs an extra if statement. I just wonder if in production this actually matters when it comes to millions of requests (have to think big :)) and these debug statements start to add up (few MS here, few MS there)
Well if you were writing this for a company, then yes it would be a bad practice. The team would kick back the code to you during a code review & have you remove the debugging code.
Since you're developing a personal project, then you can use debugging code for now. The benefit is that it will help you debug the project faster, than without it. You can clean the code up later on, when you bake it for it's final production release. Or if you're switching from a beta to gold status in a future release (say 6 months or 1 year from now).
The hard part is seeing what's going on with a server, where you don't have direct access to the source code. That's usually what occurs on a QA, test, staging or production server. You can easily edit the code on your localhost box, but you won't be able to easily edit it on those other servers.
What you don't want to do is leave this open, without writing a way to securely protect it. You don't want others to be able to trigger this code... if ($this->debug_mode) {. If you put something important inside of that PHP block... like SQL... then someone can see it & start to understand how your database is laid out. It'll give them a clue as to where to go, to start hacking into your database. They'll want to start elevating their user access privileges! You don't want them to make themselves an administrator & lock you out of your own database! So make sure that you secure your code with something like this:
if (($_SERVER['HTTP_HOST'] === 'qa.yoursite.com') && ($this->debug_mode)) {
debug_validate(); // <- QA uses this.
} else {
validate(); // <- Production uses this.
}
Note: That only runs on a QA server, but it runs for everyone who can use that server! So be careful that you don't give access to the QA server to anyone else but yourself!
If you have a password protected login with a userId, then you can use this example. It will work on any server. You can also combine these 2 code examples to lock a userId to a specific server.
if ($myUserId === 'myUserId') {
debug_validate(); // <- You'll use this.
} else {
validate(); // <- Everyone else uses this.
}
Ideally, what you want to do is to only use the validate(); code without the debug_validate(); code and setup a unit test to run against your code. It should include tests to verify that everything validates correctly. It should also include tests to verify that bad data won't validate. When your site gets to that level, then you won't need to use the debug_validate() method anymore. The unit tests will report if everything is good or if anything fails.
I wouldn't worry about the ms load time at this point. Nor the server load. The PHP code will run so fast that you won't notice it. Focus more on getting the code to work properly, than making it look pretty. You can always refactor the code later to clean it up once it's working. It'll make you feel happier when you see the working code getting cleaner!
Your production code would have to evaluate an extra "IF" conditional in either case, unless I am misunderstanding? Whether it lies within the function, or within the calling script, the additional overhead (which will likely be extremely minimal even under load, unless you have thousands of them) would be the same either way.
I have had production code with "Debug Mode" type flags in the past and it has worked fine - however, there are some problems with that approach - namely, you need to change code (i.e. toggle the flag on or off) and therefore do another build / deployment in order to debug. Also, remembering to turn on / off debug mode can be a chore. And it tends to dirty up your code / make it less readable / easily maintainable.
Personally, I would take a little bit different approach, especially if you think debugging will be a common thing for your app - You could write separate unit tests to do debugging, using a tool like PHP Unit. If you are logging incoming requests to your application in some shape or form, it should be easy to pipe production data over to your unit test code base and conduct your debugging in that environment, instead of in your production code.
If you are running a test driven development environment, writing unit tests first should be common practice anyway.
First of all - there seem to be many questions about the fundamentals of tick functionality, so I want to add the top user comment at php.net/declare to the pile for anyone looking for further information. I found it while digging around as I tried to figure out the following.
So, I'm working on writing a simple debug helper. I want to add function tracing and benchmarking - basically what tick functionality is perfect for.
Thing is, I want to enable and disable benchmarking depending on arbitrary conditions that occur during script processing. I'm not really looking for fixed debugging à la scoped declare() { ... }.
What I'm looking to do is to put declare(); at the top of my script, and then register and unregister my debugging/benchmarking (tick) function as appropriate. Un/registration won't happen (too) often, so is efficient and reasonable.
But then I got curious: when I don't have a tick function registered... does the fact that I've run declare(ticks=1); have any effect on execution efficiency? Does it cause any extra processing to become permanently enabled anyway?
Analysis of PHP(7)'s source code shows that the answer is technically yes, but I'm not yet sure how.
The answer seems to be in zend_compile.c:8200: it appears this function defers compilation processing to the appropriate routines, then if ticks are enabled it additionally emits a ZEND_TICKS opcode into the opline via zend_emit_tick() in :2167. The opcode reference page for TICKS seems consistent with this conclusion; it shows an example disassembled opcode listing which has TICKS opcodes scattered throughout it, and I was wondering how they got in there until I discovered the above.
The ZEND_TICKS handler (in zend_vm_def.h:6859) seems to call zend_ticks_function(). This is mapped to ticks_function() in zend.c:754, which is in turn mapped to php_run_ticks() in main.2013. This is finally defined in php_ticks.c, where it's all of:
void php_run_ticks(int count)
{
zend_llist_apply_with_argument(
&PG(tick_functions),
(llist_apply_with_arg_func_t) php_tick_iterator,
&count
);
}
Huh. Not bad.
But here's the thing. If I declare(ticks=1);, the above dispatch is being run for literally every statement executed. That's... ouch. For long-running scripts containing high-iteration-count, tight processing loops, I'm wondering how badly that'll add up.
Problem is, I'm not even sure how to benchmark this. The only way I could envisage to do so would be to synthesize some PHP bytecode and then figure out a way to inject that directly into the PHP bytecode interpreter.
And that leads to my question: how much of a performance impact does this additional dispatch have, in practice? How can I quantify it?
Obviously the above investigation was performed on the canonical PHP.net interpreter. I haven't looked into how HHVM does this at all (yet), but I wouldn't at all mind learning how it handles it.
When writing PHPUnit tests, some of them turn out to take quite long to execute. To check certain functions, I need to check a lot of different combinations of variables, and this leads to a very long test.
Is is possible to tell PHPUnit how far the test has progressed so it can output the intermediate progress for the test to the command line? Now it is just waiting there for 2 minutes without any indication of progress.
I am just looking for an extra progress indicator that would be part of PHPUnit, and not just a custom echo, which I can write myself just fine.
I am sorry, but this is not possible.
However, if you need to invoke the same piece of code with different data, for instance call the same method with different arguments, then you should have a look at data providers.
First of all, I don't know if it is called Unit Testing. If it has a different name, feel free to correct me.
I'm currently developing web applications like this.
Let's say I'm developing a form to save values into database. I develop HTML and PHP. I time to time press F5 in Browser and check if HTML/jQuery has no bugs and PHP doesn't give errors like missed semicolon.
When it is complete and everything is ready to be tested, I start testing small pieces of my code. Like;
-Do $_POST array correctly obtains values from form file? (I test with print_r)
-Is "$email" variable correctly sanitized to be a valid e-mail? (I test it with different posibilities, eg: aaa#bbb, aa#bb.com, a##b.net etc.)
-Is the form submitted passed all controls, and successfully inserted to the database? (I check my MySQL table.)
-Does the form shows error/success messages correctly?
-...etc.
If it works with correct values, and fails with wrong values; I believe the form is working as intended and there is no bugs, so I move onto other things.
Instead of doing this, I want to ensure things works perfectly, otherwise notifies me about the problem without spamming F5 on browser.
Like this;
<?php
/* Unit Testing Start --
-ensure: isset($_POST['submit']) returns TRUE;
-ensure: isset($email) returns TRUE;
-ensure: isValidEmail($email) returns TRUE;
-ensure: connectDatabase() returns TRUE;
-ensure: getMysqlAffectedRows() returns 1;
-ensure: hasErrors() returns false;
*/
?>
My form codes, uses the functions I posted above.
When it runs, it should show me a message like this: (preferably logging it also)
Test complete. Tested (6) possibilities, (4) succeeded, (2) failed.
Failed 1: isValidEmail() returned FALSE, expected: TRUE.
Failed 2: getMysqlAffectedRows returned NULL/0, expected INTEGER 1.
Something like this would save me alot of time and make it easier to maintain my codes.
I do NOT want to do this:
$email = 'xxx#yyy.com';
echo isValidEmail($email) ? 'true' : 'false';
Then I have to erase those lines from my PHP scripts. Tests has to stay there forever, unless I manually delete them. They should be like comments so in production website, they won't be executed. However, in my development PC, I'll install whatever I need to install and those comments has to be parsed/logged.
So, yeah. That's all. Now onto the questions:
Is it something possible to do with PHP? If so, how can I do this?
What is it called, Unit Testing, or what?
How do you guys test whatever you develop?
How do you guys manage to develop huge web applications without worrying; "If I change this function, the entire website may break."? Also, how are you being sure changing function didn't break anything? X page may work fine but Y may be broken because of something you didn't think before. Like, division by 3 on X page may work fine, but division by 0 on Y page will definitely give errors, but you only checked X page?
I'll really be glad if you can reply to my questions and help me be a better developer.
Thanks!
Yes, automated testing is a cornerstone of solid software development, precisely because it's impossible to keep checking everything manually by clicking in the browser. Unit tests are typically "technical" tests that make sure a specific "unit" (typically a function or class) returns expected return values for a specified input. Functional testing tests larger units of code for correct behavior. Acceptance testing tests the final application from the point of view of a user.
There are a number of frameworks and tools to cover these different needs:
PHPUnit - the de facto unit testing framework for PHP
Behat - a testing framework focussing on offering "business readable" tests
Codeception - a framework trying to be both readable and technical
All of the above have excellent documentation which ease you into the thinking of working with unit tests. I recommend you start by reading into PHPUnit, then look at what Behat or Codeception can offer you.
As general advice: Test whether your code behaves correctly. Do not test "what is does", test whether you get the result you expect when you do something. For example, don't test isset($_POST['submit']). That's too detailed, there's no point in covering every single line of your application with a test. Instead, test larger units of code. Test that when you submit a form with known given values that your code correctly redirects you to the next page. Or test that your authentication system correctly denies access to unprivileged users. You want tests to read like this:
Scenario: Login
Given I am on the login page
When I log in as user "Jon" with the password "foo"
Then I should be logged in
Or:
Scenario: Deny unprivileged user
Given I am logged in as the user "Jon" with access level 1
When I try to access the action "restricted"
Then I should be denied access
Not:
Scenario: Login
Given I submit a form with the values "Jon" as username and "foo" as password
When I check the $_POST array I should see "Jon" in the key "name" and ...
...
The above can literally be tests in Behat by the way.
To make sure these tests you wrote are actually worth something, you need to run them on a regular basis. Maybe you create a trigger in your version control system to run your test suite automatically when code is checked in and deny checkins which fail tests. Best though is if you have a Continuous Integration server set up which regularly gets the latest code from your repository and runs the tests on it. If you set this up properly, you'll automatically be notified about all sorts of edge problems that easily go overlooked during regular development. For instance, the CI server should try to set up and run the application from scratch every time and may alert you that there's a problem getting a dependency from a third party you depend on, or that there's a problem in your database migration scripts. Things you wouldn't have noticed otherwise, since you don't always fetch dependencies and don't always re-run your migration scripts.
In the end you're aiming for having your code "exercised" automatically on a continual basis from all possible angles. That's the only way to find problems with it.
For CI software, personally I like TeamCity a lot, but Jenkins and Hudson are very popular and there are a ton of other CI tools.
Acording to WikiPedia:
In computer programming, unit testing is a method by which individual
units of source code, sets of one or more computer program modules
together with associated control data, usage procedures, and operating
procedures, are tested to determine if they are fit for use.
I'm guessing this what you are doing can be called UnitTesting.
The best way to UnitTest is though a well know framework. There are few for PHP. I guess the most popular is PHPUnit. Appart from testing it does other cool things like code coverage report (I'm sure other frameworks do it as well).
I work on Eclipse and I have PHPUnit integrated with my IDE. While developing I don't have to switch between windows. I just run a test and see does my code work or not. It saves lots of time.
It sounds like you should check out the simpletest.
Okay, so I'm relatively naive in my knowledge of the PHP VM and I've been wondering about something lately. In particular, what the request lifecycle looks like in PHP for a web application. I found an article here that gives a good explanation, but I feel that there has to be more to the story.
From what the article explains, the script is parsed and executed each time a request is made to the server! This just seems crazy to me!
I'm trying to learn PHP by writing a little micro-framework that takes advantage of many PHP 5.3/5.4 features. As such, I got to thinking about what static means and how long a static class-variable actually lives. I was hoping that my application could have a setup phase which was able to cache its results into a class with static properties. However, if the entire script is parsed and executed on each request, I fail to see how I can avoid running the application initialization steps for every request servered!
I just really hope that I am missing something important here... Any insight is greatly apreciated!
From what the article explains, the script is parsed and executed each time a request is made to the server! This just seems crazy to me!
No, that article is accurate. There are various ways of caching the results of the parsing/compilation, but the script is executed in its entirety each time. No instances of classes or static variables are retained across requests. In essence, each request gets a fresh, never-before execute copy of your application.
I fail to see how I can avoid running the application initialization steps for every request servered!
You can't, nor should you. You need to initialize your app to some blank state for each and every request. You could serialize a bunch of data into $_SESSION which is persisted across requests, but you shouldn't, until you find there is an actual need to do so.
I just really hope that I am missing something important here...
You seem to be worried over nothing. Every PHP site in the world works this way by default, and the vast, vast majority never need to worry about performance problems.
No, you are not missing anything. If you need to keep some application state, you must do it using DB, files, Memcache etc.
As this can sound crazy if you're not used to it, it's sometimes good for scaling and other things - you keep your state in some other services, so you can easily run few instances of PHP server.
A static variable, like any other PHP variable only persists for the life of the script execution and as such does not 'live' anywhere. Persistence between script executions is handled via session handlers.