Search results matching tags 'Agile Related' and 'Testing Guidelines'

Unit Testing Denial Pattern #1: Pretending the test is no longer valid

Anonymous — Wed, 30 Apr 2008 20:21:26 GMT

Today was both a good day for e and a bad day for me. Good day because I managed to get a full suite of tests that were somehow failing to work again after a refactoring that got out of hand. Bad - because out of all the tests, one of them failed.

Here's the part I'm ashamed of:

I was working with a pair, and when we both looked at the failing test, I went out and said "Oh, I think this test is no longer valid. It may just need to be removed". We both knew it was still a valid test, that for some reason was failing. But I was on such a role with making everything work, that test was feeling like a thorn in my thumb that I wanted to get rid of.

It was a hard few minutes to convince ourselves to debug through the test and realize what the problem may be

So, denial pattern #1:

Forces: You just want to get on with your code, with only one or two tests that are "ruining the party"

Behavior:

Instead of fixing them debugging them or accepting that you might not have fixed everything, you try to tell yourself that the tests may no longer be valid without actually checking your assumption, which you really wish would be true.

How to avoid: work in pairs. It's harder to ignore a problem (or at least more awkward) with 4 eyes staring at it.

Basic Guidelines for using RowTest and data driven tests

Anonymous — Tue, 18 Dec 2007 23:28:32 GMT

I took a look at the code that was used to create the NUnit RowTest Extension (which is pretty neat) and found this in one of the samples:

The highlighted areas in the code show something which I feel is problematic and is the manifestation of a unit test anti-pattern which the XUnit book calls "Interacting tests". basically, there is a dependency that is created by the first and second row, in which the second row cannot run individually.

Bad Naming once again leads to confusion

I'm not saying that this test is bad, because perhaps that is what they author was trying to test - that the class level members are not destroyed between each row. I guess we'll never know because the naming of the test is problematic - it doesn't say what the expected behavior is, or what the scenario\context is. just what the general object being test is (and I'm not even sure about that). Still, it makes for a nice example for your own tests.

The symptoms of interacting row tests

Back to the issue at hand, the symptoms detailed below are caused by the wrong usage (in my mind) of the RowTest facility. The second argument in the RowTest is provided for the sole purpose of changing the behavior of the class under test (changing s_classMember) so that the next row will be able to assert on the string.

Now this prevents you from doing the following without going into annoying routines of code replacements:

You can't remove a row test without making sure you don't break any other rows
You can't just insert a row test randomly in between without possible breaking another row
You have to read the code in the test to understand what the row test is doing.

Introducing dependencies between row test essentially makes the specific test method unmaintainable to a large degree. Since row-tests and data driven tests are a fairly new syntax for most of us, here are a few guidelines I try to follow:

Some Basic Guidelines

Treat each test row as if it will be the only one on the test method (so that you don't feel the need to interact with other rows).
Test rows must be a simple as possible to understand - that is - have no logic. In this case we do have logic int he test row (pertaining to the next row)

Data Layer Testing: Test Inheritance Patterns

Anonymous — Thu, 29 Nov 2007 01:56:51 GMT

I'm preparing for my tutorial on Database and Data Layer unit testing that I will be giving at the Agile Practices Conference next week. It's always nice to have a full 3 hours to get pretty deep into something. 75 minutes just isn't enough to grasp a concept deeply enough, just to graze it around the edges.

When writing data layer tests, there are several patterns I've noticed, especially around the notion of inheritance within the test classes (Abstract test class pattern). I've observed at least three different patterns emerging from inheritance that I've personally used, all relating to data layer testing, but could be used elsewhere:

Abstract Utility Class

This class will usually be a base class for tests written for data layer objects. it would contain "facilities" such as database helper classes, setup and teardown methods that would automatically rollback any test and anything else a derived base class that was testing a data layer object would need (helper methods for asserting complex stuff, for example).

This is a classic "Abstract test class" pattern, and the next two are built on top of it.

Template Test Class

The template test class will contain abstract methods as unit tests, which any derived class would have to override and perform. That means that if you are always testing the same things against a data layer object (insert returns different IDs, update works etc...) test authors on new data layer classes who derive from this class will never forget to implement at least these basic tests.

they would also get the facilities for doing assertions and other things from the base class, as depicted in the previous pattern. makes sure you don't forget tests, saves some time on the infrastructure.

Abstract Driver Test Class

This class will have non abstract test case methods inside it that will always run the same way. these test methods will call (using abstract methods) to the derived class' implementation of type specific actions (insert specific for a category, or update specific to person) .

That leaves the derived class to only implement very basic things such as insert (and return the ID), update, or delete (by ID). the test methods in the base class will do the rest of the work. derived classes don't need to implement any tests unless they want to add type specific tests which don't belong on the base class.

in the image to the left the green methods are tests that the base class runs, and the yellow ones are type specific helpers that the derived class has to implement.

this makes writing tests for a new data layer object (as long as its interface is the same) very simple by inheriting a test class and implementing a few methods. all the test permutations in the base class will then run on the data layer object. saves lots of time and forgetting tests.

What patterns have you come across?

Mocking LINQ Queries, Extension methods and Anonymous Types

Anonymous — Sat, 17 Nov 2007 21:47:14 GMT

One of the things I wanted to show at my Interactive session on unit testing tips and tricks at TechEd this year was how you can "Stub" out results from LINQ Queries, or mock\stub Extension methods in .NET 3.5 (what's the difference between mocks and stubs?)

The only mocking framework in existence that can truly stub out LINQ queries and extension methods is TypeMock.NET.  Another option, if you use LINQ to SQL, is to be able to mock out the DataContext and the IOrderedQueriable interface, as outlined here, to return a custom query result set.  I'll talk about the first option in this post, since it is much easier to read and understand. I may touch on the second one in a later post.

TypeMock, I've talked about it before, is a very powerful mocking framework. much more so than Rhino Mocks or NMock, because it allows isolating static methods, private methods, constructors, and basically anything you can do in IL, because it uses the .NET Profiler APIs to intercept method calls and do whatever it wants with them.  In that regard, it is almost too powerful because it rids of of the need to actually design your code for testability, and just test it as is (how you actually write the tests is another matter for another post).

But I'd rather have the argument whether TypeMock is good or bad, than not have TypeMock on the scene at all, simply because it's just too damn valuable as is (disclosure: TypeMock is owned by a friend of mine).

It's the only real framework that can deal with real untestable legacy code and still allow you to isolate classes enough to test them without changing the code under test itself.

It's the only framework that allows to write unit tests for code that is based on .NET 3.5 and uses LINQ Queries and extension methods, allowing to isolate these two types of code and thus be able to separate things from their dependencies.

Here are some examples I wanted to show (takes fro the examples that come with the TypeMock download:

Example: Stubbing out LINQ Query Results

First, we're going to create one fake result to return (array) from the LINQ query, and also introduce the real data source from qhich to query (realCustomerList). note that the customer list has 3 entries, and the fake list has only two entries.

 ///Two Customer instances used as fake values

 private static Customer fakeCustomer1 = new Customer { Id = 0, Name = "Fake1", City = "SF" };

 private static Customer fakeCustomer2 = new Customer { Id = 1, Name = "Fake2", City = "Redmond" }; /// A fake list used as return value in the tests private List<Customer> fakeCustomers = new List<Customer> {fakeCustomer1,fakeCustomer2 }; /// A list containing 3 cusotmer use as the "real data" private List<Customer> realCustomerList = new List<Customer> { new Customer{ Id = 1, Name="Dave", City="Sarasota" }, new Customer{ Id = 2, Name="John", City="Tampa" }, new Customer{ Id = 3, Name="Abe", City="Miami" } };

Here's how the unit test looks:

 1: [TestMethod]

 2:  public void MockSimpleQuery()

 3:  {

 4:  using (RecordExpectations r = new RecordExpectations())

 5:  {

 6:  var answer = from c in realCustomerList select c;

 7:  r.Return(fakeCustomers);

 8:  }

 9: &#xA0;

 10:  var actual = from c in realCustomerList select c;

11:

 12:  Assert.AreEqual(2, actual.Count<Customer>());

 13:  Assert.IsTrue(actual.Contains(fakeCustomer1));

 14:  Assert.IsTrue(actual.Contains(fakeCustomer2));

 15:  }

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, "Courier New", courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

Line 6 is the one that tells the TypeMock record what LINQ query to intercept.

Line 7 tells TypeMock what value to return when this query is intercepted. that means the query will never really take place.

finally, we assert that what we got from the query is the fake list of customers. Pretty darn easy.

But is this a good test? Not really. this is only a demo of what you code intercept with typeMock. a real unit test will actually test a piece of code that runs this query, and, instead of providing it with a list of objects to query, will just provide it with the fake return value to receive

Example: Test that an anonymous type is created correctly

 1:  [TestMethod]

 2:  [VerifyMocks]

 3:  public void MockAnonymousTypeTest()

 4:  {

 5:  using (RecordExpectations rec = new RecordExpectations())

 6:  {

 7:  //Mock the creation of the Anonymous Type

 8:  var p1 = new { Name = "A", Price = 3 };

 9:  rec.CheckArguments();

 10: &#xA0;

 11:  //fake the value of the Name Property

 12:  rec.ExpectAndReturn(p1.Name, "John");

 13:  }

 14:  //if creation will be done diffrently an exception will be thrown.

 15:  var target = new { Name = "B", Price = 3 };

 16:  //verify that the fake value is returned

 17:  Assert.AreEqual("John", target.Name);

 18:  }

Line 8 tells the typemock recorder that this is the anonymous type creation we'd like to intercept.

Line 9 tell it to assert internally that the anonymous type is indeed created with the correct "Name" property value and the correct "Price" property value. so if I later initialize the anonymous type with the wrong values, I will get a test exception based on the expected values. Pretty neat.

Line 12 is just an example of how you can also intercept and retun whatever value you'd like from the property of an anonymous type. in this case the name will always return "John" even though it may have been initialized differently.

Example: Mocking Extension Methods

Extension methods are static methods, which means you can't replace them with a different method instance. there are ways to design yoru code so that the calls to the static methods are testable, but assuming you're testing code that cannot be changed, or that is hard to redesign, intercepting the method call itself is almost the only choice.

assume you've extended the Point class by adding this extension method:

 public static class Extend { public static Point Multiply(this Point extendedInstance, int scalar) { extendedInstance.X *= scalar; extendedInstance.Y *= scalar; return extendedInstance; } }

&#xA0;

Now you could write a test like this:

 1:  [TestMethod]

 2:  [VerifyMocks]

 3:  public void MockExtensionMethod()

 4:  {

 5:  using (RecordExpectations rec = new RecordExpectations())

 6:  {

 7:  Point mocked = new Point(7, 9);

 8:  mocked.Multiply(6);

 9:  rec.Return(new Point(5, 5));

 10:  }

 11: &#xA0;

 12:  Point target = new Point(7, 9);

 13:  Point actual = target.Multiply(6);

 14: &#xA0;

 15:  //Verify the returned vcalues

 16:  Assert.AreEqual(5, actual.X);

 17:  Assert.AreEqual(5, actual.Y);

 18:  }

in line 7 we're actually using the extension method as part of our recorder, and telling the record to return a specific value. then we later use that value in our code under test (like 13) and assert that we god the mocked value instead of the real one (the extension method is not executed).

What about other technologies?

could you use this to test code that relies on other non testable technologies, like windows workflow, WCF etc? the answer is a big yes. the same principles apply, and most of these frameworks were not built with testability in mind.

it's in those cases where people ask "how can I test such code where everything is sealed, private and static?" that I tell them that, currently, TypeMock is the only framework that solves this problem at a satisfactory result.

The various meanings of TDD

Anonymous — Tue, 09 Oct 2007 01:49:57 GMT

Now that I'm having some real time to write something I thought I'd try to write down my thoughts starting all the way back from JAOO and what I learned from it. Specifically the whole notion of TDD, and what does TDD mean to different people.

"TDD will deteriorate your design". that was the sentence I heard coming out from the mouth of Jim Coplien, a speaker at JAOO and a well respected guru in the software industry in many levels. I tried to grasp what he was saying again, so I repeated the sentence in my head. "TDD will deteriorate your design, so don't do it". Yes. I heard correctly.

It was lined throughout the discussion Jim was having about Scrum and Architecture and was just one small point he made which he didn't even take the time to review fully. "Will deteriorate your design" and then nothing. Someone from the crowd of about 400 challenged him and Jim was starting to quote a couple of research papers which I don't have a link to so I have not read them. Jim was kind enough to leave a comment on my previous post about JAOO but that did not include any links to those research papers. Could anyone leave a comment here with a link to them?

When Jim walked out of the hall I started talking to him for what ended up to be about 40 minutes of intense rhetorical debate of what does TDD really mean and then what benefit if at all does it have. I'll weave points from that in but first I need to say that I realized something: TDD is not a clear enough concept so people see it in different light depending on who you ask.

What are the possible meanings of TDD?

1) Test Driven Development: the idea of writing your code in a test first manner. You may already have an existing design in place.
2) Test Oriented Development: Having unit tests of integration tests in your code and write them out either before or after our write the code. Your code has lots of tests. You recognize the value of tests but you don't necessarily write them before you write the code. Design probably exists before you write the code
3) Test Driven Design(the eXtreme Programming way): The idea of using a test-first approach as a fully fledged design technique, where tests are a bonus but the idea is to drive full design from little to no design whatsoever. You design as you go.
4) Test Driven Development and Design: using the test-first technique to drive new code and changes, while also allowing it to change and evolve your design as an added bonus. You may already have some design in place before starting to code, but it could very well change because the tests point out various smells.

It's important to realize the distinction between these four perceived notions of what TDD could mean because when Jim said "don't do TDD" he meant one of these. Unfortunately he didn't really explain which one he meant (maybe he hadn't realized they exist) so the message he gave out was essentially "Whatever you think TDD is, don't do it".

Which of these notions corresponds to what you think TDD is? which do you think I would choose?

From my conversations with Jim it would seem that he think about TDD as option number 3 - Test Driven Design. He was saying that if you do it the XP was where you let the tests drive all of your architecture you're bound to get into trouble so leave it alone. I'm leaving the argument of whether pure test driven design works or not for later. Right now I want to make sure we're all on the same line.

If you're going to say TDD doesn't work - explain what you mean by TDD or be prepared to be bombarded with questions which you may feel have no connection with what you're talking about.

My notion of TDD is #4 - Test Driven development which can also help evolve and better your design. I always start with some sort of design, even a basic one. Sometimes so basic it's all in my head or on a whiteboard, but there's some skeleton there I use to guide me. I also let it change if my tests point out that there might be some testability problem here and there. I'm not saying pure Test Driven Design does not work. I'm sure it works for people, and I'm sure if failed others.

The Alpha Architect and Test Driven Design

One interesting thing that came up in that conversation (which many people assembled and chipped in with their on thoughts in) was the idea of the "Alpha Architect". There seemed to be agreement that the people who can do pure test driven design (like Ron Jefferies) probably already hold all the design they need in their head, so they don't get why other people can't do the same thing. These "Alpha Architects" are pretty unique and perhaps live in some sort of bubble where things like TDDesign feel much easier and simpler. I'm not sure that's the case but I can understand the idea and willing to agree that it's possible.

So, is Test Driven Design only good for teams with Alpha Architects in them? not sure, but it's a question worth asking.

So, is TDD Bad?

Are the rest of the notions of TDD bad? In our talks Jim agreed that unit tests or integration tests might not be a bad idea after all as long as they don't kill your design. When he said "but I was saying TDD kills design I said it because that's what people think TDD is!". The only reply to that is that by telling them that you are only reinforcing their belief that they know what TDD is, and not even questioning their assumption (or yours) that it could be a good thing.

I think that TDDAD(Development and Design) is a good thing. I'd love for my assumptions to be challenged. I'm still waiting to get the links to the two research articles mentioned.

your comments are appreciated.

possible topics for the future posts:

Things I realized about XUnit from talking to Brad and Jim
BDD vagueness and tooling
Ruby coolness
The ALT.NET Conference roundup

The Art Of Unit Testing - Available to Purchase Online Now

Anonymous — Wed, 26 Sep 2007 15:49:42 GMT

I'm happy to announce that you can now purchase my book The Art Of Unit Testing through the Manning Early Access Program. Currently, the first five chapters are available, but when you purchase the MEAP version you get the new chapters as they come in, before anyone else. Also, you can help make the book better and more accurate as a MEAP reader since the author forum can be used to discuss book topics you find interesting and I will be reading it at least twice a week (hopefully daily).

What's with the ugly cover art? I'm working on it. We had to have something to start putting this online. The final print will have something which I can actually stand behind.
What are the first five chapters about?

The way I see it, the book is divided into several parts. the first part (chapters 1-5) are about the basic tools you need to know to start writing unit tests in the real world. Everything from working with NUnit and the basic attributes of unit tests, and up to using fake, stub and Mock objects. There is also quite a full reference for using the Rhino.Mocks framework. The idea is that you can take someone with no unit testing experience whatsoever in .NET, and have them read those five chapters. then they have the basic knowledge to actually know what you're talking about when you say "Mocks" or "integration testing". They may even be able to write some real world tests using these chapters, even interaction tests.

Part I Getting started
1. The basics of unit testing - FREE
2. The first unit test - AVAILABLE
Part II Core techniques
3. Using Stubs to break dependencies - AVAILABLE
4. Interaction testing Using Mock Objects - AVAILABLE
5. Mock Object frameworks - AVAILABLE

Code-Reuse in unit tests is more important than you think

Anonymous — Sat, 22 Sep 2007 10:51:06 GMT

Scott Bellware mentions how he really likes the new ideas in XUnit.NET, especially the lack of [Setup] and [TearDown] methods. (Brad Wilson wrote in the comments to my post about XUnit.NET that you can mimic this behavior by using the constructor of the test class and the IDisposable.Dispose() method of the class -just implement IDisposable) .

He especially says this:

"I think that Jim is on the right track, but I'm the kind of guy that feels that a test class's greatest responsibility is to document behavior in the clearest possible way, even if that means sacrificing design qualities best reserved for functional code - like reuse."

(emphasis mine)

That's silly. Scott tries to write "pure" TDD tests by saying that tests are 100% different than functional code, but they are not. Proof -- Tests that don't have code re-use in the form of setup and teardown or various specialized factory methods will be harder to maintain because every time you change some semantic aspect of your production code (constructors, initialization phases etc..) many of your tests will break and you'll have to fix them all one by one because you don't have re-use for object creation in your tests. Pretty soon you'll just stop maintaining your tests and that means you'll stop writing new ones. Ta-da! mission accomplished.

If we don't all realize that in the real world tests have to be as maintainable (or even more so) as production code, we won't come out of the middle ages of unit testing.

he then goes on to say some rules he has about Setup methods. one of which is:

"Don't put anything in a setup method that doesn't pertain to every test method in the test class"

I completely agree with this. You NEED to have re-use but you also need to make sure you don't abuse these abilities.

TDD Pattern: Composite Logic Development and Testing

Anonymous — Tue, 18 Sep 2007 12:32:26 GMT

Here's a video I made that captures my current thoughts on writing complex business rules which have lots of validation logic.

Here's the scenario I show in the video:

A Validate method on an object actually uses lots of smaller methods to return a result. The idea is to actually write separate tests for the underlying helper methods (which are actually individual rule checks) and a separate test for the overall logic driver (the Validate method.

Different from traditional TDD technique

This is different from the traditional way I would do TDD in that in the traditional sense, in TDD you probably would not have these helper methods but would actually write all that logic inside the validate method, and only later you would refactor the code into separate smaller methods.

TDD: Start with a simple method and refactor it into smaller methods if and when you need to while making the tests pass.
This method (Composite Logic TDD - CLTDD???): drive the creation of the separate rules as part of the TDD effort (part of the initial design).

But why? Because every time I tried to accomplish the same thing with the traditional way I came up with problems thinking about my next tests. every time I would add another rules to the implementation it would break my existing tests because they had to account for my new rule.

In the new method I gain full control over what the rules actually return so that I can tell the validate method - "Assume that all the rules pass, except this one". then I can go on and test things separately. here's what I mean:

So the video details how you would write all this in a test driven manner. I'm personally not entirely happy with it and i think the technique is still rough around the edges (takes time to learn, not very intuitive etc..) but it's currently the only way I could come up with to test the logical driven separately from the individual rules it uses.

Separation of concerns

The thing I like about the end result is that the individual rules methods (HasDIgit() etc..) can be coded and tested independently from the logic that uses them. much like separating a call to a web service from the actual web service implementation and testing them separately.

Your input is welcome. I'm looking for:

Giving a proper name for this technique\pattern (the XUNit test patterns book calls the pattern "Test Specific Subclass" with the variation of "Behavior changing subclass" but what I'm doing is a pattern for TDD that tests a composite logic piece of code. what should we call this technique?
Tell me where I'm missing something in the technique that could lead to potential problems
tell me if you think you have a better way of accomplishing this (in TDD or not)

Here's the video, 44 minutes.(the demo of the validate method starts at minute 21)

in the video(you can also download it 57 MB):

understanding the problem
the different kinds of tests for this problem
a simple demo of the extract and override pattern
the full demo of using the Validate method

here are the slides for this video.

Mocks and Stubs - The difference is in the flow of information

Anonymous — Sun, 16 Sep 2007 21:26:55 GMT

In class today when explaining the difference between stubs and mocks, I realized a simple explanation to this:

Stubs and mocks may seem the same but the flow of information from each is very different.

Stubs provide input for the application under test so that the test can be performed on something else.
Mocks provide input to the test to decide on pass\fail. the opposite direction.

The following diagram shows the flow of critical information:

The red arrows signify the "critical" flow of information that is used to decide if the test had passed. A stub can never fail a test, so there is no flow of information back to the test. the mock records things and then tells our test if it's expectations have been met according to its recording.

A stub is application facing, and a mock is test facing. Here's a more in depth overview of the two, but I haven't seen this explained as I just did anywhere. It's important to know and distinguish the two since many frameworks for mocking use these terms for different kinds of objects.