Stephan Schwab - tdd tag

Quality control in software development

Sun, 24 Feb 2008 02:13:24 GMT

When it comes to software testing a number of terms are used. You hear the simple term testing, then some talk about QA and others call for QE.

In common language test refers to the act of trying something out, checking on knowledge or behavior and other forms of comparing expected results/behavior to what can be observed. In the case of software testing you expect certain results/behavior from a software system. So you enter some values, perform a number of activities in the user interface and then compare what the systems shows or provides you with what you expected. If it matches, the software passes the test. If it doesn't match, it is deemed faulty and you write a report about the problem. Those reports are usually called defect or bug reports.

I looked up the terms QA and QE on Wikipedia and for QA I found:

Quality assurance - Wikipedia.org:
Quality assurance, or QA for short, is the activity of providing evidence needed to establish quality in work, and that activities that require good quality are being performed effectively. All those planned or systematic actions necessary to provide enough confidence that a product or service will satisfy the given requirements for quality.

For Quality Engineering (QE) it has been a bit hard to find a good definition. Wikipedia thinks it's just a synonym for Quality Assurance (QA) and if you use Google, you will find links to people calling themselves Quality Engineer as well as companies offering courses in Quality Engineering.

So I'd like to look closer at the terms and try to define them myself for the purpose of this blog post.

Quality Assurance (QA) and Quality Engineering (QE) have two different notions. These terms refer to quality and in the case of QA people want to assure quality and in the case of QE quality is being engineered. That prompts to ask how can we assure or engineer quality in software. But before we can answer that question we need to understand what quality in itself is. Here are a few quotes from an article about quality on Wikipedia:

Quality - Wikipedia.org:

Quality in everyday life and business, engineering and manufacturing has a pragmatic interpretation as the non-inferiority, superiority or usefulness of something. This is the most common interpretation of the term.

[...]

One key distinction to make is there are two common applications of the term Quality as form of activity or function within a business. One is Quality Assurance which is the "prevention of defects", such as the deployment of a Quality Management System and preventative activities like FMEA. The other is Quality Control which is the "detection of defects", most commonly associated with testing which takes place within a Quality Management System typically referred to as Verification and Validation.

So there we have it. Quality means that something is fit for a specific purpose and is not bad or faulty. And we can have people ensuring that a product doesn't get shipped with defects (Quality Assurance) while others control the level of quality by detecting and counting defects in shipped products.

Those definitions apply very well to manufactured goods. In a factory you can take every nth product from the assembly line and check it for defects. Then you create some statistics about the defects you find and in the end you may use those numbers to prompt some action, if the quality drops below a certain level you have defined as unacceptable. Then some people in the quality assurance department get asked to come up with solutions to prevent these manufacturing defects. They may talk to product engineers and change something in the product to make production easier in some way. My knowledge about manufactoring processes is kind of non-existent but apparently it works well in that industry. Where does that leave us for software?

A lot of software development practices are inspired by other industries. There is a lot of people who perceive the act of creating software as some kind of engineering and call programmers software engineers. Probably to distinguish what they do from what a programmer perceivably does. An engineers takes on more complex tasks, creates something new or enhances a machine or building while a programmer writes a sequence of instructions to tell a machine what to do in which order. That implies as well that a programmer usually does not come up with something on her own but instead gets told what the machine should do.

Now, if you have software engineers who create software, then you may want quality engineers to work on the quality of the software before it gets used by people outside of your engineering organization. One team of quality engineers may control the quality of the product before it gets shipped. Another team of quality engineers may assure that less defects make it into the product being shipped. Just as in manufacturing - isn't it?

Ways of controlling and assuring software quality

Black box testing and white box testing describe two distinctive approaches of controling software quality. A black box doesn't reveal much details to the observer. You can tell its size, maybe it's weight and then there is the fact that the box is black. That's about it. Software without access to the source code is quite similar to such a black mysterious box. A white box is viewed as one that reveals its inner workings to the observer.

When we do black box testing we explore the functionality of the software, verify its behavior and the results by comparing it to a description of what should happen. Basically the quality engineer executes the software and does what a regular user would do. Whatever goes wrong he reports as a defect. When black box testing a software the only way of performing tests is by means of the user interface provided by the software. If the software uses a database, the quality engineer can compare before and after values in the DB. But most of the times the only way of knowing whether the software does what was expected is by looking at the user interface again. Does the report show the correct values? Got the selected element highlighted? Does it print and the output looks as expected?

That requires a lot of manual work and is per se error prone. A smart quality engineer wants to automate tests to create a number of baseline tests. When he gets a new version of the software he's testing, he simply runs his test suite and can concentrate on what's new instead of manually testing the same stuff over and over again. There is a number of tools - commercial and open-source - available for programmatically drive the UI of Windows, Java and web applications. Quite famous is the open-source tool Selenium for web applications. On OpenQA, a place for open-source QA tools, you can find as well tools to automate Java Swing or Windows GUI testing. For more tools, including commercial ones, look here.

My personal opinion about black box testing is that it appears to be a good way to control the quality of a finished software product and provides information to decide whether it is safe to ship it. But it is extremely important to keep in mind that this kind of testing is based on the user interface and you still might miss hundreds of defects just because you don't execute the software in the way that makes them show up. Astonishingly a lot of organizations view this kind of testing as the most important form of testing. Probably because most people perceive software as the user interface. So what they can see and touch that must be the thing they call software.

In the beginning of this post I was talking about quality assurance and quality control. So far we have identified a tool to control the quality of the user interface. I think it's safe to say it that way. What about controlling or even assuring the quality of the software's functions? To achieve that we need to look on the inside.

White box testing requires us to work with the source code, which allows us to perform tests on internal mechanisms instead of just the user interface. The keywords that come to mind now are TDD (Test-Driven Development), unit tests and tools like those from the JUnit family. This is going to be a complex topic, as it is not only about controlling quality by measuring things. To assure quality you have to start with the process from the very beginning, which is when you analyze the problem you want to solve with software. It touches the way you develop (your development process) and essentially is not any more a matter of the QA/QE team. I think it makes sense to cover white box testing in a second post.

Coverage != testing

Fri, 16 Nov 2007 16:01:01 GMT

This morning I stumpled upon this:

Alex Miller in Coverage != testing:
Testing verifies your code, coverage verifies your testing, but you can’t say that coverage tests your code.

That's just so true. It's a good idea to start a new class with a unit test where you can model it and show how to use it. After all the test is where you first use the new code. And a unit test makes great documentation as well. It's likely that you will gain 100% coverage from the unit test. But you have not really tested yet. The missing piece is the integration with other parts of your system.

In my opinion it's a good idea to have a team member wearing the QA hat. That's the person who should be vigilantly checking whether there are enough integration tests that really take on edge cases and correctly simulate user behavior or other forms of input.

Why I love test-driven development

Mon, 16 Apr 2007 06:23:27 GMT

Here is a very good example why I am a big fan of TDD (test-driven development). Instead of writing a specification or a requirements document I wrote the following test case and let Eclipse create a few new classes for me. Then I wrote enough code to get the basic functionality right and added a mock UserService with a few expectation set.

public class EnrollmentTest extends TestCase {
  public void testSignupTrialWithoutPayingOrganization() throws UserNotFoundException {
    PaymentDetails paymentDetails = new CreditCardDetails() ;

    Person firstPerson = new Person() ;
    firstPerson.setRfc822Address("sns@caimito.net") ;

    Person secondPerson = new Person() ;
    secondPerson.setRfc822Address("fidel@caimito.net") ;

    UserService userService = EasyMock.createMock(UserService.class) ;
    EasyMock.expect(userService.findUser(firstPerson.getRfc822Address())).andThrow(new UserNotFoundException()) ;
    userService.saveUser(new User(firstPerson.getRfc822Address())) ;
    EasyMock.expect(userService.findUser(secondPerson.getRfc822Address())).andThrow(new UserNotFoundException()) ;
    userService.saveUser(new User(secondPerson.getRfc822Address())) ;
    EasyMock.replay(userService) ;
		
    SignupManager signupManager = new PerUserSignupManager() ;
    signupManager.setUserService(userService) ;

    try {
      Customer customer = signupManager.signup(firstPerson, paymentDetails) ;
      User firstUser = customer.createUserAccount(firstPerson) ;
      assertNotNull(firstUser) ;
      assertEquals(firstPerson.getRfc822Address(), firstUser.getUsername()) ;
			
      User secondUser = customer.createUserAccount(secondPerson) ;
      assertNotNull(secondUser) ;
      assertEquals(secondPerson.getRfc822Address(), secondPerson.getUsername()) ;
    } catch (SignupFailedException e) {
      fail(e.getMessage()) ;
    } catch (UserAlreadyExistsException e) {
      fail(e.getMessage()) ;
    }
  }
}

This test case is very high-level. There is more to be implemented. The work of implementing the details can be distributed amongst the members of a team. That way everybody gets involved a bit, which is good to foster shared ownership of the code base, and it allows to delegate some easier tasks to Junior Developers.

Such a high-level test case can be used as well by a Senior Developer with more design experience to more precisely prepare a task for a learner. The high-level design is there and the Junior Developer can fill in the missing pieces and design a bit of the lower level parts. He will be able to learn good design and contribute working code to the project.

[TSE] Meeting Requirements through Acceptance and Stress Testing

Sun, 10 Dec 2006 13:57:45 GMT

My fellow German developer Eberhard Wolff is speaking this morning about acceptance and stress testing to meet requirements.

As an introduction he's wrapping up what Spring provides in the org.springframework.test package.

Acceptance Tests

Now we are getting into Acceptance Tests. You use them to find errors in the implementation of business requirements or use cases. One can see them as formalized requirements. Unfortunately many times they are done manually and after the whole system has been developed. Fit/FitNesse is a tool for acceptance tests. The customer and the developer should write the acceptance tests together - so does XP say. Fit/FitNesse uses HTML as input format. Test data and expected results are provided as HTML tables. You need to write a simple class as a wrapper for business logic, which can be injected using @Configurable. This class is then used to pass data into the test and check the results.

Eberhard is working on a general Fit Exporter for Spring. It's currently Alpha code.

The SpringActionFixture can be used to test actions that are usually performed in a GUI. You specify input data and actions - e.g. press add button - and then look for results. In the HTML input file you use the expression "press" to call a method. So the UI stuff is left out, but the method that normally be called by the UI is now being called by FitNesse.

One question from the audience is whether you can run Fit/Fitnesse as part of a Maven build. As it's a command line tool that should be no problem.

Fit uses HTML files as input. Fitnesse uses Wiki pages. Triggered by a question from the audience Eberhard opinions that probably Word documents that get exported to HTML are the best solution to get business people to provide test data and rules.

Performance Tests

JMeter is a good tool to performance test web based applications. JMeter creates a large number of requests, but how do you measure the performance in the whole system. You want to measure each part of the system. Using a regular profiler for Java would not be sufficient for the purpose of performance tests, because it only focuses on the Java code itself. JAMon is a monitoring framework, which offers a Spring interceptor. For web application you configure a filter in web.xml. The JAMon filter measures HTTP request/response. JAMon profiles method calls using AOP. JAMon can be used as well to profile other parts of the system by simply declaring other AOP aspects, e.g. you could use it to profile SQL queries.

One could use JAMon even in production as the overhead is very little. It can use exiting Pointcuts and Spring AOP proxies.

[TSE] The Building Blocks of Domain Driven Design

Fri, 08 Dec 2006 16:04:58 GMT

Keith Donald's talk at The Spring Experience has started. Right now he's summarizing what Eric told us before.

A domain model does not capture every aspect of the real world. Instead it's a simplistic view of the world to serve a purpose. Discoveries during the implementation feed back into the model design.

Isolate the domain by using a layered architecture. Do not mix UI, database, external services and business objects all together. The very same isolation allows unit testing and Spring helps in the wiring of the units.

Keith shows a diagram with the user interface on top of the application and the application on top of the domain and everything on top of the infrastructure. The application layer delegates tasks to the domain layer for complex solutions.

The domain layer represents business rules and states.

The infrastructure layer interacts with the database, constructs UI widgets and deal with everything technical.

The domain layer is to isolate the domain objects from other functions in the system to avoid mixing technical and domain concepts. One should bind the model the implementation early. Implement the behavior of the application within the domain model classes without thinking about the technical infrastructure. The resulting artifact is a unit test.

In the beginning map domain entities literally to Java objects. Some of these mappings may be wrong in the end and you will need to refactor, but it's a start. Then start writing unit tests using those classes. Keith is now showing actual code of unit tests for his reward dining example.

Organize domain behaviors into coarse-grain, high-level user operations. Make user operations part of the application layer. Unit tests in the domain layer are a good place to look for application layer use-cases. The application layer encapsulates complex domain layer objects and rules and allows for freedom in the implementation. After defining the boundaries of the application layer focus on streamlining the domain model implementation.

Streamline associations between model objects. If you don't understand why an association is needed, avoid it. Association create coupling, which is generally to be avoided. Impose a traversal direction on associations.

Entities, value objects and services. Often entities are persisted to a database. Entities maintain their identity that never changes. Entities may be used by other applications and need to be tracked. They have a life cycle and the model needs to define what it means to be the same thing. String entity objects down to the most intrinsic characteristics and only add required behavior and attributes needed for that behavior. For everything else add associated objects.

Value objects are identified by their attributes and often they are interchangeable and transient. Value objects can be shared and are good candidates for immutability. They reduce bugs by avoiding invariants.

Services are activities or actions, not a thing. A service is something that makes it happen and is stateless. Services should be thin coordinators and one should resist the urge to develop fat services that attempt to solve the problem all on their own. Look for natural opportunities to encapsulate behavior in services that doesn't make sense within Entities or Value Objects. The main responsibility of a service is to orchestrate things, to coordinate. It delegates to entities to do the work.

The creation of Entities should be done using factories and repositories that restore an entity object from a persistent form. Services control and coordinate the life cycle of entities.

Someone asked whether a repository isn't a service. Keith answers is that a repository is a DAO service, but with a very simple job. It doesn't coordinate work and therefore it wouldn't be justified to call it a service.

Make sure that your factories and repositories return fully initialized objects to their callers.