This is Matt Raible's report from the first Selenium Users meeting at Google in Mountain View. I post the link here, as it contains many useful information about the current use and future of Selenium.

Matt Raible in Last Night's Selenium Users Meetup at Google:
Last night, I attended the inaugural Selenium User Group meetup at Google's Campus in Mountain View. It was an excellent event, with many of the core committers on hand to present and answer questions. Each presenter had about 5 minutes to speak and we learned many things about the Selenium Project itself, what's coming in the future and how Google has standardized on Selenium as their integration testing tool of choice.

When it comes to software testing a number of terms are used. You hear the simple term testing, then some talk about QA and others call for QE.

In common language test refers to the act of trying something out, checking on knowledge or behavior and other forms of comparing expected results/behavior to what can be observed. In the case of software testing you expect certain results/behavior from a software system. So you enter some values, perform a number of activities in the user interface and then compare what the systems shows or provides you with what you expected. If it matches, the software passes the test. If it doesn't match, it is deemed faulty and you write a report about the problem. Those reports are usually called defect or bug reports.

I looked up the terms QA and QE on Wikipedia and for QA I found:

Quality assurance - Wikipedia.org:
Quality assurance, or QA for short, is the activity of providing evidence needed to establish quality in work, and that activities that require good quality are being performed effectively. All those planned or systematic actions necessary to provide enough confidence that a product or service will satisfy the given requirements for quality.

For Quality Engineering (QE) it has been a bit hard to find a good definition. Wikipedia thinks it's just a synonym for Quality Assurance (QA) and if you use Google, you will find links to people calling themselves Quality Engineer as well as companies offering courses in Quality Engineering.

So I'd like to look closer at the terms and try to define them myself for the purpose of this blog post.

Quality Assurance (QA) and Quality Engineering (QE) have two different notions. These terms refer to quality and in the case of QA people want to assure quality and in the case of QE quality is being engineered. That prompts to ask how can we assure or engineer quality in software. But before we can answer that question we need to understand what quality in itself is. Here are a few quotes from an article about quality on Wikipedia:

Quality - Wikipedia.org:

Quality in everyday life and business, engineering and manufacturing has a pragmatic interpretation as the non-inferiority, superiority or usefulness of something. This is the most common interpretation of the term.

[...]

One key distinction to make is there are two common applications of the term Quality as form of activity or function within a business. One is Quality Assurance which is the "prevention of defects", such as the deployment of a Quality Management System and preventative activities like FMEA. The other is Quality Control which is the "detection of defects", most commonly associated with testing which takes place within a Quality Management System typically referred to as Verification and Validation.

So there we have it. Quality means that something is fit for a specific purpose and is not bad or faulty. And we can have people ensuring that a product doesn't get shipped with defects (Quality Assurance) while others control the level of quality by detecting and counting defects in shipped products.

Those definitions apply very well to manufactured goods. In a factory you can take every nth product from the assembly line and check it for defects. Then you create some statistics about the defects you find and in the end you may use those numbers to prompt some action, if the quality drops below a certain level you have defined as unacceptable. Then some people in the quality assurance department get asked to come up with solutions to prevent these manufacturing defects. They may talk to product engineers and change something in the product to make production easier in some way. My knowledge about manufactoring processes is kind of non-existent but apparently it works well in that industry. Where does that leave us for software?

A lot of software development practices are inspired by other industries. There is a lot of people who perceive the act of creating software as some kind of engineering and call programmers software engineers. Probably to distinguish what they do from what a programmer perceivably does. An engineers takes on more complex tasks, creates something new or enhances a machine or building while a programmer writes a sequence of instructions to tell a machine what to do in which order. That implies as well that a programmer usually does not come up with something on her own but instead gets told what the machine should do.

Now, if you have software engineers who create software, then you may want quality engineers to work on the quality of the software before it gets used by people outside of your engineering organization. One team of quality engineers may control the quality of the product before it gets shipped. Another team of quality engineers may assure that less defects make it into the product being shipped. Just as in manufacturing - isn't it?

Ways of controlling and assuring software quality

Black box testing and white box testing describe two distinctive approaches of controling software quality. A black box doesn't reveal much details to the observer. You can tell its size, maybe it's weight and then there is the fact that the box is black. That's about it. Software without access to the source code is quite similar to such a black mysterious box. A white box is viewed as one that reveals its inner workings to the observer.

When we do black box testing we explore the functionality of the software, verify its behavior and the results by comparing it to a description of what should happen. Basically the quality engineer executes the software and does what a regular user would do. Whatever goes wrong he reports as a defect. When black box testing a software the only way of performing tests is by means of the user interface provided by the software. If the software uses a database, the quality engineer can compare before and after values in the DB. But most of the times the only way of knowing whether the software does what was expected is by looking at the user interface again. Does the report show the correct values? Got the selected element highlighted? Does it print and the output looks as expected?

That requires a lot of manual work and is per se error prone. A smart quality engineer wants to automate tests to create a number of baseline tests. When he gets a new version of the software he's testing, he simply runs his test suite and can concentrate on what's new instead of manually testing the same stuff over and over again. There is a number of tools - commercial and open-source - available for programmatically drive the UI of Windows, Java and web applications. Quite famous is the open-source tool Selenium for web applications. On OpenQA, a place for open-source QA tools, you can find as well tools to automate Java Swing or Windows GUI testing. For more tools, including commercial ones, look here.

My personal opinion about black box testing is that it appears to be a good way to control the quality of a finished software product and provides information to decide whether it is safe to ship it. But it is extremely important to keep in mind that this kind of testing is based on the user interface and you still might miss hundreds of defects just because you don't execute the software in the way that makes them show up. Astonishingly a lot of organizations view this kind of testing as the most important form of testing. Probably because most people perceive software as the user interface. So what they can see and touch that must be the thing they call software.

In the beginning of this post I was talking about quality assurance and quality control. So far we have identified a tool to control the quality of the user interface. I think it's safe to say it that way. What about controlling or even assuring the quality of the software's functions? To achieve that we need to look on the inside.

White box testing requires us to work with the source code, which allows us to perform tests on internal mechanisms instead of just the user interface. The keywords that come to mind now are TDD (Test-Driven Development), unit tests and tools like those from the JUnit family. This is going to be a complex topic, as it is not only about controlling quality by measuring things. To assure quality you have to start with the process from the very beginning, which is when you analyze the problem you want to solve with software. It touches the way you develop (your development process) and essentially is not any more a matter of the QA/QE team. I think it makes sense to cover white box testing in a second post.

An interesting and well written article to be forward to the C* in many organizations big and small: Getting Clueful: 7 Things CIOs Should Know About Agile Development

Really like that piece. It reminds me of a few things I witnessed not so long ago.

Marc Andreessen in Silicon Valley after a Microsoft/Yahoo merger: a contrarian view:

Build something of value -- something that people want, and something that will be profitable at the appropriate point -- and the world is yours.

Successful companies -- companies that have built something of value -- have many options. They can stay private and throw off dividends. They can go public. They can get acquired by big companies who suddenly decide, hey, that looks really valuable, let's buy that. They can sell minority stakes to big investors or strategic partners at very high valuations. All options that are typically not open to the startup that started with the goal of getting bought and didn't build something of independent value.

Or, reduced to a phrase: the best way to get bought is to not be for sale.

Update:

When I posted Thoughts on the ideal project team a few days ago I received a comment to which I'd like to respond in more detail here. I really appreciate that kind of comment. Who ever is xcosyns: Thank you!

Commenter xcosyns in Thoughts on the ideal project team at my blog:
Often you will need someone handling the infrastructure, uat, itt should match the real production environment. And from experience this is not always the case, and not that easy to achieve when multiple applications have to interact, one way or the other, together? And we are not yet talking about backups, db replication, san drives, authentification systems, load balancing…

Development infrastructure should be as simply as it makes sense. And it's the developers who should have control over it. For example in our case mostly I do that and when you have some experience it's not that hard to do. I encourage my fellow team members to not limit themselves to the "code monkey" role, but learn how to administer their own systems and team systems. My belief is that every software developer needs to be able to do sysadmin duties - at the very least for those systems he works with on a daily basis.

By that I don't mean to say that SAN drives and load balancing in a production environment should be maintained by the developers of a system. That's the job of an operations group. But developers who can do this for their little development environment have a much better understanding of these things and know what to do and what not to do when creating an application for such an environment.

Also, developers can really benefit of a DB expert, often java developers really sucks when it comes to db performance, and most developers are rarely aware about what their db can really do. A good DB expert can speed up legacy applications with no or minor changes.

You are right. Often Java developers really suck when it comes to DB performance. Why is that? Probably because their education is too shallow. One thing is to merely understand the language and a few common libraries/frameworks. And another thing is to understand software development in a holistic way. That's the result of experience and natural curiosity. For example in my own case I have been forced to learn Unix system administration, networking (up to managing a larger network for an ISP I co-founded; think of the whole zoo of routers, access concentrators and the various protocols such as BGP, OSPF, etc.) plus several high and low level programming languages including all the, literally, zillions of libraries and frameworks for those. It took a while, but I consider this the difference between an apprentice and a master. The ideal project team practicing agile development should be comprised of masters. There is a room for apprentices. You can easily assign to each master an apprentice and delegate some tasks to those.

Some documentation should be written, the business analysts can do the functional part, developers or the architect can do the technical part. But when the project/projects/teams scales up it becomes a time-consuming task and it can partially be delegated.

Agreed. If you have a need for more extensive documentation, then you certainly can add a tech writer. I see one question that remains. If you make the tech writer part of the team and follow the very good practice of "done done" for your user stories, that would mean that a story is only "done done" when the documention related to the story is completed as well. In this case you add another constraint besides the programmers. Probably it depends on the type of application and the target market whether that makes sense.

Also someone needs to follow up the development process, a project manager or a team leader. Someone that can prioritize the tasks in function of the business needs and business gains, someone that has a global view of the project and the company. And managing the budget, logistics, recruiting, etc…

For that there is the role of the Product Owner. His job is to prioritize stories based on business value. The customer is the only one who can really know the business value of stories so he should be the only one providing that indicator. Budgeting, logistics and recruiting are unrelated to the work the team is supposed to perform. You form the team before you get started - obviously ;-) - and then that's your team. I would not move people in and out of a team, because that negatively affects the accumulated domain knowledge and slows down.

The original post:

---

As I'm preparing some material for an upcoming event, I thought I could as well share this little piece here on the blog. Basically it is about the ideal software development team, the roles people play and the skills each team member needs to possess to be able to make meaningful contributions.

Customer

The customer is not that company or that guy who pays. The customer actually drives the project and is part of the team. It is important to engage the customer in a constant dialog or at the very least to give him a means to collaborate and respond to questions easily.

Programmer

Programmers write the code and are supported by Testing Programmers. The programmers are the constraint on the team. Their number needs to grow in order to be able to handle a larger project. I share the common opinion that when you add two programmers, you should add one testing programmer as well. It seems to be a good idea to add programmers in pairs, because that allows them to pair on difficult tasks.

Skills required: Expert knowledge in the choosen programming language, tools and other technologies used for the project.

Testing Programmer

The Testing Programmer is just an ordinary programmer, but in that role he looks out of test coverage, performs manual and automated tests of the integrated software system and checks for completenes of the implemented solution based on user stories and in cooperation with the Business Analyst acting as Product Owner.

Skills required: Expert knowledge in the choosen programming language, tools and other technologies used for the project plus strong testing skills and expert knowledge in test automation.

Information Architect

Almost every system has some kind of user interface. The Information Architect designs the user interface, creates wireframe models for communication purposes, works with the Business Analytics and the Customer and helps the programmers when they implemented the user interface. He creates graphical elements for the user interface.

Skills required: feeling for good user interface design, graphic design, communication abilities

Business Analyst

The Business Analytics acts as Scrum Product Owner for the team and maintains a constant dialog with the customer. His job is to understand the customer's goal and expectations for the project. He collaborates with the customer to create user stories.

Skills required: Strong analytical skills and ability to gain expert knowledge in the customer's business in short time. Very good communication skills oral and in writing. Expert in writing user stories.

An interesting find:

Mary Poppendieck in Another perspective on Command and Control:

Take a look at this:

http://www.carlisle.army.mil/pksoi/StaticDocuments/Army/fm6.pdf

Scroll to page 30 (1-15) and look at Figure 1.4.  This shows that there are two concepts of command and control:  Mission command and Detailed command.  Mission command is what we want, Detailed command is what doesn’t work.  Both are considered different techniques of command and control.  One kind of command and control is about telling people what to do, and the other kind of command and control is about organizing the environment and the people so that the mission is clear and effectively accomplished. 

I think using the word “command and control” as a pejorative is too simplistic, as is assuming that self-organizing teams is the one antidote to solve all problems.