A Basic Test
Engineering Primer

Version 0.3

Copyright © 2005-2009 Greg Gonnerman.
All Rights Reserved.



Table of Contents

   1. Introduction
   2. Basic Concepts
   3. Writing Test Procedures
   4. Problem Reports
   5. Managing Your Test Effort
   6. Staffing
   7. Quality Standards
   8. Lies, Damned Lies, and Automated Testing Tools
   9. Closing Comments

Document Revisions

Version
 Date
 Description
0.1
 2005/08/07 Rough Draft
0.2 
 2005/11/13
 Minor corrections.
 0.3 2009/05/20
 Format changes.

1. Introduction

This basic test primer is written for the benefit of the engineer, developer, or manager who isn't too familiar with software or system test techniques but has been tasked with these responsibilities. This document assumes that the reader is at least somewhat familiar with basic hardware and software development practices, but knows little about test engineering principles. Other aspects of test engineering, including testing in a manufacturing environment, environmental testing, emissions (FCC) testing, etc., are not covered in this basic primer.

It's important to start off by establishing what testing is and is not. It is not an effort to test every path through the code or every facet of the hardware design. It simply can't be either of those things. Most modern applications and devices are far too complex to allow for that level of testing. Even if your test team had the technical prowess necessary to identify every possible test scenario, you almost certainly wouldn't have enough people or time to carry out that much testing. Instead, testing must be an effort in risk management and risk mitigation. At every turn, you need to consider the potential of an error occurring and the possible impact such an error might have. These concepts are covered in detail later in this document.

Another important aspect of system or software testing are the expectations of your management and of the customer. Not only is it important to meet their expectations, but it's also important to manage their expectations. Delivery dates, budget, and potential latent defects are important to everyone involved, and to a large extent, the success of your test effort will be measured by how well you've met their expectations.  Effective communication up front can go a long way towards managing these expectations and improving the perception of your success.

Finally, remember your role as the test engineer. It isn't your place to hold up a release if testing isn't complete or too many bugs remain. Ultimately that's a management decision. If you've provided your management with the information they need to make such a decision, including known and perceived risks, then you need to defer to their judgment. Right or wrong in the decision they make, it is their decision to make.

2. Basic Concepts

Testing is normally divided into Unit, Integration, and System test phases. Unit testing is concerned with the individual software module or feature. This phase of testing is typically performed by the software developer on his or her code before it is integrated into the complete application being developed or modified. This is followed by integration testing. In some cases, this is a phase of testing, but in others it may be more aptly described as the activity of assembling or integrating various hardware and software modules, without necessarily having a formal test component. Integration testing is then followed by system test, which is normally the final phase of in-house testing.

Before we discuss in detail the various phases of testing, let's first define white box and black box testing, as these terms will be important in differentiating test phases below. White box testing relies on the tester having an intimate knowledge of the internals of the item under test. Based on this "inside" knowledge of the software or hardware, the tester will develop test cases that deliberately test many paths through the code or hidden facets of the hardware design. Taking a white box approach will limit the focus of the tester on how the product under test is really used, but testing will benefit from the tester's insight into possible problem areas that may not be visible to others. Black box testing, as you might imagine, relies on the tester having no idea what is going on in the code or hardware under test. The item under test is figuratively a "black box", i.e., you can't see what's going on inside. Taking this approach the tester will likely find defects not exposed during white box testing because many defects are only exposed when unexpected inputs are used, or unusual interactions between subsystems are encountered. White box and black box testing are complimentary and both are required to thoroughly shake out the bugs in the item under test.

Unit testing is normally the responsibility of the developer writing the code, and benefits from their intimate knowledge of the module under test. Occasionally though, a test engineer may be assigned this task. Unit testing should be performed using a white box approach. Rarely is the term "unit test" applied in hardware development.

Integration may either refer to a specific test phase or an activity. As a test phase, integration testing is performed immediately after all software and hardware components are brought together (integrated) for the first time. Ideally, the test procedures for this phase trace to a separate and distinct group of integration requirements. Be aware though, that the term "integration" is sometimes applied to the activity of bringing all of the components together. When used in this way, formal testing may not be part of the equation;  any testing in this type of process step is ad hoc in nature, and only the most obvious of defects will be found. A short test suite executed during the integration phase may be called smoke testing.

The final in-house phase of testing is normally system test. This phase of testing is entirely black box in nature, and all test procedures are derived from high level requirements, and perhaps from a basic understanding of how the end user is likely to interact with the item under test.

All phases of testing may incorporate both regression and new feature testing. Today's regression testing is simply yesterday's new feature testing. Test cases written for a new feature for the current release will become regression tests when they are executed again in advance of the next release.

An almost universally accepted type of testing that may not be driven by requirements is stress testing. This form of testing involves pushing the item under test beyond its limits. Stress testing of a distributed software application with a centralized database might involve recruiting users or technical staff from your organization to exercise the application from a number of nodes at a faster rate than would normally be expected. One might also use automated test tools in this case, which could simulate numerous users interacting with the system.  Stress testing of a combined hardware and software release, like for a computer and the associated operating system, might incorporate a variety of strategies that exercise hardware components, device drivers, and user level application code. Load testing is quite similar to stress testing, except the goal is to push the application or device under test to its limits, rather than beyond its limits.

A particular phase of testing might be executed twice. First as a dry-run to identify problems with the test procedures and for early identification of defects in the item under test, and later as formal test run. Dry-run testing is simply a dress rehearsal. Formal testing is what really counts, and it is sometimes called run-for-record testing.

Some organizations supplement the three primary phases of testing with field testing, which may be called field trials or beta testing. Also, if you are delivering a product for a specific customer, they may choose to perform acceptance testing.


3. Writing Test Procedures

Let's start out by differentiating test plans and test procedures. A test plan should provide the high level overview of your test process, methodology, resources, staffing needs, etc., while the test procedures document the specific test scenarios defined for a test effort. These documents may share some of the same introductory wording, but they are separate and distinct documents. With that said, in some organizations, these documents are one in the same. In this case the combined document incorporates the contents for both. Generally, a combined plan and procedure document works well for a smaller test effort with a limited number of requirements and just one or two involved test engineers. Larger test efforts benefit from separate plan and procedure documents; in fact, there may be multiple procedure documents for the various features, subsystems, or subassemblies.

The place to start with writing good test procedures is to identify the requirements. With any luck, your organization has one or more system engineers who have the responsibility for generating requirements. If you're really lucky, the system engineers have also provided you with use cases. The requirements should describe what the device or application should and should not do. They will be phrased something like "The GUI shall prompt the user for a login and password" or "The GUI shall not accept any user input other than login and password until a correct login and password have been entered." Use cases, if provided, give the developer, engineer, or tester greater detail than that defined by the requirements, and they generally describe specific ways in which the device or application might be used and might respond to the user. Use cases for the requirements above might also describe the look and feel of the GUI login screen, specific error messages for incorrect logins, or any other details not included in the requirements. If you have detailed requirements, then generating test cases is not difficult. If you have use cases, then generating test cases becomes a walk in the park. Unfortunately, not all organizations see the need for requirements. Many projects plunge ahead with development and eventually with testing without having completed this necessary first step. As the test engineer, you are probably not in a position to change this.  The best strategy in this case is to identify the requirements yourself, and move forward from there. Now you may be thinking, if there are no documented requirements, how exactly does one identify them?  Well, just because they aren't documented doesn't mean they don't exist. The hardware engineer or software developer obviously had some idea of what was needed, or else you wouldn't even have anything to test. They wouldn't have been directed to work on this item unless someone in management or marketing had an idea of what was needed. If there are no documented requirements, start with any documentation relevant for the item being developed and tested. This may include marketing documentation, engineering specifications, or possibly a statement-of-work (SOW) if this is being developed for a specific customer. From those sources, work up a list of requirements as you understand them, phrased like the earlier examples. Review these requirements with the various involved parties and try to get consensus that your list is complete and accurate. This may take some time but it is time well spent. You may discover that there are differences of opinion concerning what is really required of this device or application. It's much better to sort this out now than after it's been released to the customer.

Each test case you write should reference the requirement(s) that it covers. This is commonly referred to as requirements traceability.  Tracing test cases to requirements is normally accomplished by including the assigned number for each requirement directly in the associated test case. Some test documentation tools facilitate this by providing the ability to link from within a document directly to requirements in the database. It really doesn't matter how you handle requirements traceability. Doing this manually or using the latest tools accomplishes the same important task. It provides the assurance that all requirements have been covered in documented test cases. Upon completion of testing, the requirements traceability should be reviewed one more time. Regardless of how thoroughly your test procedures and requirements were reviewed, something was almost certainly missed. Perhaps a test case wasn't executed because a last minute design change rendered it incorrect or because a test tool didn't become available, or maybe a requirement was added after testing had begun. No matter what the cause, it's important to find these discrepancies and resolve them. In the final analysis, the completed, successful test results must show that all requirements were properly implemented. One more thing to consider is whether a particular test case fully covers all aspects of a requirement, or if it just covers one part of it. It's best to discuss this during the peer review of the procedures, but preliminary discussions can start during the review of the requirements.

One important distinction to make is the difference between requirements validation and requirements verification. Validation concerns the validity of the requirement; that is, was it really a good requirement to begin with?  Is this requirement consistent with the needs of the end user, does it fit within the product road map, and is this something we really want our product to do?  Requirements verification, however, makes no judgment on these matters. It is simply confirmation that the product performs as stated in the requirements. The former is not normally a function of software or system test, but the latter most certainly is. If your organization has quality processes in place, and requirements are peer reviewed before you start writing test cases for them, there should be no question about the validity of the requirements. The questions posed during any sort of validation step are already answered, and the requirements are understood to be entirely valid. Be aware however, that some references (notably IEEE Std 1012-1998) use these terms differently; they instead use these terms to differentiate between test methods, with verification being used to refer to most types of testing, and validation covering inspection, analysis, and fault injection techniques. Still others refer to the up-front work of inspections, walkthroughs, and reviews as all being a part of the verification step, while the actual hands-on testing is the validation step. In this document and others from the same author, these terms will only be used as described above, in the context of requirements.

The primary focus of any software or system test effort must be on requirements verification; first with positive or sunny day scenarios. Sunny day scenarios are those that assume that the product under test will work as designed. Take for example the login prompt requirements described earlier. A sunny day scenario would have the tester confirm the presence of the login prompt under normal conditions, and then log in with a correct login and password. Basically, for each requirement to be verified, determine the expected behavior of the product under test and establish test scenarios that confirm this behavior, assuming all aspects of the feature have been implemented correctly.

After you've covered sunny day scenarios for the documented or understood requirements, you should next consider boundary and error conditions, which we will call rainy day scenarios. Boundary condition refers to those inputs or outputs around an established minimum, maximum, or other value of interest. As an example, in advance of the Y2K (Year 2000) transition, a lot of testing centered on the transition from December 31, 1999 to January 1, 2000. This boundary was obviously the primary focus of Y2K testing, but dates earlier and later were also of interest. One of those dates was February 29, 2000. Not everyone knew that the year 2000 would be a leap year. Years divisible by four are leap years, except those that divisible by one hundred. However, years divisible by four hundred are the exception to this rule, that is, they are leap years. Since there was some confusion about this exception, testing of the transition into and out of this leap day was considered important. While Y2K has come and gone, you should consider similar boundary conditions when defining test cases for the hardware or software you are testing. Another example of a boundary that deserves testing would be an input that is supposed to accept values up to 100. In this case, 99, 100, and 101 would be good input values to test. You should also consider possible error conditions. As an example, if you are testing an application that is required to accept numeric input, make sure to define test cases for non numeric input. If  your application is required to display an error for negative input, make sure to test negative inputs. Boundary and error conditions are not always defined in the requirements, but they are always important to test. Make sure you've covered all of these conditions. While it may not be possible to generate and execute test cases for every path through the code or every facet of the hardware design, it's normally feasible to test the important rainy day scenarios. During the review of your test procedures, solicit input from your peers on scenarios you might have missed. They might have some insight into problematic areas that you aren't aware of.

As you develop test procedures, don't be afraid to trust your intuition. Consider ways in which the product might "break" or exhibit some type of undesirable behavior. Think about your past experience with this product or ones like it. You'll likely come up with additional scenarios not directly related to any documented requirements. In this case, it's not necessary to worry about requirements traceability. Be aware, however, that you will be asked during the review why your test case doesn't trace to a requirement. Just be prepared for this, and if you really believe it's a good test, be prepared to defend it through the review cycle.

When writing test procedures, always consider the audience. That is, for whom are you writing these procedures?  Although typically you will be executing test cases that you've written the first time around, at some later date they will likely become part of the regression tests. Unless you plan on staying around forever to run these tests, you need to consider them from the perspective of someone who isn't as intimately familiar with them. It's best to assume that these tests will be executed by someone with your general experience and knowledge, but without specific knowledge of this area of functionality. There are several disadvantages to providing too much detail. Eventually these procedures will need to be revised to accommodate changes to the requirements. Lengthy, wordy, detailed test procedures are much more difficult to revise and support than succinct, well worded ones with the appropriate level of detail. Also, detailed test cases can become laborious to follow for someone who doesn't need all that detail. This may lead the test engineer to skim through the test procedures looking for what really needs to be done, and this in turn can lead to missed test steps. Too little detail in test procedures can also lead to problems. Primarily with extra time and effort being spent by the test engineer trying to figure out what needs to be done. This too can lead to missed items in the test procedures when vague test procedures don't convey the necessary information. If possible, reference other documents in the procedures with detailed information concerning test setups, software and hardware configurations, etc., if such reference documents exist.

There are many ways in which your efforts could be diverted into unproductive testing. First, not everyone knows how to write good test cases. If you're working on a brand new project, with no legacy test procedures, then you're lucky. You don't have to worry about poorly written legacy test cases that accomplish nothing. Also, some test cases might have been completely necessary for a prior release, perhaps when the feature being tested was first introduced, but they might not be necessary now. Generally a carefully selected subset of test cases can be integrated into your suite of regression test cases and provide adequate test coverage. Make sure that the tests you are running are adding value to your overall test effort. If you have test cases that don't appear to be necessary for your current test effort, work through your organization's processes to remove them from the list.


4. Problem Reports

One of the most important tasks you will perform as a test engineer is reporting defects. This is normally done through a failure reporting system on your organization's intranet. These failure reports may be known as Change Requests (CRs), System Problem Reports (SPRs), Technical Action Requests (TARs), or by a number of other names. In this document, we'll call them problem reports.

Regardless of what your organization calls them, these problem reports are used to document the problem (obviously), the test configuration, software and hardware under test, and just about anything else that's relevant. Take these problem reports seriously! If you are in a larger organization, many will only know you through these reports. You will be judged by how thorough, complete, and accurate you are in completing them. An occasional unreproducible defect or "user error" is to be expected, but if you take the developers or design engineers on too many fruitless searches for nonexistent defects you will soon lose all credibility. You need to document completely everything they might need to know when trying to reproduce the problem and isolate its root cause. Take a moment before writing the problem report.  Make sure that your test configuration was correct. If time permits, attempt to reproduce the scenario that led up to the failure or error. Document fully all steps that were necessary to get into this failure condition. It's important though to follow management guidance on how much time to spend confirming bugs before your write problem reports for them. If the developers or engineers are sitting idle and the test team is in "crunch mode", they may direct you to just quickly write problem reports for any perceived problem and let others sort out whether they are real problems. Alternatively, if the quality metrics for the organization are adversely affected by problem reports, you may be directed to thoroughly flesh out a problem before reporting it, perhaps even going as far as isolating the root cause with the help of others.

It's important to understand that the developers or design engineers have put a great deal of effort into the software or device under test. So much so that they might be upset with you (and in denial) when you report a problem. For them, this application or device might be a bit like their child. When you report a problem it's like you're insulting their child. This makes it especially important that you are thorough and complete when you document the problem, and it's also important that you remain calm and dispassionate when dealing with them on this issue.

If you find out that a problem report you wrote was closed without being fixed, either as unreproducible or unimportant, stay cool. You did your job. You found the problem and reported it.  If you have the time and feel that a problem report closed as unreproducible was a real problem, by all means, reconfirm it. Invite the necessary people back to your test area to witness the problem, but don't get emotionally involved in the situation. If the reported problem caused a test failure that left a requirement untested or worse yet failed, leave it as such. In your test report you'll need to document this untested or failed requirement, reference the problem report and leave it at that. Be prepared, however, to justify this. There will almost certainly be questions when your test report is distributed for review,

With any luck the problem you identified will be fixed and the problem report will come back to you for verification of the fix. If you documented the problem, then you should know what to do to verify that it's now fixed. If you are verifying a fix for a problem reported by someone else, make sure you understand what they did to get into the failing state to begin with. If they were thorough and complete when writing the problem report, then you should have the information you need. But if you have questions, ask!. If you close out the problem report and the problem isn't really fixed, it could completely escape detection and end up in the released product for your customers to find.

In a perfect world, each requirement would have one test case and each problem report would be directly associated with one test case. In case you didn't notice, we don't live in a perfect world. A requirement may have one corresponding test case or many. A problem report may or may not be directly associated with a test case. You might encounter people that have a problem with this. They might demand to know which requirement is affected by the problem you found, or which test case has a failed status because of it. Understand that you will find yourself in a position like this and try to be patient. Take the time to explain the requirements and test cases for the feature under test. At the heart of the matter is the fact that not every conceivable aspect of the design is documented in the requirements, thus, not ever problem will be directly associated with a particular requirement. Since test cases are written for verification of this incomplete set of requirements, not every problem you find with be closely associated with a documented test scenario. This issue might also crop up when it comes to verifying a problem fix. You might encounter the expectation that a previously documented test procedure can be used for verification of every problem fix. That certainly isn't the case, and you'll need to explain this to the people involved.

As with every aspect of system and software testing, obsessive compulsive tendencies are generally a good thing. It's important to be thorough and document each problem fully. Also completely document what you did to verify the problem fix. Being thorough will help those who are tasked with fixing the problem, and help ensure that the problem really does get resolved and is not simply ignored.


5. Managing Your Test Effort      

If you've been tasked with managing a test effort, one of the most important areas you need to consider are the expectations your management and those of the customer. They each have their own expectations for your test effort. Your management is concerned with the budget, which can be influenced by staffing levels and schedule. Your customer is concerned with features and quality . Both your management and the customer have an expectation about the release date for the product, and since your test effort is one of the last steps before the release, rightly or wrongly you may be held responsible for any slip in this date.

Throughout your test effort you need to be aware of these and other expectations they might have. You need to try to address them to the extent that you can, but you may also need to attempt to modify their expectations to match reality. Take for example the common situation of a schedule slip. Your team didn't receive the final software build until two weeks after the scheduled date. There are several things you can do; first, try pushing back for schedule relief for this lost time. If you aren't able to recoup the lost time, try adding staff. In a situation like this your only option will likely be to pull in people from other areas of the organization. Technical staff are preferred, but you should never have developers or design engineers testing features, subsystems, or assemblies that they helped create. Since they developed it and presumably performed unit testing, they aren't in a good position to find any remaining defects. In this example, it may be difficult or even impossible to modify the customer's expectation about the release date. However, in some specific circumstances this might be possible and is worth considering. Ultimately, you and your test team will be judged by how well you met or by how far you missed meeting the collective expectations of management and the customer.

Probably the most important tool for managing your test effort and communicating the current status is the test matrix, also known as a test case spreadsheet.  At a minimum, this spreadsheet lists all test cases to be executed, the software and hardware versions under test for each test case, the date each test was executed, the tester's initials, and finally, any comments or problem report numbers. Additionally, formulas may be created in the spreadsheet to display the current percent of test cases that have been completed, percent passed, percent failed, etc. You can also go a step further and use the data provided to generate graphs and charts to graphically display your current test status, but that's beyond the scope of this document.

The following table provides an example of what a test matrix might look like.

Test #
 Test Name
 HW Ver
 SW Ver
 Date Tested
 Results P/F
 Initials
 Comments and Problem Reports
1.2
 Start Up Test
 3.0
 0.2
 2005/5/30
 P
 GMG
1.3
 Login Test
 3.0
 0.3
 2005/6/1
 F
 JFK
 Can't login, PR 32153
1.4
 Data Entry





...
 ...
 ...
 ...
 ...
 ...
 ...
 ...
1.10 Halt Test









Percent Passed
 10%





Percent  Failed
 10%





Percent Complete
 20%


The test matrix should be kept in a shared directory on your organization's intranet. All members of your team should enter their daily test results at the close of business each day. Ideally, everyone with an interest in this test effort should know about this spreadsheet and have access to it so they can review it anytime they want. The test matrix can be an invaluable tool for recording results for each test, tracking progress, and communicating current status to the powers-that-be. Make sure that everyone understands how to use it and the importance of keeping it updated.
      

6. Staffing for your Test Effort

If you are involved in staffing a test effort, there are several important things to consider. First, the type of person who makes a good test engineer is a little different than one who makes a good developer or hardware engineer. The best test engineers are borderline obsessive-compulsive. They pay close attention to detail and are very particular in their test environment and configuration. Problems can be missed when procedures aren't followed carefully, or the test environment isn't exactly what it needs to be. Educational credentials are normally an important consideration when hiring developers or engineering staff, but for a test engineer, it's better to focus on the general technical knowledge the individual possesses. Someone you might consider a "jack of all trades" can make a good test engineer because they are comfortable working with a variety of technologies and can learn new ones more quickly.

There are several ways to gauge how many test engineers are needed for your test team. One rule of thumb some organizations use is to have one test engineer for every two developers. Sadly, this isn't a widely accepted rule and many organizations employ far fewer test engineers than developers. If you are revising a previously released product, and on the last run through testing you recorded the time required as is described in the section above, then you know exactly how many people are needed. Having recorded the time required for each test last time puts you in a good position to argue for the staffing level you need. You will still need to provide estimates for new test procedures, but this becomes a simple mathematical exercise based on your experience with the last test run. Consider the number of  new requirements, and the number of regression test cases compared to the number of regression requirements to determine roughly how many new test procedures will be required. Estimate the time required for these new, undefined test cases by looking at your last test run. You can either use an overall average for all test cases, or you can use the time requirements for similar features. Once you have an idea of how many new test cases will be needed, and how much time will be required to run these tests, then you need to consider how much time is scheduled to perform this testing. Almost certainly a release date has already been defined, and you have a soft commitment for when testing can start. Let's say that 600 hours will be required for regression testing, and your estimate for new features testing is 260 hours.  Let's further assume that there are six weeks set aside in the schedule for this test effort (860 hours / 6 weeks).  How many people do you need?  You might think that four people will be enough (ok, it actually works out to 3.58, but trying finding 0.58 of a person!). If you think four is enough than you must be assuming each person will be performing hands on testing for forty hours per week. Is that reasonable?  What about staff meetings, questions from developers about problem reports, and anger management sessions with human resources? These things all take time too. Some test managers assume that each test engineer will perform as little as 24 hours per week of actual testing. You are in the best position to estimate how many hours there are in a person-week in your organization, but never assume forty. You'll have your test staff there nights and weekends trying to make up for your planning mistake. Back to our calculation... if you assume there are 24 hours in the work week, then you'll need six test engineers.

Finally, after you've established what you are looking for in test engineers and the number of people that are required for your team, you need to consider the mix of people, skills, and experience levels. Generally, it's best to staff your team with equal numbers of people with various experience levels. One approach is to fill one third of your available positions with senior level test engineers having ten or more years of experience, one third with moderately experienced engineers, and finally, one third with those having little or no experience. For this last group, it's especially important to consider the personality traits described above. At first glance, it may seem desirable to fully staff your team with the most experienced test engineers you can find. While there may be advantages to this, there are some huge disadvantages. First, more experienced test engineers usually aren't very interested in doing the hands-on, manual testing that is almost always a huge part of a test effort. People with little or no experience usually don't mind doing this type of testing. Further, there are a number of right ways to accomplish the goal of finding defects and delivering a quality product. Two seasoned test engineers are likely to have slightly different approaches to accomplish these goals. Differences of opinion between experienced test engineers can quickly escalate into heated religious debates. An extra benefit offered by bringing people with less experience into your team is the fresh perspective they offer. They might be more likely in some circumstances to notice smaller issues that more experienced staff members have missed. Less experienced testers more literally interpret the test procedures they are executing, and because of this, they may notice a discrepancy between the procedures and the product that others have missed.

If the product under test is for a specific niche market, and you plan on staffing a larger (6+ person) test team, you should consider bringing into the team one or more people who have some experience as users in this market. A clinical information system intended for intensive care units, or an accounting application intended for real estate brokers might be to two good candidates. While usability testing isn't usually the focus or even the responsibility of integration or system test engineers, having some insight into how the product will be used can be beneficial in shaking out defects in the requirements and design. If the product under test is fairly generic, like a point-of-sale application or a photo editing utility, it shouldn't be necessary to bring product or domain experts into your team.

As you start to staff your test team, the very first test engineer you should add needs to be the most experienced you plan to hire. This individual will be in the best position to influence process decisions, review requirements for testability, and assist with additional staffing. Bringing an experienced test engineer in early can set the tone for the remainder of your test effort, and improve your chances of success.

7. Quality Standards

If you are involved with testing for very long you will almost certainly encounter (or be subject to) one of the many quality standards out there. The most widely known include ISO 9000, SEI, and the FDA's GMP (Good Manufacturing Processes), among others. Rather than describe each of these in detail, which would be beyond the scope of this document, let's review what they have in common. It won't take long for you to realize that each of these has a "bureaucratic" aspect that doesn't seem to contribute in any way to the quality of the product you are delivering. With that said, there are some aspects of each that can really improve quality.

One of the most important and common aspects is the idea of "Do what you say and say what you do."  In other words, your actions and processes should always be entirely consistent with what your documentation indicates.

Another important common directive is in the area of document control. Virtually every quality or standards body requires that you have in place means for controlling documentation. You need to have in place a formal peer review policy, and the means to ensure that the documents used for current work are themselves current and approved.

Eventually you may be audited by a quality auditor. These individuals may be directly associated with the standards or regulatory organization, or they may be with an independent auditing firm. In any case, there are some important considerations when dealing with them. First and foremost, be honest! Don't lie to them or stretch the truth. With that said, you shouldn't volunteer information either. Answer each question honestly and succinctly. Now is not the time to either cover up an ongoing process issue or air dirty laundry about a disagreement you've had with your management. Be honest, brief, and cordial.

If the product your team is developing is for a specific customer, and this work has been done at their expense, you may be audited by them. These audits are generally less stressful than the quality audits and you may be in a better position to guide your auditor through the process.  If any members of your team lack good social skills, you might want to keep them at a distance during this process. You should expect the customer's auditor to ask to review the test results, problem reports, requirements traceability, and any other relevant data. Honesty is the best policy. If you fully disclose all that was wrong and what your team did to resolve it, you are more likely to gain their confidence and respect. They will be less likely to press you hard on minor issues if you've gone out of your way to fully disclose everything to them. Just because you are being honest and showing them everything, this doesn't mean that you can't use a certain amount of "spin control". Sometimes the difference between a major issue and a minor one is how it's perceived.

8. Lies, Damned Lies, and Automated Testing Tools

With apologies to Mark Twain, Benjamin Disraeli, or whoever first made the quote I've modified for this section title, pitfalls associated with automated test tools are so important that they deserve mention in this basic primer. A detailed analysis and comparison of the popular tools will be provided in a follow-on document. For now it's important to understand what can go wrong when the test tool vendor manages to speak with your management. I've never actually witnessed one of these presentations, but they must be impressive. Otherwise clear headed and intelligent managers come away from them thinking that this automated test utility is almost magical. They seem to believe that all one needs to do is "press a button" and the automated tool tests all by itself. These comments and concerns apply primarily to GUI testing tools; other test utilities don't necessarily suffer from the same shortcomings.

The reality is that even with the best test tools, setting it up to do the testing can take a while. The tester either needs to record scenarios manually or program them in a scripting language. Either way, this can be a time consuming and laborious task, and if there are even minor changes in the application under test, you might have to do this all over again.

With that said, these GUI test tools can be invaluable in certain situations. If you are trying to reproduce a bug that has been reported in the field that only occurs after days of continuous operation, the time it takes to record a scenario and set it up to loop is time well spent.

9. Closing Comments

No matter what you are testing, be it hardware, an operating system, a GUI, a database, or a web based application, there's plenty you can do to find the bugs so that your customers don't have to.  However, no matter what you do, it's quite unlikely you'll ever find all of the defects. Be thorough, honest, and a little obsessive-compulsive and you'll do all right.

Copyright © 2005 by Greg Gonnerman. All Rights Reserved.