Is your Engineering Development project stalled in QA? Have expectations throughout your organization been lowered to the point where extremely long, ponderous QA cycles are planned for and expected? This could be the result of an Involuntary Prototyping process that can become a trap that is expensive, not only from a product time line/opportunity cost basis, but also in employee morale and resource turnover.
Projects in general are date-driven, that is the business goals are to deploy something in a specified time-frame and with necessary features with reasonable quality. Planning around this is difficult but not impossible — often QA takes a back seat to development with respect to time-weight concerns, causing any delays in development to be thrown on the backs of QA teams, demanding that they complete the same amount of testing with less time. As development is delayed, the reluctance to accurately modify the schedule to a reasonable period decreases — stakeholders become more and more optimistic with their times, and consequently, any shock to the system will result in blown schedules and scrambling to “make the date”. This scrambling can bring on Involuntary Prototyping — when this happens, be prepared for a bear to eat your schedule and poop it off a cliff.
The Role of QA
The role of Quality Assurance (QA) in an enterprise is to use a planned and systematic approach to the evaluation of the quality of software product/applications, and their adherence to standards, processes, and procedures. QA warrants that standards and procedures are established and followed throughout a product/application life cycle.
Software development and control processes must include highly defined and published approval points, where a QA evaluation of the product may be accomplished in line with applicable standards. This involves collaboration with Development Groups through all stages of a product’s life cycle. Acceptance Testing provides a quantifiable gate for formally moving a product/application from the Development Phase to the QA Phase.
QA predicts their expected testing time with the assumption that they will get a Functionally Complete build from development that can pass a full Acceptance Tests (generally a build with all Designed Functionality included with zero or very few blocked areas of code). As projects get delayed because builds are not ready, resources become less and less available, and additional, serialized projects either have to wait or be taken on by smaller, ad hoc teams and offshore resources. Keeping offshore resources aligned with the onsite project teams requires complete documentation, plans and tight management. With at-risk projects, this is almost never the case.
Increased Project Risk Results in Process Failure
QA takes all projects at face value and goes to work. At-risk, large projects often force QA to take builds before they are complete in order to give them a fighting chance to make their release dates. These functionally incomplete projects soon become overwhelmed, and anti-processes form; reulting project delays become inevitable.
By testing for designed features and issuing, tracking and closing bugs that should have been caught before QA actual, mission-critical problems that must be found are deferred at best or just neglected, as the time for deployment draws near and the Product/Development teams’ launch pressures increase.
Delays in the QA process from a product with more-than-expected issues and compressed dates in guarantee a lapse into an “in for a penny, in for a pound” mentality. In the end the project will face delay, and all projects serialized behind it will be correspondingly backlogged in dramatic fashion, creating a potentially exponential expansion in delays as projects, dependencies and development resources are repurposed in a haphazard manner.
This inefficiency rolls up quickly. Initial show-stopping bugs trigger new builds. Builds are often weekly, causing resources to cycle between fully utilized to dormant depending upon their roles in the process. Risks also increase with respect to changes in configuration and regression.
Prototyping and its Pitfalls
If an incomplete product is sent through a Formal QA Testing Cycle, the result will be continuous builds, large numbers of bugs and issues, and further development/test cycles before a Release Candidate may be obtained. This process is known as “Prototyping”.
“A prototype is an easily modified and extensible model (representation, simulation or demonstration) of a planned software system, likely including its interface and input/output functionality. It is not a production-ready, fully-functional product ready for QA testing and deployment”
While prototyping is excellent for functional gap analysis and continuous improvement of an application under development, inefficiencies exist when a centralized QA team must test products from many different channels with limited resources. These resources will become overwhelmed when shocks to the system occur. These system shocks are identified as:
• Loss of resources or rapid turnover
• Increased production from development teams
• Increase in the number of products to deploy
• Geographical and cultural issues from off shoring
• Political issues and pet projects disrupting the schedule
As these shocks manifest themselves singularly, management must move to allow for scheduling issues, level resources and argue against artificial schedule changes. Delays are inevitable as resources are shuffled, and the output of the QA Team will degrade.
If these shocks come in as a group, management may become overwhelmed at the number of changes needed to be made and QA may have to place projects on “hold”, create an ad hoc triage process to test applications that may not be in alignment with business goals, and possibly lose all ability to test products for indefinite periods of time. Multiple shocks will also affect team morale and possibly create attrition issues and collisions of will between them and various Development/Business groups.
Prototyping most large organization is not deliberate. When a product is released to QA, the expectation of the Development/Business group is that Acceptance Candidacy has occurred and the product is to be tested and sent to production. When Designed Functionality is not met and major roadblocks are found, the product devolves into an Involuntary Prototype and the peculiar, often repeated and destructive dance of negotiation, multiple builds, huge bug backlogs and long delays commence.
Figure 1 — The Involuntary Prototype anti-process showing relaxed QE Acceptance standards and the cyclical nature of bug fixes and testing.
A Demonstrably Complete Product must be the Standard for Acceptance
To prevent Involuntary Prototyping during an application lifecycle, all products entering the QE environment must meet a minimum acceptable standard for initial testing:
• All Designed Functionality for an expected release must be met in an environment that is identical to, or certifiably mimics, the environment that QE will test the application in
• All Use Cases for the application must be documented, published to, and completely understood by QE stakeholders before initial Acceptance Testing.
• QA Environments must be understood, deployable and created.
• Product Managers and Development Leads must certify in writing that the product/application meets all Designed Functionality, with a checklist of all Use Cases tested.
• The expectations surrounding the schedule for QA testing and later deployment phases.
• All documentation, in a coherent and meaningful format, must be published to the QA group in advance of any Acceptance Testing so that QA may formulate its test plans.
This Acceptance Candidate, as defined above, is a functionally complete, verifiable, documented and demonstrable build that represents what the development group certifies as a potential Release Candidate.
QA must Reject Products that do not meet their Acceptance Threshold
QA must perform a full Acceptance Test before a Formal QA Testing Cycle may begin. It should be institutionalized throughout the entire organization that the mission of QA is to maintain corporate standards, and in a testing environment concentrate wholly on identifying problems with edge cases and border conditions, performance and difficult to find, non-obvious bugs.
A professional QA group is NOT in the business of identifying and logging Designed Functionality issues and sloppy development such as spelling errors, layout mistakes, non-functioning log-ins, network and connectivity issues. The responsibility for identifying and fixing the aforementioned issues must lie solely and completely with the Product/Development group that is responsible for the engineering of said product/application. Any product/application submitted for QA testing should be regarded as complete and demonstrably tested from the aspect of all Functional Design attributes.
Current development processes/groups may have various and disparate methods of engaging QA, and also have widely different views with respect to the role of QA within an organization. The engagement process and role QA changes in the an organization’s Development Doctrine and Executive Championing. The changes to the process should allow for most, if not all organizations to continue using successful, ingrained strategies, only needing to add QA touch points and additional demonstrations to ensure that all product/application stakeholders have met their responsibilities for delivering an application that is Functionally Complete and passing the required Acceptance Candidate Test.
The following items would be the expected minimum requirements for a product/application to be accepted for a full, Formal QA Testing Cycle.
• The Acceptance Threshold must be determined at project kick off and revisited before delivery to QA
• Full documentation of Use Cases and Deployment Diagrams must be available to QA resources for testing and configuration.
• Before Acceptance Testing begins, the Product Manager will verify the functionality of the product is complete through a Acceptance Candidate Test.
• Products not meeting the Acceptance Test will be rejected from QA and possibly lose their “window”, at the very least they will need to be re-entered into the testing queue.
• Disputes over Acceptance Testing, Rejecting and Product Acceptance will be mediated by the senior staff for the product and QA departments, with minutes and notes from the meetings sent to the VP-level staff for review.
These bullets should be easily integrated within most current development processes, and indeed many groups meet most, if not all, of the above responsibilities. The object is to formalize the process into the “DNA” of the Product/Development groups and give the QA team a “gate” and set of rules that allow for responsible and sensible pushback that allows for the entire organization to benefit, and reduce the bottleneck around QA entry and exit points.
The Complete Process as Integrated within the Development Cycle
Involvement with QA should begin as early as possible, and frequent touchbacks are a method of guaranteeing success.
As the Development Process is kicked off, documentation and expectations are transmitted to the QA group to communicate an understanding of the product/application and set a baseline for acceptance criteria. This documentation should include, but is not limited to, all design documentation, Use Cases, Product Requirements, and schedule expectations. As development continues, this document repository should be updated and extended as needs arise.
During the development phases, Product Managers and development stakeholders should get together on a pre-determined basis with QA resources as optional observers to demonstrate the features that have been added to the product and self-police any limitations or errors that have arisen at that point. Issues around any shortcomings in Designed Functionality should be created and addressed before the final Acceptance Candidate Test.
Once the development phase has reached conclusion, the Acceptance Candidate Test should be conducted, ensuring that all Designed Functionality, configuration, layout, spelling and craftsmanship are complete and up to industry standards. The Acceptance Candidate Test should guarantee that the product/application would pass the QA Acceptance Test either through the publishing of results or by direct observation from a responsible QA stakeholder. At this point the Product Manager and Development Lead verifies and guarantees in writing that Designed Functionality is complete.
Upon passing the Acceptance Candidate Test, the product is Ready for the QA Acceptance test. If the Acceptance Candidate Test has been properly conducted and communicated with QA, this test may be nothing more than a required formality. It also will allow QA to familiarize itself with the product/application, receive any training needed to test it correctly, finalize test plans and communicate any forward-looking concerns. If for any reason the QA Acceptance test fails, QA will reject the application wholly and the product will be sent back to development accompanied with a full report as to why the rejection occurred. This report shall be communicated to all stakeholders, managers and interested executives within the Product/Development group.
Figure 2 represents the integration of Development and QA processes to prevent Involuntary Prototyping and ensure that all Designed Functionality has been met before a Formal QA Testing Cycle begins.
Figure 2 — QE & Development Process Integration to prevent Involuntary Prototyping
Once the Acceptance Test passes, the Formal QA Testing Cycle begins. This cycle is similar, if not identical to, what the current cycle entails, except the burden of finding and logging Designed Functionality problems will be virtually non-existent, and issues that will cause full cycles to stall should occur with very low frequencies. Bugs will be logged and if required, new builds and testing cycles will occur as needed. Once this Testing Cycle ends, the product will be cleared for deployment.