Acceptance Testing: A Complete Guide
What is Acceptance Testing?
Acceptance Testing is software testing in which a system is evaluated for its acceptability. The primary goal of this test is to determine whether the system complies with the business requirements and whether it is suitable for delivery.
Acceptance Testing Standard Definition:
It is formal testing based on user needs, requirements, and business processes to determine whether a system meets the acceptance criteria and allows users, customers, or other authorized entities to decide whether or not to accept the system.
Acceptance Testing is the final stage of software testing that occurs after System Testing and before the system is made available for use.
Read more: Different Types of Software Testing
Why are Acceptance Tests used?
Although system testing has been completed successfully, the customer has requested an acceptance test. The tests are repetitive here because they would have been covered in System testing.
So, why are customers conducting this Testing?
This is due to:
- To gain trust in the product that is about to be released to the market.
- To ensure that the product is functioning properly.
- Ensure that the product meets current market standards and is competitive with similarly priced products.
Types
This Testing comes in a variety of forms.
Among them are the following:
1) User Acceptance Testing (UAT)
UAT determines whether the product is working properly for the user. Specific requirements that end users frequently use are primarily chosen for testing purposes. End-User Testing is another term for this.
The term “User” refers to the end-users for whom the product/application is intended. Thus, Testing is done from their perspective and point of view.
2) Business Acceptance Testing (BAT)
This is to determine whether or not the product meets the business’s goals and objectives.
BAT primarily focuses on business benefits (finances), which are quite challenging due to changing market conditions/advanced technologies. The current implementation may need to change, resulting in additional budgets.
Even if the product meets the technical requirements, it may fail BAT for these reasons.
3) Contract Acceptance Testing (CAT)
This contract states that once the product goes live, the acceptance test must be performed within a certain time frame and pass all acceptance use cases.
The contract signed here is known as a Service Level Agreement (SLA), which specifies that payment will be made only if the Product services meet all requirements, indicating that the contract has been fulfilled.
This contract may be signed before the product goes live. In either case, a contract should be well defined regarding the testing period, testing areas, conditions on issues encountered later in the process, payments, and so on.
4) Testing for Regulations/Compliance Acceptance (RAT)
This is to determine whether the product violates the rules and regulations established by the country’s government in which it is released. This may be unintentional, but it will have a negative impact on the business.
RAT is typically required for developed products/applications intended for global release, as different countries/regions have different rules and regulations defined by their governing bodies.
Suppose any of the rules and regulations for any country are broken. In that case, that country or a specific region within that country will be unable to use the product and be considered a failure. Vendors of the Product will be held directly liable if the product is released despite a violation.
5) Testing for Operational Acceptance (OAT)
This is non-functional Testing to determine the product’s operational readiness. It primarily consists of Testing for recovery, compatibility, maintainability, technical support availability, reliability, fail-over, and localization, among other things.
Before releasing a product to production, OAT primarily ensures its stability.
6) Alpha Testing
This is to evaluate the product in the development/testing environment by a team of specialized alpha testers. In this case, the testers’ feedback and suggestions help to improve the product’s usability and to resolve certain bugs.
Testing takes place in a controlled environment.
7) Beta Testing/Field Testing
This is done to evaluate the product by exposing it to real end-users, also known as beta testers/beta users, in their environment. Users’ feedback is constantly gathered, and issues are resolved. This also aids in the enhancement/improvement of the product to provide a rich user experience.
Testing occurs in an uncontrolled environment, meaning a user has no restrictions on how the product is used.
All of these types have the same goal:
- Ensure that you gain/increase confidence in the product.
- Ascertain that the product is ready for use by actual users.
Read more: Unit Testing in Software Testing: Types, Tools & Best Practices
Who does Acceptance Testing?
Only members of the organization (who developed the product) perform Testing for the Alpha type. These individuals are not directly involved with the project (project managers/leads, developers, testers). Management, Sales, and Support teams are typically in charge of Testing and providing feedback.
Except for the Alpha type, all other types of acceptance are typically performed by different stakeholders. Customers, customers’ customers, and organization-specific testers (not always).
It is also good to involve Business Analysts and Subject Matter Expertise while performing this Testing based on its type.
Qualities of Acceptance Testers
Testers with the below qualities are qualified as Acceptance testers:
- Ability to think logically and analytically.
- Good domain knowledge.
- Capable of researching competing products in the market and analyzing them in the developed product.
- Having end-user perception while testing.
- Understand the business needs for each requirement and test accordingly.
Impact of Issues found during this Testing
Any issues discovered during the Acceptance test phase should be treated as a high priority and resolved as soon as possible. This necessitates performing Root Cause Analysis on every issue discovered.
The testing team is critical in providing RCAs for Acceptance issues. These also aid in determining how effectively Testing is carried out.
Furthermore, valid issues in the acceptance test will impact the testing and development teams’ efforts in terms of impressions, ratings, customer surveys, etc. When the testing team’s ignorance of validations is discovered, it can lead to escalation.
Read more: Integration Testing: Definition, Types & Examples
Use
This Testing is useful in several aspects.
A few of these include:
- To figure out the issues missed during the functional testing phase.
- How well the product is developed.
- A product is what the customers need.
- Feedback/surveys conducted help in improving the Product performance and user experience.
- Improve the process followed by having RCAs as input.
- Minimize or eliminate the issues arising from the Production Product.
Differences between System Testing, Acceptance Testing, and User Acceptance Testing
| System Testing | Acceptance Testing | User Acceptance Testing |
| End-to-end testing is performed to verify whether Product meets all the specified requirements | Testing is performed to verify whether Product meets customer requirements for acceptability | Testing is performed to verify whether end-users requirements are fulfilled for acceptability |
| A product is tested as the whole focusing only on functional and non-functional needs | Product is tested for business needs – user acceptability, business goals, rules and regulations, operations, etc. | Product is tested only for user acceptability |
| Testing team performs System Testing | Customer, Customers’ customers, tester (rarely), management, Sales, Support teams performs acceptance testing depending on the type of test carried out | Customer, Customers’ customer, testers (rarely) performs user acceptance testing |
| Test cases are written and executed | Acceptance tests are written and executed | User Acceptance tests are written and executed |
| Can be functional and non-functional | Usually Functional, but non-functional in case of RAT, OAT, etc | Only Functional |
| Only test data is used for testing | Real-time data/production data is used for testing | Real-time data / Production data is used for testing |
| Positive and negative tests are performed | Usually Positive tests are performed | Only Positive tests are performed |
| Issues found are considered as bugs and fixed based on severity and priority | Issues found marks Product as Failure, and considered to be fixed immediately | Issues found marks Product as Failure and considered to be fixed immediately |
| Controlled manner of testing | Can be controlled or uncontrolled based on type of testing | Uncontrolled manner of testing |
| Testing on Development environment | Testing on Development environment or pre-production environment or production environment, based on type | Testing is always on Pre-Production environment |
| No assumptions, but if any can be communicated | No assumptions | No assumptions |
Acceptance Tests
We have acceptance tests, which are similar to product test cases. Acceptance criteria for user stories are used to generate acceptance tests. These are high-level scenarios describing what the product must do under various conditions.
It does not provide a clear picture of how to conduct tests as test cases do. Acceptance tests are written by testers who thoroughly understand the product, known as Subject Matter Expertise. All written tests are reviewed by a customer and/or business analyst.
These tests are run as part of the acceptance test. A detailed document detailing any necessary setups and acceptance tests must be prepared. It should include every minute detail, including screenshots, setup values, conditions, etc.
Acceptance Test Mattress
This Testing requires a separate testbed that is similar to a regular testbed. Platform with all required hardware, software, operating products, network setup and configurations, server setup and configurations, database setup and configurations, licenses, plug-ins, and so on, must be configured exactly like the Production environment.
The acceptance testbed is a platform/environment in which the designed acceptance tests are run. It is a good practice to check for environmental issues and the product’s stability before handing over the Acceptance test environment to the customer.
A stable testing environment can be used if no separate environment is set up for acceptance testing. However, it will be a mess because the test data from regular System Testing and the real-time data from acceptance testing are kept in the same environment.
The acceptance testbed is typically set up on the customer’s side (i.e., in the laboratory) with limited access to the development and testing teams.
Teams will be required to access this environment via VMs/or specially designed URLs using special access credentials, and all access to this environment will be tracked. Nothing on this environment should be added/modified/deleted without the customer’s permission, and any changes should be communicated to them.
Entry and Exit Criteria for AT
Acceptance testing, like any other phase in the STLC, has a set of entry and exit criteria that must be clearly defined in the Acceptance Test Plan (which is covered in the latter part of this tutorial).
This phase follows System testing and concludes before the Production launch. As a result, the System testing Exit criteria become part of the AT Entry criteria. Similarly, the AT Exit criteria become part of the Production Launch Entry criteria.
Entry Criteria
Given below are the conditions to be fulfilled before starting:
- Business requirements should be clear and available.
- The system and Regression testing phase should be completed.
- All Critical, Major, and Minor bugs should be fixed and closed (Minor bugs accepted mainly are cosmetic bugs that do not disturb the product usage).
- A list of known issues should be created and distributed to stakeholders.
- Acceptance Test Bed should be set up and a high-level check should be performed for no environmental issues.
- The system Testing phase should be completed, allowing the product to proceed to the AT phase (Usually done through Email communication).
Exit Criteria
There are certain conditions to be fulfilled by AT to let the product go for a Production Launch.
They are as follows:
- Acceptance tests should be executed, and all the tests should pass.
- No Critical/Major defects left Open. All the defects should be fixed and verified immediately.
- AT should be Signed-off-by all the included stakeholders with Go/No-Go Decision on the product.
Read more: What is Regression Testing? Definition, Tools and How to Begin
Acceptance Testing Process
In the V-Model, the AT phase runs concurrently with the Requirements phase.
The actual AT procedure is as follows:
Business Requirements Analysis
Business requirements are analyzed by consulting all available project documents.
Some of which are:
- System Requirement Specifications
- Business Requirements Document
- Use Cases
- Workflow diagrams
- Designed data matrix
Design Acceptance Test Plan
Certain items must be documented in the Acceptance Test Plan.
Let’s look at a few of them:
- Acceptance Testing Strategy and Methodology
- Entry and exit criteria must be clearly defined.
- The scope of AT should be clearly stated, and it should only cover business requirements.
- The acceptance test design approach should be detailed so that anyone writing tests understands how they should be written.
- The setup of the test bed and the actual testing schedule/timelines should be mentioned.
- Because various stakeholders do Testing, details on logging bugs should be mentioned because the stakeholders may be unaware of the procedure used.
Design and Review Acceptance Tests
Acceptance tests should be written at the scenario level, stating what needs to be done (not in detail to include how to do it). These should only be written for the identified scope areas for business requirements, and each test must be mapped to its referencing requirement.
All written acceptance tests must be reviewed to achieve high coverage of business requirements.
This ensures that no other tests outside the scope mentioned are included, allowing Testing to occur within the scheduled timelines.
Acceptance Test Bed Set up
The examination The bed should be configured similarly to a production environment. To confirm the stability and usage of the environment, very high-level checks are required. Only give the credentials to use the environment to a stakeholder conducting the Testing.
Acceptance Test Data Set-Up
Production data must be prepared/populated in the systems as test data. A detailed document should also state that the data must be used for Testing.
Instead of having test data like TestName1, TestCity1, and so on, have Albert, Mexico, and so on. This provides a rich experience of real-time data, and Testing will be up to date.
Acceptance Test Execution
At this point, designed acceptance tests must be run on the environment. Ideally, all of the tests should pass on the first try. Suppose there are any functional bugs discovered during Acceptance testing. In that case, they should be reported as a high priority to be fixed.
Again, bugs that have been fixed must be verified and closed as a high-priority task. A daily report on test execution is required.
Bugs discovered during this phase should be discussed in a bug-triage meeting. They must go through the Root Cause Analysis procedure. This is the only point at which acceptance testing determines whether or not the product meets all of the business requirements.
Business Decision
A Go/No-Go decision is made for the product to be launched in production. The product will be released to the market if the decision is made. A no-go decision declares the product a failure.
A Few No-Go Decision Factors:
- Product of poor quality.
- Too Many open Functional Bugs.
- Deviation from business requirements
- Not up to market standards and requires improvements to meet current market standards.
This Testing’s Success Factors
Once this test has been planned, create a checklist to increase its success rate. Before the Acceptance test begins, some actions must be taken.
They are as follows:
- Have a well-defined scope and ensure that the scope identified for this Testing serves a business need.
- At least once, run acceptance tests during the system testing phase.
- Ad hoc testing should be extended for each of the acceptance test scenarios.
Conclusion
Acceptance testing, in a nutshell, aids in determining the efficiency of development and testing teams.
There are several tools for carrying out this activity. Still, it is usually preferred to be done manually because it involves real users and other stakeholders with no technical background, and it may not be feasible for them.