In my last post, I explained few of the theoretical aspects of software reliability testing. Today I would like to explain in step by step how we can perform reliability testing in practice. Lets consider a simple example for this. You wan to test the reliability factor of sms application.
Step 1: Establish reliability goal
Before starting reliability testing we need to define what reliability is as per the customer expectation in a measurable format. For example customer is ready to accept your application only if 5 or less errors result from 1000 live operations. One more example can be customer can accept no more than two class 2 failure per release. This goal should be defined before design phase of software development and everyone in development and testing team should have clear understanding of the same.
Along with reliability goal definition process, failure classification should also be done and documented. For example:
Class 1 - System state and data are corrupted
Class 2 - Occurs with certain input data not reproducible always etc.
Step 2: Create operational profile
As a developer or tester you should be well aware of how end user/customer is going to use the application delivered to them in actual production environment.Each and every probable operation triggered by customer or user should be documented with their probabilities of occurrences. Following table will help you create a operational profile for application.
| Operation | Operation per hour | Probability |
| Send sms | 1000 | 0.50 |
| Receive sms | 500 | 0.35 |
| Create draft | 10 | 0.15 |
| Total | 1510 | 1 |
Step 3:Test planning, design and execution
Operational profile derives test planning and execution for reliability testing. Probability and critical factors are used to allocate test cases. This ensures that the testing exercises the most important or most frequently used functions first and in proportion to their significance to the system. For example in the above operational profile we need to give more importance to sending sms feature.
Next step is designing your test cases. From the requirement it is clear that we cannot achieve this target doing manual testing. We have to automate our test cases. Design your test case in such a way that you can achieve and demonstrate your reliability goal. In this example we need to send 1000 sms per hour. Automate the same and monitor the progress and application performance after a certain interval for example after each 500 iteration take a system log dump and keep a note of any other errors like crash/hang etc. You need to monitor failure intensity in order to guide the test process, and to determine feasibility of release of the software. Once execution is completed prepare a reliability test report considering the following items in mind.
Initial state
Detail about the s/w and h/w versions used for this testing.
Detain about memory status like initial free RAM size, free swap memory size etc.
Any other resources used you would like to mention specifically etc.
Resource usage overview for the test rounds:
System memory usage
System load
Network usage
Processes memory usage
Resource usage changes for each of the test rounds:
Round 1
Round 2
.
.
Round n
Summary of changes between all the rounds after the initial one:
Comparative summary of changes observed after each round with initial state can be stated in this section
Error/failure summary
List of failure summary with frequency and iteration of occurrence can be added here.
Resource usage summary
In this section you can give an overall resource usage summary during the testing duration.
Step 4: Using test result to drive decisions
Creating reliability growth curve by plotting test time vs failure intensity will give you an fair idea about when your software user test will attend the desired level of reliability. You can use the same as a guideline for your exit criteria. You need to make sure that after each test iterations identified high priority faults are getting fixed by development team. For the next test iteration use the new version of the software with the relevant fixes. As and when failure intensity drops, you will approaches the defined reliability goal.
With this I am ending this discussing on reliability testing. Do share your comments and experience on the same.