Confidence is a set of metrics and constraints used to measure software reliability. It allows for evaluation of software in agile development phases. The goal of confidence assessment is to make sure software is safe for deployment in a production environment.
This study presents which metrics can be relevant and useful when assessing the confidence in a continuous delivery pipeline. Continuous delivery pipelines are used to produce software in short cycles and contain a lot of information. To find potential relevant data for confidence assessment, first, a literature review is performed. Then, implementation of a traceability solution in a continuous delivery pipeline is investigated. Software traceability is the ability to trace artifacts in the pipeline. In this study, it is enabled by using the Eiffel framework. A case study is conducted with eight quality assurance experts with varying work experience. The results of the case study are analyzed to identify important metrics and concepts to consider when it comes to assessing the confidence in a continuous delivery pipeline.
The results indicate that build outcome, confidence level change and test coverage are the most relevant metrics. Build outcome is the rate of various outcomes from building a software project. Confidence level change is the trend in the level of confidence in the continuous delivery pipeline. Test coverage is the degree to which the source code is executed during testing. In the conducted interviews, all the experts ranked build outcome as highly as possible. 87.5% of the experts also ranked confidence level change as high as they could, while 75% did it for test coverage. The main indication of this study is that some of the identified metrics found by collecting data using the implemented traceability solution have not been discussed in the state-of-the-art. Specifically, the metrics: build outcome, confidence level change, test suite outcome and issues resolved.