Evaluation: Bayes factors using summary statistics

Introduction

This vignette illustrates how an auditor can use jfa to evaluate their sample outcomes with the Bayes factor using summary statistics alone. An example application in the context of quality control is described below.

Application

Two-stage sampling plans are required if an initial sample does not provide enough assurance to accept (or reject) a population definitively. In the field of quality control, ISO 28596 ‘Sampling procedures for inspection by attributes — Two-stage sampling plans for auditing and for inspection under prior information’ provides two-stage sampling plans by attributes for inspection for a proportion of nonconforming items in a target population of discrete units. For example, this standard applies for inspections of a proportion of nonconforming items in a lot of product items, or inspections of the proportion of nonconforming test characteristics of a product subject to an acceptance test.

Standard procedure

The idea behind the standard two-stage sampling plan is simple. In the first stage, after drawing \(n_1\) items and observing \(k_1\) nonconforming items, a confidence interval for the proportion of nonconforming items is calculated. The decision in the first stage proceeds according to one of the three following scenarios:

If the confidence interval completely lies below the tolerance value, the lot is accepted.
If the confidence interval completely lies above the tolerance value, the lot is rejected.
If the confidence interval contains the tolerance value, no decision can be made in the first stage. In this case, the auditor proceeds to the second stage.

In the second stage, a number \(k_2\) of nonconforming items is observed in a sample of size \(n_2\). A new confidence interval for the proportion of nonconforming items is calculated based on the total number \(k_1 + k_2\) of nonconforming items. The decision in the second stage proceeds according to one of the two following scenarios:

If the confidence interval completely lies above or below the tolerance value, the decision for approval and rejection is analogous to that in the first stage.
If the interval contains the tolerance value, the decision is based on whether a bigger portion of the confidence interval lies below or above the tolerance value. If a larger portion of the interval lies above the tolerance value, the lot is rejected. If a larger portion of the interval lies below the tolerance value, the lot is accepted.

Numerical example

The auditor is inspecting a lot of products and wants to make a statement with 90% confidence against a tolerance value of 5%. They formulate the hypothesis of intolerable deviations as \(H_0:\theta \geq 0.05\) and the hypothesis of tolerable deviations as \(H_1: \theta < 0.05\).

confidence <- 0.90 # 90% confidence
tolerance  <- 0.05 # 5% tolerance

In the first stage, the auditor inspects \(n_1 = 32\) items and observes \(k_1 = 2\) nonconforming items. The shortest two-sided confidence interval (Göb & Lurz, 2012) for the proportion of nonconforming items is [0.0167; 0.1866]. This confidence interval contains the tolerance value and therefore no decision can be made in the first stage. The auditor therefore continues to the second stage.

ISO28596:::CI.binom.shortest(size = 32, x = 2, level = confidence)$bounds
#      x estimate      lower     upper
# [1,] 2   0.0625 0.01674365 0.1866428

In the second stage, the auditor inspects an additional \(n_2 = 50\) items and observes \(k_2 = 0\) nonconforming items. After observing these data, the shortest two-sided confidence interval for the proportion of nonconforming items (using \(n = n_1 + n_2\) and \(k = k_1 + k_2\)) is [0.0065; 0.0734]. At this point, the confidence interval still contains the tolerance value but the auditor determines that the portion of the confidence interval that lies below the tolerance value is larger than the portion above the tolerance value. Therefore, they accept the lot.

ISO28596:::CI.binom.shortest(size = 32 + 50, x = 2 + 0, level = confidence)$bounds
#      x   estimate       lower      upper
# [1,] 2 0.02439024 0.006504196 0.07341118

Bayes factors

The jfa package can be used to calculate Bayes factors for the hypothesis of tolerable deviations \(H_1\) in the lot versus the hypothesis of intolerable deviations \(H_0\) in the lot. The Bayes factor can provide insight into the weight of evidence in favor of acceptance of the lot versus the evidence in favor of rejection of the lot. Especially in the situation where the confidence interval contains the tolerance value the Bayes factor can provide an easy interpretation of the evidence in favor of acceptance versus rejection.

\[BF_{10} = \underbrace{\frac{p(\text{data} | H_1)}{p(\text{data} | H_0)}}_{\text{Relative evidence}} \]

Prior distribution

To calculate the Bayes factor the auditor must first specify the prior distribution for the proportion of nonconforming items in the lot. An appropriate prior distribution can be one that assigns equal probability to the event of acceptance and the event of rejection (Derks et al., 2021), see also the vignette Prior distributions. The code below creates this prior distribution.

prior <- auditPrior(method = "impartial", likelihood = "binomial", materiality = 0.05)

Bayes factor after stage 1

Using the evaluation() function the auditor can create the prior distribution and calculate the Bayes factor in favor of acceptance of the lot versus rejection of the lot. In this case, the Bayes factor in favor of acceptance (\(BF_{-+}\)) is 0.66. This Bayes factor represents only anecdotal evidence in favor of rejection of the lot, since the data is 1 / 0.66 = 1.51 times more likely under the hypothesis \(H_0\) than under the hypothesis \(H_1\) (see the vignette Testing misstatement for the interpretation of \(BF_{-+}\)).

evaluation(materiality = tolerance, x = 2, n = 32, prior = prior)

## Warning in evaluation(materiality = tolerance, x = 2, n = 32, prior = prior):
## using 'method = binomial' from 'prior'

## 
##  Bayesian Audit Sample Evaluation
## 
## data:  2 and 32
## number of errors = 2, number of samples = 32, taint = 2, BF₁₀ = 0.66487
## alternative hypothesis: true misstatement rate is less than 0.05
## 95 percent credible interval:
##  0.0000000 0.1319424
## estimate:
##  0.04493028 
## estimates obtained via method 'binomial' + 'prior'

Bayes factor after stage 2

The Bayes factor in favor of acceptance after the second sample of items is 6.24 which represents moderate evidence in favor of acceptance. Hence, we should continue selecting and inspecting items to gather sufficient evidence for one of the two hypotheses.

evaluation(materiality = tolerance, x = 2 + 0, n = 32 + 50, prior = prior)

## Warning in evaluation(materiality = tolerance, x = 2 + 0, n = 32 + 50, prior =
## prior): using 'method = binomial' from 'prior'

## 
##  Bayesian Audit Sample Evaluation
## 
## data:  2 and 82
## number of errors = 2, number of samples = 82, taint = 2, BF₁₀ = 6.2413
## alternative hypothesis: true misstatement rate is less than 0.05
## 95 percent credible interval:
##  0.00000000 0.06444852
## estimate:
##  0.02116102 
## estimates obtained via method 'binomial' + 'prior'

References

Derks, K., de Swart, J., Wagenmakers, E.-J., and Wetzels, R. (2021). A default Bayesian hypothesis test for audit sampling. Working paper.
Derks, K., de Swart, J., van Batenburg, P., Wagenmakers, E.-J., and Wetzels, R. (2021). Priors in a Bayesian audit: How integration of existing information into the prior distribution can improve audit transparency and efficiency. International Journal of Auditing, 25(3), 621-636.
Göb R. & Lurz, K. (2014). Shortest two-sided confidence intervals for a probability under prior information. Metrika: International Journal for Theoretical and Applied Statistics, 77 (3), 389-413.