1.1 License

This work is (c) the author(s).

This work is licensed under a Creative Commons Attribution 4.0 International License unless otherwise marked.

For the avoidance of doubt and explanation of terms please refer to the full license notice and legal code.

1.2 Citation

If you wish to use any of the material from this paper please cite as:

• Ben Anderson and Tom Rushby. (2018) Statistical Power, Statistical Significance, Study Design and Decision Making: A Worked Example (Sizing Demand Response Trials in New Zealand), Southampton: University of Southampton.

This work is (c) 2018 the authors.

1.3 History

Code & report history:

• Paper history

1.4 Data:

This report uses circuit level extracts for ‘Heat Pumps’ from the NZ GREEN Grid Household Electricity Demand Data (https://dx.doi.org/10.5255/UKDA-SN-853334 (Anderson et al. 2018)). These have been extracted using the code found in https://github.com/CfSOtago/GREENGridData/blob/master/examples/code/extractCleanGridSpy1minCircuit.R

1.5 Acknowledgements

This work was supported by:

• The University of Otago;
• The University of Southampton;
• The New Zealand Ministry of Business, Innovation and Employment (MBIE) through the NZ GREEN Grid grant (Contract ID: UOCX1203);
• The UK Office of Gas and Electricity Markets through the Low Carbon Network Fund-funded ‘Solent Achieving Value from Efficiency’ (SAVE) project;
• SPATIALEC - a Marie Skłodowska-Curie Global Fellowship based at the University of Otago’s Centre for Sustainability (2017-2019) & the University of Southampton’s Sustainable Energy Research Group (2019-202).

4.1 Means

Observations are summarised to mean W per household during 16:00 - 20:00 on weekdays for year = 2015.

## Warning: replacing previous import 'data.table::melt' by 'reshape2::melt'
## when loading 'weGotThePower'

## Warning: replacing previous import 'data.table::dcast' by 'reshape2::dcast'
## when loading 'weGotThePower'

Figure 4.1 shows the initial p = 0.01 plot.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.1: Power analysis results (p = 0.01, power = 0.8)

## Saving 7 x 5 in image

Effect size at n = 1000: 28.37.

Figure 4.2 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.2: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

4.2 Proportions

Does not require a sample. As a relatively simple example, suppose we were interested in the adoption of heat pumps in two equal sized samples. Suppose we thought in one sample (say, home owners) we thought it might be 40% and in rental properties it would be 25% (ref BRANZ 2015). What sample size would we need to conclude a significant difference with power = 0.8 and at various p values?

pwr::pwr.tp.test() (ref pwr) can give us the answer…

We can repeat this for other values of p1 and p2. For example, suppose both were much smaller (e.g. 10% and 15%)… Clearly we need much larger samples.

The above used an arcsine transform.

As a double check, using eqn to assess margin of error…

\[me = +/- z * \sqrt{\frac{p(1-p)} {n-1}}\]

If:

• p = 0.4 (40%)
• n = 151

then the margin of error = +/- 0.078 (7.8%). So we could quote the Heat Pump uptake for owner-occupiers as 40% (+/- 7.8% [or 32.2 - 47.8] with p = 0.05).

This may be far too wide an error margin for our purposes so we may instead have recruited 500 per sample. Now the margin of error is +/- 0.043 (4.3%) so we can now quote the Heat Pump uptake for owner-occupiers as 40% (+/- 4.3% [or 35.7 - 44.3] with p = 0.05).

5.1 Getting it ‘wrong’

T test group 1

The results show that the mean power demand for the control group was 162.67W and for Intervention 1 was 58.81W. This is a (very) large difference in the mean of 103.86. The results of the t test are:

• effect size = 104W or 64% representing a substantial bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -236.83 to 29.1 representing considerable uncertainty/variation;
• p value of 0.122 representing a relatively low risk of a false positive result but which (just) fails the conventional p < 0.05 threshold.

T test Group 2

Now:

• effect size = 128W or 78.4% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -258.11 to 3.05 representing even greater uncertainty/variation;
• p value of 0.055 representing a higher risk of a false positive result which fails the conventional p < 0.05 threshold and also the less conservative p < 0.1.

To detect Intervention Group 2’s effect size of 78.4% would have required control and trial group sizes of 31 respectively.

5.2 Getting it ‘right’

Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidence intervals for the sample mean)

re-run T tests Control vs Group 1

In this case:

• effect size = 101.9274343W or 63.53% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = 82.18 to 121.67 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

re-run T tests Control vs Group 2

In this case:

• effect size = 124.0940533W or 77.34% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = 104.79 to 143.4 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

6.1 Statistical power and sample design

6.2 Reporting statistical tests of difference (effects)

6.3 Making inferences and taking decisions