1.1 License

This work is (c) the author(s).

This work is licensed under a Creative Commons Attribution 4.0 International License unless otherwise marked.

For the avoidance of doubt and explanation of terms please refer to the full license notice and legal code.

1.2 Citation

If you wish to use any of the material from this paper please cite as:

• Ben Anderson and Tom Rushby. (2018) Statistical Power, Statistical Significance, Study Design and Decision Making: A Worked Example (Sizing Demand Response Trials in New Zealand), Southampton: University of Southampton.

This work is (c) 2018 the authors.

1.3 History

Code & report history:

• Paper history

1.4 Data:

This report uses circuit level extracts for ‘Heat Pumps’ from the NZ GREEN Grid Household Electricity Demand Data (https://dx.doi.org/10.5255/UKDA-SN-853334 (Anderson et al. 2018)). These have been extracted using the code found in https://github.com/CfSOtago/GREENGridData/blob/master/examples/code/extractCleanGridSpy1minCircuit.R

1.5 Acknowledgements

This work was supported by:

• The University of Otago;
• The University of Southampton;
• The New Zealand Ministry of Business, Innovation and Employment (MBIE) through the NZ GREEN Grid grant (Contract ID: UOCX1203);
• The UK Office of Gas and Electricity Markets through the Low Carbon Network Fund-funded ‘Solent Achieving Value from Efficiency’ (SAVE) project;
• SPATIALEC - a Marie Skłodowska-Curie Global Fellowship based at the University of Otago’s Centre for Sustainability (2017-2019) & the University of Southampton’s Sustainable Energy Research Group (2019-202).

4.1 Means

Observations are summarised to mean W per household during 16:00 - 20:00 on weekdays for year = 2015.

## Warning: replacing previous import 'data.table::melt' by 'reshape2::melt'
## when loading 'weGotThePower'

## Warning: replacing previous import 'data.table::dcast' by 'reshape2::dcast'
## when loading 'weGotThePower'

Figure 4.1 shows the initial p = 0.01 plot.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.1: Power analysis results (p = 0.01, power = 0.8)

## Saving 7 x 5 in image

Effect size at n = 1000: 28.37.

Figure 4.2 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.2: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

4.2 Proportions

Does not require a sample.

Figure 4.3 shows the initial p = 0.05 plot. This shows the difference that would be required

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.3: Power analysis results for proportions (p = 0.05, power = 0.8)

## Saving 7 x 5 in image

Figure 4.4 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.4: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

5.1 Getting it ‘wrong’

T test group 1

The results show that the mean power demand for the control group was 162.67W and for Intervention 1 was 35.14W. This is a (very) large difference in the mean of 127.53. The results of the t test are:

• effect size = 128W or 78% representing a substantial bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -258.11 to 3.05 representing considerable uncertainty/variation;
• p value of 0.055 representing a relatively low risk of a false positive result but which (just) fails the conventional p < 0.05 threshold.

T test Group 2

Now:

• effect size = 104W or 63.85% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -236.83 to 29.1 representing even greater uncertainty/variation;
• p value of 0.122 representing a higher risk of a false positive result which fails the conventional p < 0.05 threshold and also the less conservative p < 0.1.

To detect Intervention Group 2’s effect size of 63.85% would have required control and trial group sizes of 47 respectively.

5.2 Getting it ‘right’

Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidence intervals for the sample mean)

re-run T tests Group 1

In this case:

• effect size = 107.8760107W or 61.37% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -128.48 to -87.27 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

6.1 Statistical power and sample design

6.2 Reporting statistical tests of difference (effects)

6.3 Making inferences and taking decisions