1.1 Paper circulation:

• Public

1.2 License

This work is made available under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) License.

This means you are free to:

• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material for any purpose, even commercially.

Under the following terms:

• Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
• No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Notices:

• You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.
• No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material. #YMMV

For the avoidance of doubt and explanation of terms please refer to the full license notice and legal code.

1.3 Citation

If you wish to use any of the material from this paper please cite as:

• Ben Anderson and Tom Rushby. (2018) Statistical Power, Statistical Significance, Study Design and Decision Making: A Worked Example (Sizing Demand Response Trials in New Zealand), Southampton: University of Southampton.

This work is (c) 2018 the authors.

1.4 History

Code & report history:

• Paper history

1.5 Data:

This report uses circuit level extracts for ‘Heat Pumps’ from the NZ GREEN Grid Household Electricity Demand Data (https://dx.doi.org/10.5255/UKDA-SN-853334 (Anderson et al. 2018)). These have been extracted using the code found in https://github.com/CfSOtago/GREENGridData/blob/master/examples/code/extractCleanGridSpy1minCircuit.R

1.6 Acknowledgements

This work was supported by:

• The University of Otago;
• The University of Southampton;
• The New Zealand Ministry of Business, Innovation and Employment (MBIE) through the NZ GREEN Grid grant (Contract ID: UOCX1203);
• The UK Office of Gas and Electricity Markets through the Low Carbon Network Fund-funded ‘Solent Achieving Value from Efficiency’ (SAVE) project;
• SPATIALEC - a Marie Skłodowska-Curie Global Fellowship based at the University of Otago’s Centre for Sustainability (2017-2019) & the University of Southampton’s Sustainable Energy Research Group (2019-202).

4.1 Means

## Warning: replacing previous import 'data.table::melt' by 'reshape2::melt'
## when loading 'weGotThePower'

## Warning: replacing previous import 'data.table::dcast' by 'reshape2::dcast'
## when loading 'weGotThePower'

Figure 4.1 shows the initial p = 0.05 plot.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.1: Power analysis results (p = 0.05, power = 0.8)

## Saving 7 x 5 in image

Effect size at n = 1000: 11.12.

Figure 4.2 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.2: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

4.2 Proportions

Does not require a sample.

Figure 4.3 shows the initial p = 0.05 plot. This shows the difference that would be required

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.3: Power analysis results for proportions (p = 0.05, power = 0.8)

## Saving 7 x 5 in image

Figure 4.4 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.4: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

5.1 Getting it ‘wrong’

T test group 1

The results show that the mean power demand for the control group was 162.67W and for Intervention 1 was 35.14W. This is a (very) large difference in the mean of 127.53. The results of the t test are:

• effect size = 128W or 78% representing a substantial bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -258.11 to 3.05 representing considerable uncertainty/variation;
• p value of 0.055 representing a relatively low risk of a false positive result but which (just) fails the conventional p < 0.05 threshold.

T test Group 2

Now:

• effect size = 104W or 63.85% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -236.83 to 29.1 representing even greater uncertainty/variation;
• p value of 0.122 representing a higher risk of a false positive result which fails the conventional p < 0.05 threshold and also the less conservative p < 0.1.

To detect Intervention Group 2’s effect size of 63.85% would have required control and trial group sizes of 47 respectively.

5.2 Getting it ‘right’

Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidence intervals for the sample mean)

re-run T tests Group 1

In this case:

• effect size = 110.3910563W or 65.29% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -130.92 to -89.87 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

6.1 Statistical power and sample design

6.2 Reporting statistical tests of difference (effects)

6.3 Making inferences and taking decisions