1.1 License

This work is (c) the author(s).

This work is licensed under a Creative Commons Attribution 4.0 International License unless otherwise marked.

For the avoidance of doubt and explanation of terms please refer to the full license notice and legal code.

1.2 Citation

If you wish to use any of the material from this paper please cite as:

• Ben Anderson and Tom Rushby. (2018) Statistical Power, Statistical Significance, Study Design and Decision Making: A Worked Example (Sizing Demand Response Trials in New Zealand), Southampton: University of Southampton.

This work is (c) 2018 the authors.

1.3 History

Code & report history:

• Paper history

1.4 Data:

This report uses circuit level extracts for ‘Heat Pumps’ from the NZ GREEN Grid Household Electricity Demand Data (https://dx.doi.org/10.5255/UKDA-SN-853334 (Anderson et al. 2018)). These have been extracted using the code found in https://github.com/CfSOtago/GREENGridData/blob/master/examples/code/extractCleanGridSpy1minCircuit.R

1.5 Acknowledgements

This work was supported by:

• The University of Otago;
• The University of Southampton;
• The New Zealand Ministry of Business, Innovation and Employment (MBIE) through the NZ GREEN Grid grant (Contract ID: UOCX1203);
• The UK Office of Gas and Electricity Markets through the Low Carbon Network Fund-funded ‘Solent Achieving Value from Efficiency’ (SAVE) project;
• SPATIALEC - a Marie Skłodowska-Curie Global Fellowship based at the University of Otago’s Centre for Sustainability (2017-2019) & the University of Southampton’s Sustainable Energy Research Group (2019-202).

4.1 Means

Observations are summarised to mean W per household during 16:00 - 20:00 on weekdays for year = 2015.

## Warning: replacing previous import 'data.table::melt' by 'reshape2::melt'
## when loading 'weGotThePower'

## Warning: replacing previous import 'data.table::dcast' by 'reshape2::dcast'
## when loading 'weGotThePower'

Figure 4.1 shows the initial p = 0.01 plot.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.1: Power analysis results (p = 0.01, power = 0.8)

## Saving 7 x 5 in image

Effect size at n = 1000: 9.08.

Figure 4.2 shows the plot for all results.

## Scale for 'y' is already present. Adding another scale for 'y', which
## will replace the existing scale.

Figure 4.2: Power analysis results (power = 0.8)

## Saving 7 x 5 in image

At same effect size (9.0816159%, n = 1000, p = 0.01):

• p = 0.05, n = 575
• p = 0.1, n = 425
• p = 0.2, n = 250

Full table of results:

## Using 'effectSize' as value column. Use 'value.var' to override

4.2 Proportions

Does not require a sample. As a relatively simple example, suppose we were interested in the adoption of heat pumps in two equal sized samples. Suppose we thought in one sample (say, home owners) we thought it might be 40% and in rental properties it would be 25% (ref BRANZ 2015). What sample size would we need to conclude a significant difference with power = 0.8 and at various p values?

pwr::pwr.tp.test() (ref pwr) can give us the answer…

We can repeat this for other values of p1 and p2. For example, suppose both were much smaller (e.g. 10% and 15%)… Clearly we need much larger samples.

The above used an arcsine transform.

As a double check, using eqn to assess margin of error…

\[me = +/- z * \sqrt{\frac{p(1-p)} {n-1}}\]

If:

• p = 0.4 (40%)
• n = 151

then the margin of error = +/- 0.078 (7.8%). So we could quote the Heat Pump uptake for owner-occupiers as 40% (+/- 7.8% [or 32.2 - 47.8] with p = 0.05).

This may be far too wide an error margin for our purposes so we may instead have recruited 500 per sample. Now the margin of error is +/- 0.043 (4.3%) so we can now quote the Heat Pump uptake for owner-occupiers as 40% (+/- 4.3% [or 35.7 - 44.3] with p = 0.05).

5.1 Getting it ‘wrong’

Use base GREENGrid and number of people but re-sample slightly.

NB: we create a small sample roughly 2 * the size of the GREEN Grid data. Due to small numbers and the random re-sampling with replacement process, there will be random fluctuations in the results with each run. As a result the results in this section will probably not match the results in the paper…

So a sample of 52.

T test 1 <-> 3

The results show that the mean power demand for the control group was 429.27W and for Intervention 1 was 147.93W. This is a (very) large difference in the mean of 281.35. The results of the t test are:

• effect size = 281W or 66% representing a substantial bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -471.49 to -91.2 representing considerable uncertainty/variation;
• p value of 0.006 representing a relatively low risk of a false positive result but which (just) fails the conventional p < 0.05 threshold.

T test 1 <-> 4+

Now:

• effect size = 322W or 68.55% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -502.9 to -141.89 representing even greater uncertainty/variation;
• p value of 0.001 representing a higher risk of a false positive result which fails the conventional p < 0.05 threshold and also the less conservative p < 0.1.

5.2 Getting it ‘right’

NB: we create a larger sample roughly 40 * the size of the GREEN Grid data. Due to the random re-sampling with replacement process, there will be random fluctuations in the results with each run. As a result the results in this section will probably not exactly match the results in the paper but as the sample is large they should be quite close…

So n = 1040

Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidence intervals for the sample mean)

re-run T tests 1 vs 3

In this case:

• effect size = 351.7837236W or 68.84% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -396.47 to -307.1 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

re-run T tests 1 person vs 4+

In this case:

• effect size = 258.1328841W or 61.85% representing a still reasonable bang for buck for whatever caused the difference;
• 95% confidence interval for the test = -297.89 to -218.38 representing much less uncertainty/variation;
• p value of 0 representing a very low risk of a false positive result as it passes all conventional thresholds.

6.1 Statistical power and sample design

6.2 Reporting statistical tests of difference (effects)

6.3 Making inferences and taking decisions