Skip to content

GitLab

Administrator approval is now required for registering new accounts. If you are registering a new account, and are external to the University, please ask the repository owner to contact ServiceLine to request your account be approved. Repository owners must include the newly registered email address, and specific repository in the request for approval.

Commit bce9ea01 authored by Ben Anderson's avatar Ben Anderson
Browse files

rejigged season order; amended final t tests for large sample for comparability; run to html

parent 301c0016
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 7531 additions and 72 deletions
paper/.DS_Store
View file @ bce9ea01
No preview for this file type
paper/weGotThePowerDraftPaper.Rmd
View file @ bce9ea01
......@@ -9,10 +9,6 @@ author: '`r paste0(params$author)` (Contact: b.anderson@soton.ac.uk, `@dataknut`
date: 'Last run at: `r Sys.time()`'
always_allow_html: yes
output:
bookdown::word_document2:
fig_caption: yes
toc: yes
toc_depth: 2
bookdown::html_document2:
code_folding: hide
fig_caption: yes
......@@ -21,6 +17,10 @@ output:
toc: yes
toc_depth: 2
toc_float: yes
bookdown::word_document2:
fig_caption: yes
toc: yes
toc_depth: 2
bookdown::pdf_document2:
fig_caption: yes
keep_tex: yes
......@@ -394,9 +394,10 @@ This may be far too wide an error margin for our purposes so we may instead have
```{r smallNTable}
# mimic an intervention using seasons
testDT <- testDT[season == "Winter", group := "Control"]
testDT <- testDT[season == "Summer", group := "Intervention 1"]
testDT <- testDT[season == "Spring", group := "Intervention 2"]
testDT <- testDT[season == "Spring", group := "Intervention 1"]
testDT <- testDT[season == "Summer", group := "Intervention 2"]
testDT <- testDT[season == "Autumn", group := "Intervention 3"]
t <- testDT[, .("mean W" = mean(meanW),
......@@ -445,10 +446,10 @@ tTest <- t.test(testDT[group == "Intervention 1"]$meanW, testDT[group == "Contro
tTestTidy <- broom::tidy(tTest)
tTestTidy$`Control mean` <- tTestTidy$estimate2
tTestTidy$`Group 1 mean` <- tTestTidy$estimate1
tTestTidy$`Intervention 1 mean` <- tTestTidy$estimate1
tTestTidy$`Mean difference` <- tTestTidy$estimate
knitr::kable(tTestTidy[c("Control mean", "Group 1 mean", "Mean difference",
knitr::kable(tTestTidy[c("Control mean", "Intervention 1 mean", "Mean difference",
"statistic", "p.value", "conf.low", "conf.high")], caption = "T test results (Group 1 vs Control)")
controlW <- tTest$estimate[[2]]
......@@ -474,10 +475,10 @@ tTest <- t.test(testDT[group == "Intervention 2"]$meanW, testDT[group == "Contro
tTestTidy <- broom::tidy(tTest)
tTestTidy$`Control mean` <- tTestTidy$estimate2
tTestTidy$`Group 2 mean` <- tTestTidy$estimate1
tTestTidy$`Intervention 2 mean` <- tTestTidy$estimate1
tTestTidy$`Mean difference` <- tTestTidy$estimate
knitr::kable(tTestTidy[c("Control mean", "Group 2 mean", "Mean difference",
knitr::kable(tTestTidy[c("Control mean", "Intervention 2 mean", "Mean difference",
"statistic", "p.value", "conf.low", "conf.high")], caption = "T test results (Group 2 vs Control)")
controlW <- tTest$estimate[[2]]
......@@ -537,23 +538,23 @@ myCaption <- paste0("Hypothetical large n sample",
makeMeanCIPlot(plotDT)
```
re-run T tests Group 1
re-run T tests Control vs Group 1
```{r largeNtTest1}
```{r largeNtTestControl-1}
# now compare winter & spring for a smaller effect
tTest <- t.test(largeTestDT[group == "Intervention 2"]$meanW, largeTestDT[group == "Control"]$meanW)
tTest <- t.test(largeTestDT[group == "Control"]$meanW, largeTestDT[group == "Intervention 1"]$meanW)
tTestTidy <- broom::tidy(tTest)
tTestTidy$`Control mean` <- tTestTidy$estimate2
tTestTidy$`Group 1 mean` <- tTestTidy$estimate1
tTestTidy$`Control mean` <- tTestTidy$estimate1
tTestTidy$`Intervention 1 mean` <- tTestTidy$estimate2
tTestTidy$`Mean difference` <- tTestTidy$estimate
knitr::kable(tTestTidy[c("Control mean", "Group 1 mean", "Mean difference",
"statistic", "p.value", "conf.low", "conf.high")], caption = "T test results (Group 1 vs Control)")
knitr::kable(tTestTidy[c("Control mean", "Intervention 1 mean", "Mean difference",
"statistic", "p.value", "conf.low", "conf.high")], caption = "T test results (Intervention 2 vs Control)")
controlW <- tTest$estimate[[2]]
intW <- tTest$estimate[[1]]
controlW <- tTest$estimate[[1]]
intW <- tTest$estimate[[2]]
cil <- tTest$conf.int[[1]]
ciu <- tTest$conf.int[[2]]
```
......@@ -564,6 +565,34 @@ In this case:
* 95% confidence interval for the test = `r round(cil,2)` to `r round(ciu,2)` representing _much less_ uncertainty/variation;
* p value of `r round(tTest$p.value,4)` representing a _very low_ risk of a false positive result as it passes all conventional thresholds.
re-run T tests Control vs Group 2
```{r largeNtTestControl-2}
# now compare winter & spring for a smaller effect
tTest <- t.test(largeTestDT[group == "Control"]$meanW, largeTestDT[group == "Intervention 2"]$meanW)
tTestTidy <- broom::tidy(tTest)
tTestTidy$`Control mean` <- tTestTidy$estimate1
tTestTidy$`Intervention 2 mean` <- tTestTidy$estimate2
tTestTidy$`Mean difference` <- tTestTidy$estimate
knitr::kable(tTestTidy[c("Control mean", "Intervention 2 mean", "Mean difference",
"statistic", "p.value", "conf.low", "conf.high")], caption = "T test results (Intervention 2 vs Control)")
controlW <- tTest$estimate[[1]]
intW <- tTest$estimate[[2]]
cil <- tTest$conf.int[[1]]
ciu <- tTest$conf.int[[2]]
```
In this case:
* effect size = `r controlW - intW`W or `r round(100 * (1-(intW/controlW)),2)`% representing a still _reasonable bang for buck_ for whatever caused the difference;
* 95% confidence interval for the test = `r round(cil,2)` to `r round(ciu,2)` representing _much less_ uncertainty/variation;
* p value of `r round(tTest$p.value,4)` representing a _very low_ risk of a false positive result as it passes all conventional thresholds.
# Summary and recommendations
......
paper/weGotThePowerDraftPaper.html
View file @ bce9ea01
......@@ -239,7 +239,7 @@ div.tocify {
<h1 class="title toc-ignore">Statistical Power, Statistical Significance, Study Design and Decision Making: A Worked Example</h1>
<h3 class="subtitle"><em>Sizing Demand Response Trials in New Zealand</em></h3>
<h4 class="author"><em>Ben Anderson and Tom Rushby (Contact: <a href="mailto:b.anderson@soton.ac.uk">b.anderson@soton.ac.uk</a>, <code>@dataknut</code>)</em></h4>
<h4 class="date"><em>Last run at: 2018-11-13 09:57:46</em></h4>
<h4 class="date"><em>Last run at: 2018-11-13 13:49:39</em></h4>
</div>
......@@ -626,16 +626,16 @@ Figure 4.2: Power analysis results (power = 0.8)
</tr>
<tr class="even">
<td align="left">Intervention 1</td>
<td align="right">35.13947</td>
<td align="right">83.90258</td>
<td align="right">22</td>
</tr>
<tr class="odd">
<td align="left">Intervention 2</td>
<td align="right">58.80597</td>
<td align="right">113.53102</td>
<td align="right">26</td>
</tr>
<tr class="odd">
<td align="left">Intervention 2</td>
<td align="right">35.13947</td>
<td align="right">83.90258</td>
<td align="right">22</td>
</tr>
<tr class="even">
<td align="left">Intervention 3</td>
<td align="right">68.37439</td>
......@@ -651,7 +651,7 @@ Figure 4.2: Power analysis results (power = 0.8)
<thead>
<tr class="header">
<th align="right">Control mean</th>
<th align="right">Group 1 mean</th>
<th align="right">Intervention 1 mean</th>
<th align="right">Mean difference</th>
<th align="right">statistic</th>
<th align="right">p.value</th>
......@@ -662,20 +662,20 @@ Figure 4.2: Power analysis results (power = 0.8)
<tbody>
<tr class="odd">
<td align="right">162.6691</td>
<td align="right">35.13947</td>
<td align="right">-127.5297</td>
<td align="right">-1.990661</td>
<td align="right">0.0552626</td>
<td align="right">-258.11</td>
<td align="right">3.050644</td>
<td align="right">58.80597</td>
<td align="right">-103.8632</td>
<td align="right">-1.587604</td>
<td align="right">0.1216582</td>
<td align="right">-236.8285</td>
<td align="right">29.10212</td>
</tr>
</tbody>
</table>
<p>The results show that the mean power demand for the control group was 162.67W and for Intervention 1 was 35.14W. This is a (very) large difference in the mean of 127.53. The results of the t test are:</p>
<p>The results show that the mean power demand for the control group was 162.67W and for Intervention 1 was 58.81W. This is a (very) large difference in the mean of 103.86. The results of the t test are:</p>
<ul>
<li>effect size = 128W or 78% representing a <em>substantial bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = -258.11 to 3.05 representing <em>considerable</em> uncertainty/variation;</li>
<li>p value of 0.055 representing a <em>relatively low</em> risk of a false positive result but which (just) fails the conventional p &lt; 0.05 threshold.</li>
<li>effect size = 104W or 64% representing a <em>substantial bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = -236.83 to 29.1 representing <em>considerable</em> uncertainty/variation;</li>
<li>p value of 0.122 representing a <em>relatively low</em> risk of a false positive result but which (just) fails the conventional p &lt; 0.05 threshold.</li>
</ul>
<p>T test Group 2</p>
<table>
......@@ -683,7 +683,7 @@ Figure 4.2: Power analysis results (power = 0.8)
<thead>
<tr class="header">
<th align="right">Control mean</th>
<th align="right">Group 2 mean</th>
<th align="right">Intervention 2 mean</th>
<th align="right">Mean difference</th>
<th align="right">statistic</th>
<th align="right">p.value</th>
......@@ -694,22 +694,22 @@ Figure 4.2: Power analysis results (power = 0.8)
<tbody>
<tr class="odd">
<td align="right">162.6691</td>
<td align="right">58.80597</td>
<td align="right">-103.8632</td>
<td align="right">-1.587604</td>
<td align="right">0.1216582</td>
<td align="right">-236.8285</td>
<td align="right">29.10212</td>
<td align="right">35.13947</td>
<td align="right">-127.5297</td>
<td align="right">-1.990661</td>
<td align="right">0.0552626</td>
<td align="right">-258.11</td>
<td align="right">3.050644</td>
</tr>
</tbody>
</table>
<p>Now:</p>
<ul>
<li>effect size = 104W or 63.85% representing a still <em>reasonable bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = -236.83 to 29.1 representing <em>even greater</em> uncertainty/variation;</li>
<li>p value of 0.122 representing a <em>higher</em> risk of a false positive result which fails the conventional p &lt; 0.05 threshold and also the less conservative p &lt; 0.1.</li>
<li>effect size = 128W or 78.4% representing a still <em>reasonable bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = -258.11 to 3.05 representing <em>even greater</em> uncertainty/variation;</li>
<li>p value of 0.055 representing a <em>higher</em> risk of a false positive result which fails the conventional p &lt; 0.05 threshold and also the less conservative p &lt; 0.1.</li>
</ul>
<p>To detect Intervention Group 2’s effect size of 63.85% would have required control and trial group sizes of 47 respectively.</p>
<p>To detect Intervention Group 2’s effect size of 78.4% would have required control and trial group sizes of 31 respectively.</p>
</div>
<div id="getting-it-right" class="section level2">
<h2><span class="header-section-number">5.2</span> Getting it ‘right’</h2>
......@@ -726,27 +726,27 @@ Figure 4.2: Power analysis results (power = 0.8)
<tbody>
<tr class="odd">
<td align="left">Control</td>
<td align="right">169.60064</td>
<td align="right">328.56355</td>
<td align="right">1140</td>
<td align="right">160.44582</td>
<td align="right">317.03541</td>
<td align="right">1128</td>
</tr>
<tr class="even">
<td align="left">Intervention 1</td>
<td align="right">34.50149</td>
<td align="right">82.94015</td>
<td align="right">907</td>
<td align="right">58.51839</td>
<td align="right">109.58111</td>
<td align="right">984</td>
</tr>
<tr class="odd">
<td align="left">Intervention 2</td>
<td align="right">59.84020</td>
<td align="right">112.74650</td>
<td align="right">1008</td>
<td align="right">36.35177</td>
<td align="right">83.36952</td>
<td align="right">903</td>
</tr>
<tr class="even">
<td align="left">Intervention 3</td>
<td align="right">73.24102</td>
<td align="right">148.25869</td>
<td align="right">1145</td>
<td align="right">70.69426</td>
<td align="right">147.43129</td>
<td align="right">1185</td>
</tr>
</tbody>
</table>
......@@ -756,13 +756,45 @@ Figure 4.2: Power analysis results (power = 0.8)
Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidence intervals for the sample mean)
</p>
</div>
<p>re-run T tests Group 1</p>
<p>re-run T tests Control vs Group 1</p>
<table>
<caption><span id="tab:largeNtTestControl-1">Table 5.5: </span>T test results (Intervention 2 vs Control)</caption>
<thead>
<tr class="header">
<th align="right">Control mean</th>
<th align="right">Intervention 1 mean</th>
<th align="right">Mean difference</th>
<th align="right">statistic</th>
<th align="right">p.value</th>
<th align="right">conf.low</th>
<th align="right">conf.high</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td align="right">160.4458</td>
<td align="right">58.51839</td>
<td align="right">101.9274</td>
<td align="right">10.12667</td>
<td align="right">0</td>
<td align="right">82.18316</td>
<td align="right">121.6717</td>
</tr>
</tbody>
</table>
<p>In this case:</p>
<ul>
<li>effect size = 101.9274343W or 63.53% representing a still <em>reasonable bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = 82.18 to 121.67 representing <em>much less</em> uncertainty/variation;</li>
<li>p value of 0 representing a <em>very low</em> risk of a false positive result as it passes all conventional thresholds.</li>
</ul>
<p>re-run T tests Control vs Group 2</p>
<table>
<caption><span id="tab:largeNtTest1">Table 5.5: </span>T test results (Group 1 vs Control)</caption>
<caption><span id="tab:largeNtTestControl-2">Table 5.6: </span>T test results (Intervention 2 vs Control)</caption>
<thead>
<tr class="header">
<th align="right">Control mean</th>
<th align="right">Group 1 mean</th>
<th align="right">Intervention 2 mean</th>
<th align="right">Mean difference</th>
<th align="right">statistic</th>
<th align="right">p.value</th>
......@@ -772,20 +804,20 @@ Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidenc
</thead>
<tbody>
<tr class="odd">
<td align="right">169.6006</td>
<td align="right">59.8402</td>
<td align="right">-109.7604</td>
<td align="right">-10.59573</td>
<td align="right">160.4458</td>
<td align="right">36.35177</td>
<td align="right">124.0941</td>
<td align="right">12.61266</td>
<td align="right">0</td>
<td align="right">-130.0807</td>
<td align="right">-89.44015</td>
<td align="right">104.7925</td>
<td align="right">143.3956</td>
</tr>
</tbody>
</table>
<p>In this case:</p>
<ul>
<li>effect size = 109.7604326W or 64.72% representing a still <em>reasonable bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = -130.08 to -89.44 representing <em>much less</em> uncertainty/variation;</li>
<li>effect size = 124.0940533W or 77.34% representing a still <em>reasonable bang for buck</em> for whatever caused the difference;</li>
<li>95% confidence interval for the test = 104.79 to 143.4 representing <em>much less</em> uncertainty/variation;</li>
<li>p value of 0 representing a <em>very low</em> risk of a false positive result as it passes all conventional thresholds.</li>
</ul>
</div>
......@@ -807,7 +839,7 @@ Figure 5.1: Mean W demand per group for large sample (Error bars = 95% confidenc
</div>
<div id="runtime" class="section level1">
<h1><span class="header-section-number">8</span> Runtime</h1>
<p>Analysis completed in 51.37 seconds ( 0.86 minutes) using <a href="https://cran.r-project.org/package=knitr">knitr</a> in <a href="http://www.rstudio.com">RStudio</a> with R version 3.5.1 (2018-07-02) running on x86_64-apple-darwin15.6.0.</p>
<p>Analysis completed in 46.02 seconds ( 0.77 minutes) using <a href="https://cran.r-project.org/package=knitr">knitr</a> in <a href="http://www.rstudio.com">RStudio</a> with R version 3.5.1 (2018-07-02) running on x86_64-apple-darwin15.6.0.</p>
</div>
<div id="r-environment" class="section level1">
<h1><span class="header-section-number">9</span> R environment</h1>
......
paper/weGotThePowerDraftPaper_files/bootstrap-3.3.5/css/bootstrap.css.map 0 → 100644
View file @ bce9ea01
This source diff could not be displayed because it is too large. You can view the blob instead.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment