diff --git a/Rmd/cleaningFeederData.Rmd b/Rmd/cleaningFeederData.Rmd
index 0e3e22755e3abeaf866a305344e3e67e0ee0643d..c0e765fc08fb920c7525b43b5fe67962b22388ea 100644
--- a/Rmd/cleaningFeederData.Rmd
+++ b/Rmd/cleaningFeederData.Rmd
@@ -27,7 +27,6 @@ output:
toc: yes
toc_depth: 2
fig_width: 5
-always_allow_html: yes
bibliography: '`r paste0(here::here(), "/bibliography.bib")`'
---
@@ -64,7 +63,7 @@ We have some electricity substation feeder data that has been cleaned to give me
There seem to be some NA kW values and a lot of missing time stamps. We want to select the 'best' (i.e most complete) days within a day-of-the-week/season/year sampling frame. If we can't do that we may have to resort to seasonal mean kW profiles by hour & day of the week...
-Code used to generate this report: https://git.soton.ac.uk/ba1e12/spatialec/-/blob/master/isleOfWight/cleaningFeederData.Rmd
+The code used to generate this report is in: https://git.soton.ac.uk/ba1e12/dataCleaning/Rmd/
# Data prep
@@ -78,8 +77,7 @@ origDataDT <- drake::readd(origData) # readd the drake object
uniqDataDT <- drake::readd(uniqData) # readd the drake object
kableExtra::kable(head(origDataDT), digits = 2,
- caption = "Counts per feeder (long table)") %>%
- kable_styling()
+ caption = "First 6 rows of data")
```
Do a duplicate check by feeder_ID, dateTime & kW. In theory there should not be any.
@@ -89,16 +87,19 @@ message("Original data nrows: ", tidyNum(nrow(origDataDT)))
message("Unique data nrows: ", tidyNum(nrow(uniqDataDT)))
-message("So we have ", tidyNum(nrow(origDataDT) - nrow(uniqDataDT)), " duplicates...")
+nDups <- tidyNum(nrow(origDataDT) - nrow(uniqDataDT))
+
+message("So we have ", tidyNum(nDups), " duplicates...")
pc <- 100*((nrow(origDataDT) - nrow(uniqDataDT))/nrow(origDataDT))
message("That's ", round(pc,2), "%")
feederDT <- uniqDataDT[!is.na(rDateTime)] # use dt with no duplicates
origDataDT <- NULL # save memory
+
```
-There were `r tidyNum(nrow(origDataDT) - nrow(uniqDataDT))` duplicates - that's `r round(pc,2)` % of the observations loaded.
+There were `r tidyNum(nDups)` duplicates - that's ~ `r round(pc,2)` % of the observations loaded.
So we remove the duplicates...
@@ -321,23 +322,19 @@ ggplot2::ggplot(aggDT, aes(x = rDate, colour = season,
aggDT[, rDoW := lubridate::wday(rDate, lab = TRUE)]
h <- head(aggDT[season == "Spring"][order(-propExpected)])
kableExtra::kable(h, caption = "Best Spring days overall",
- digits = 3) %>%
- kable_styling()
+ digits = 3)
h <- head(aggDT[season == "Summer"][order(-propExpected)])
kableExtra::kable(h, caption = "Best Summer days overall",
- digits = 3) %>%
- kable_styling()
+ digits = 3)
h <- head(aggDT[season == "Autumn"][order(-propExpected)])
kableExtra::kable(h, caption = "Best Autumn days overall",
- digits = 3) %>%
- kable_styling()
+ digits = 3)
h <- head(aggDT[season == "Winter"][order(-propExpected)])
kableExtra::kable(h, caption = "Best Winter days overall",
- digits = 3) %>%
- kable_styling()
+ digits = 3)
```
# Summary
diff --git a/Rmd/cleaningFeederData_allData.log b/Rmd/cleaningFeederData_allData.log
new file mode 100644
index 0000000000000000000000000000000000000000..6b5d84b71adad7b6aebdb2e1541ba91b2f1c5685
--- /dev/null
+++ b/Rmd/cleaningFeederData_allData.log
@@ -0,0 +1,1167 @@
+This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020) (preloaded format=pdflatex 2020.7.9) 9 JUL 2020 10:04
+entering extended mode
+ restricted \write18 enabled.
+ %&-line parsing enabled.
+**/home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData.te
+x
+
+(/home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData.tex
+LaTeX2e <2020-02-02> patch level 5
+L3 programming layer <2020-06-18> (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/b
+ase/article.cls
+Document Class: article 2019/12/20 v1.4l Standard LaTeX document class
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/base/size10.clo
+File: size10.clo 2019/12/20 v1.4l Standard LaTeX file (size option)
+)
+\c@part=\count167
+\c@section=\count168
+\c@subsection=\count169
+\c@subsubsection=\count170
+\c@paragraph=\count171
+\c@subparagraph=\count172
+\c@figure=\count173
+\c@table=\count174
+\abovecaptionskip=\skip47
+\belowcaptionskip=\skip48
+\bibindent=\dimen134
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/lmodern.sty
+Package: lmodern 2009/10/30 v1.6 Latin Modern Fonts
+LaTeX Font Info: Overwriting symbol font `operators' in version `normal'
+(Font) OT1/cmr/m/n --> OT1/lmr/m/n on input line 22.
+LaTeX Font Info: Overwriting symbol font `letters' in version `normal'
+(Font) OML/cmm/m/it --> OML/lmm/m/it on input line 23.
+LaTeX Font Info: Overwriting symbol font `symbols' in version `normal'
+(Font) OMS/cmsy/m/n --> OMS/lmsy/m/n on input line 24.
+LaTeX Font Info: Overwriting symbol font `largesymbols' in version `normal'
+(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 25.
+LaTeX Font Info: Overwriting symbol font `operators' in version `bold'
+(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 26.
+LaTeX Font Info: Overwriting symbol font `letters' in version `bold'
+(Font) OML/cmm/b/it --> OML/lmm/b/it on input line 27.
+LaTeX Font Info: Overwriting symbol font `symbols' in version `bold'
+(Font) OMS/cmsy/b/n --> OMS/lmsy/b/n on input line 28.
+LaTeX Font Info: Overwriting symbol font `largesymbols' in version `bold'
+(Font) OMX/cmex/m/n --> OMX/lmex/m/n on input line 29.
+LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `normal'
+(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 31.
+LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `normal'
+(Font) OT1/cmss/m/n --> OT1/lmss/m/n on input line 32.
+LaTeX Font Info: Overwriting math alphabet `\mathit' in version `normal'
+(Font) OT1/cmr/m/it --> OT1/lmr/m/it on input line 33.
+LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `normal'
+(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 34.
+LaTeX Font Info: Overwriting math alphabet `\mathbf' in version `bold'
+(Font) OT1/cmr/bx/n --> OT1/lmr/bx/n on input line 35.
+LaTeX Font Info: Overwriting math alphabet `\mathsf' in version `bold'
+(Font) OT1/cmss/bx/n --> OT1/lmss/bx/n on input line 36.
+LaTeX Font Info: Overwriting math alphabet `\mathit' in version `bold'
+(Font) OT1/cmr/bx/it --> OT1/lmr/bx/it on input line 37.
+LaTeX Font Info: Overwriting math alphabet `\mathtt' in version `bold'
+(Font) OT1/cmtt/m/n --> OT1/lmtt/m/n on input line 38.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsfonts/amssymb.sty
+Package: amssymb 2013/01/14 v3.01 AMS font symbols
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsfonts/amsfonts.sty
+Package: amsfonts 2013/01/14 v3.01 Basic AMSFonts support
+\@emptytoks=\toks15
+\symAMSa=\mathgroup4
+\symAMSb=\mathgroup5
+LaTeX Font Info: Redeclaring math symbol \hbar on input line 98.
+LaTeX Font Info: Overwriting math alphabet `\mathfrak' in version `bold'
+(Font) U/euf/m/n --> U/euf/b/n on input line 106.
+)) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsmath/amsmath.sty
+Package: amsmath 2020/01/20 v2.17e AMS math features
+\@mathmargin=\skip49
+For additional information on amsmath, use the `?' option.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsmath/amstext.sty
+Package: amstext 2000/06/29 v2.01 AMS text
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsmath/amsgen.sty
+File: amsgen.sty 1999/11/30 v2.0 generic functions
+\@emptytoks=\toks16
+\ex@=\dimen135
+)) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsmath/amsbsy.sty
+Package: amsbsy 1999/11/29 v1.2d Bold Symbols
+\pmbraise@=\dimen136
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsmath/amsopn.sty
+Package: amsopn 2016/03/08 v2.02 operator names
+)
+\inf@bad=\count175
+LaTeX Info: Redefining \frac on input line 227.
+\uproot@=\count176
+\leftroot@=\count177
+LaTeX Info: Redefining \overline on input line 389.
+\classnum@=\count178
+\DOTSCASE@=\count179
+LaTeX Info: Redefining \ldots on input line 486.
+LaTeX Info: Redefining \dots on input line 489.
+LaTeX Info: Redefining \cdots on input line 610.
+\Mathstrutbox@=\box45
+\strutbox@=\box46
+\big@size=\dimen137
+LaTeX Font Info: Redeclaring font encoding OML on input line 733.
+LaTeX Font Info: Redeclaring font encoding OMS on input line 734.
+\macc@depth=\count180
+\c@MaxMatrixCols=\count181
+\dotsspace@=\muskip16
+\c@parentequation=\count182
+\dspbrk@lvl=\count183
+\tag@help=\toks17
+\row@=\count184
+\column@=\count185
+\maxfields@=\count186
+\andhelp@=\toks18
+\eqnshift@=\dimen138
+\alignsep@=\dimen139
+\tagshift@=\dimen140
+\tagwidth@=\dimen141
+\totwidth@=\dimen142
+\lineht@=\dimen143
+\@envbody=\toks19
+\multlinegap=\skip50
+\multlinetaggap=\skip51
+\mathdisplay@stack=\toks20
+LaTeX Info: Redefining \[ on input line 2859.
+LaTeX Info: Redefining \] on input line 2860.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/iftex/ifxetex.sty
+Package: ifxetex 2019/10/25 v0.7 ifxetex legacy package. Use iftex instead.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/iftex/iftex.sty
+Package: iftex 2020/03/06 v1.0d TeX engine tests
+)) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/iftex/ifluatex.sty
+Package: ifluatex 2019/10/25 v1.5 ifluatex legacy package. Use iftex instead.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/base/fixltx2e.sty
+Package: fixltx2e 2016/12/29 v2.1a fixes to LaTeX (obsolete)
+Applying: [2015/01/01] Old fixltx2e package on input line 46.
+
+Package fixltx2e Warning: fixltx2e is not required with releases after 2015
+(fixltx2e) All fixes are now in the LaTeX kernel.
+(fixltx2e) See the latexrelease package for details.
+
+Already applied: [0000/00/00] Old fixltx2e package on input line 53.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/base/fontenc.sty
+Package: fontenc 2020/02/11 v2.0o Standard LaTeX package
+LaTeX Font Info: Trying to load font information for T1+lmr on input line 11
+2.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/t1lmr.fd
+File: t1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/base/inputenc.sty
+Package: inputenc 2018/08/11 v1.3c Input encoding file
+\inpenc@prehook=\toks21
+\inpenc@posthook=\toks22
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hyperref/hyperref.sty
+Package: hyperref 2020-05-15 v7.00e Hypertext links for LaTeX
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty
+Package: ltxcmds 2019/12/15 v1.24 LaTeX kernel commands for general use (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty
+Package: pdftexcmds 2020-06-27 v0.33 Utility functions of pdfTeX for LuaTeX (HO
+)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/infwarerr/infwarerr.sty
+Package: infwarerr 2019/12/03 v1.5 Providing info/warning/error messages (HO)
+)
+Package pdftexcmds Info: \pdf@primitive is available.
+Package pdftexcmds Info: \pdf@ifprimitive is available.
+Package pdftexcmds Info: \pdfdraftmode found.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/keyval.sty
+Package: keyval 2014/10/28 v1.15 key=value parser (DPC)
+\KV@toks@=\toks23
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/kvsetkeys/kvsetkeys.sty
+Package: kvsetkeys 2019/12/15 v1.18 Key value parser (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/kvdefinekeys/kvdefinekeys.sty
+Package: kvdefinekeys 2019-12-19 v1.6 Define keys (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/pdfescape/pdfescape.sty
+Package: pdfescape 2019/12/09 v1.15 Implements pdfTeX's escape features (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hycolor/hycolor.sty
+Package: hycolor 2020-01-27 v1.10 Color options for hyperref/bookmark (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/letltxmacro/letltxmacro.sty
+Package: letltxmacro 2019/12/03 v1.6 Let assignment for LaTeX macros (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/auxhook/auxhook.sty
+Package: auxhook 2019-12-17 v1.6 Hooks for auxiliary files (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/kvoptions/kvoptions.sty
+Package: kvoptions 2019/11/29 v3.13 Key value format for package options (HO)
+)
+\@linkdim=\dimen144
+\Hy@linkcounter=\count187
+\Hy@pagecounter=\count188
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hyperref/pd1enc.def
+File: pd1enc.def 2020-05-15 v7.00e Hyperref: PDFDocEncoding definition (HO)
+Now handling font encoding PD1 ...
+... no UTF-8 mapping file for font encoding PD1
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/intcalc/intcalc.sty
+Package: intcalc 2019/12/15 v1.3 Expandable calculations with integers (HO)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/etexcmds/etexcmds.sty
+Package: etexcmds 2019/12/15 v1.7 Avoid name clashes with e-TeX commands (HO)
+)
+\Hy@SavedSpaceFactor=\count189
+Package hyperref Info: Option `unicode' set `true' on input line 4338.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hyperref/puenc.def
+File: puenc.def 2020-05-15 v7.00e Hyperref: PDF Unicode definition (HO)
+Now handling font encoding PU ...
+... no UTF-8 mapping file for font encoding PU
+)
+Package hyperref Info: Hyper figures OFF on input line 4464.
+Package hyperref Info: Link nesting OFF on input line 4469.
+Package hyperref Info: Hyper index ON on input line 4472.
+Package hyperref Info: Plain pages OFF on input line 4479.
+Package hyperref Info: Backreferencing OFF on input line 4484.
+Package hyperref Info: Implicit mode ON; LaTeX internals redefined.
+Package hyperref Info: Bookmarks ON on input line 4717.
+\c@Hy@tempcnt=\count190
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/url/url.sty
+\Urlmuskip=\muskip17
+Package: url 2013/09/16 ver 3.4 Verb mode for urls, etc.
+)
+LaTeX Info: Redefining \url on input line 5076.
+\XeTeXLinkMargin=\dimen145
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/bitset/bitset.sty
+Package: bitset 2019/12/09 v1.3 Handle bit-vector datatype (HO)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/bigintcalc/bigintcalc.sty
+Package: bigintcalc 2019/12/15 v1.5 Expandable calculations on big integers (HO
+)
+))
+\Fld@menulength=\count191
+\Field@Width=\dimen146
+\Fld@charsize=\dimen147
+Package hyperref Info: Hyper figures OFF on input line 6347.
+Package hyperref Info: Link nesting OFF on input line 6352.
+Package hyperref Info: Hyper index ON on input line 6355.
+Package hyperref Info: backreferencing OFF on input line 6362.
+Package hyperref Info: Link coloring OFF on input line 6367.
+Package hyperref Info: Link coloring with OCG OFF on input line 6372.
+Package hyperref Info: PDF/A mode OFF on input line 6377.
+LaTeX Info: Redefining \ref on input line 6417.
+LaTeX Info: Redefining \pageref on input line 6421.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/atbegshi/atbegshi.sty
+Package: atbegshi 2019/12/05 v1.19 At begin shipout hook (HO)
+)
+\Hy@abspage=\count192
+\c@Item=\count193
+\c@Hfootnote=\count194
+)
+Package hyperref Info: Driver (autodetected): hpdftex.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hyperref/hpdftex.def
+File: hpdftex.def 2020-05-15 v7.00e Hyperref driver for pdfTeX
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/atveryend/atveryend.sty
+Package: atveryend 2019-12-11 v1.11 Hooks at the very end of document (HO)
+Package atveryend Info: \enddocument detected (standard20110627).
+)
+\Fld@listcount=\count195
+\c@bookmark@seq@number=\count196
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/rerunfilecheck/rerunfilecheck.sty
+Package: rerunfilecheck 2019/12/05 v1.9 Rerun checks for auxiliary files (HO)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/uniquecounter/uniquecounter.sty
+Package: uniquecounter 2019/12/15 v1.4 Provide unlimited unique counter (HO)
+)
+Package uniquecounter Info: New unique counter `rerunfilecheck' on input line 2
+86.
+)
+\Hy@SectionHShift=\skip52
+)
+Package hyperref Info: Option `breaklinks' set `true' on input line 30.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/geometry/geometry.sty
+Package: geometry 2020/01/02 v5.9 Page Geometry
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/iftex/ifvtex.sty
+Package: ifvtex 2019/10/25 v1.7 ifvtex legacy package. Use iftex instead.
+)
+\Gm@cnth=\count197
+\Gm@cntv=\count198
+\c@Gm@tempcnt=\count199
+\Gm@bindingoffset=\dimen148
+\Gm@wd@mp=\dimen149
+\Gm@odd@mp=\dimen150
+\Gm@even@mp=\dimen151
+\Gm@layoutwidth=\dimen152
+\Gm@layoutheight=\dimen153
+\Gm@layouthoffset=\dimen154
+\Gm@layoutvoffset=\dimen155
+\Gm@dimlist=\toks24
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/color.sty
+Package: color 2019/11/23 v1.2a Standard LaTeX Color (DPC)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics-cfg/color.cfg
+File: color.cfg 2016/01/02 v1.6 sample color configuration
+)
+Package color Info: Driver file: pdftex.def on input line 147.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics-def/pdftex.def
+File: pdftex.def 2018/01/08 v1.0l Graphics/color driver for pdftex
+)) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/fancyvrb/fancyvrb.sty
+Package: fancyvrb 2020/05/03 v3.6 verbatim text (tvz,hv)
+\FV@CodeLineNo=\count266
+\FV@InFile=\read2
+\FV@TabBox=\box47
+\c@FancyVerbLine=\count267
+\FV@StepNumber=\count268
+\FV@OutFile=\write3
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/framed/framed.sty
+Package: framed 2011/10/22 v 0.96: framed or shaded text with page breaks
+\OuterFrameSep=\skip53
+\fb@frw=\dimen156
+\fb@frh=\dimen157
+\FrameRule=\dimen158
+\FrameSep=\dimen159
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/graphicx.sty
+Package: graphicx 2019/11/30 v1.2a Enhanced LaTeX Graphics (DPC,SPQR)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/graphics.sty
+Package: graphics 2019/11/30 v1.4a Standard LaTeX Graphics (DPC,SPQR)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/trig.sty
+Package: trig 2016/01/03 v1.10 sin cos tan (DPC)
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics-cfg/graphics.cfg
+File: graphics.cfg 2016/06/04 v1.11 sample graphics configuration
+)
+Package graphics Info: Driver file: pdftex.def on input line 105.
+)
+\Gin@req@height=\dimen160
+\Gin@req@width=\dimen161
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/grffile/grffile.sty
+Package: grffile 2019/11/11 v2.1 Extended file name support for graphics (legac
+y)
+Package grffile Info: This package is an empty stub for compatibility on input
+line 40.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/etoolbox/etoolbox.sty
+Package: etoolbox 2019/09/21 v2.5h e-TeX tools for LaTeX (JAW)
+\etb@tempcnta=\count269
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/booktabs/booktabs.sty
+Package: booktabs 2020/01/12 v1.61803398 Publication quality tables
+\heavyrulewidth=\dimen162
+\lightrulewidth=\dimen163
+\cmidrulewidth=\dimen164
+\belowrulesep=\dimen165
+\belowbottomsep=\dimen166
+\aboverulesep=\dimen167
+\abovetopsep=\dimen168
+\cmidrulesep=\dimen169
+\cmidrulekern=\dimen170
+\defaultaddspace=\dimen171
+\@cmidla=\count270
+\@cmidlb=\count271
+\@aboverulesep=\dimen172
+\@belowrulesep=\dimen173
+\@thisruleclass=\count272
+\@lastruleclass=\count273
+\@thisrulewidth=\dimen174
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/tools/longtable.sty
+Package: longtable 2020/01/07 v4.13 Multi-page Table package (DPC)
+\LTleft=\skip54
+\LTright=\skip55
+\LTpre=\skip56
+\LTpost=\skip57
+\LTchunksize=\count274
+\LTcapwidth=\dimen175
+\LT@head=\box48
+\LT@firsthead=\box49
+\LT@foot=\box50
+\LT@lastfoot=\box51
+\LT@cols=\count275
+\LT@rows=\count276
+\c@LT@tables=\count277
+\c@LT@chunks=\count278
+\LT@p@ftn=\toks25
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/tools/array.sty
+Package: array 2019/08/31 v2.4l Tabular extension package (FMi)
+\col@sep=\dimen176
+\ar@mcellbox=\box52
+\extrarowheight=\dimen177
+\NC@list=\toks26
+\extratabsurround=\skip58
+\backup@length=\skip59
+\ar@cellbox=\box53
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/multirow/multirow.sty
+Package: multirow 2019/05/31 v2.5 Span multiple rows of a table
+\multirow@colwidth=\skip60
+\multirow@cntb=\count279
+\multirow@dima=\skip61
+\bigstrutjot=\dimen178
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/wrapfig/wrapfig.sty
+\wrapoverhang=\dimen179
+\WF@size=\dimen180
+\c@WF@wrappedlines=\count280
+\WF@box=\box54
+\WF@everypar=\toks27
+Package: wrapfig 2003/01/31 v 3.6
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/float/float.sty
+Package: float 2001/11/08 v1.3d Float enhancements (AL)
+\c@float@type=\count281
+\float@exts=\toks28
+\float@box=\box55
+\@float@everytoks=\toks29
+\@floatcapt=\box56
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/colortbl/colortbl.sty
+Package: colortbl 2020/01/04 v1.0e Color table columns (DPC)
+\everycr=\toks30
+\minrowclearance=\skip62
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/pdflscape/pdflscape.sty
+Package: pdflscape 2019/12/05 v0.12 Display of landscape pages in PDF (HO)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics/lscape.sty
+Package: lscape 2000/10/22 v3.01 Landscape Pages (DPC)
+)
+Package pdflscape Info: Auto-detected driver: pdftex on input line 81.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/tabu/tabu.sty
+Package: tabu 2019/01/11 v2.9 - flexible LaTeX tabulars (FC+tabu-fixed)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/varwidth/varwidth.sty
+Package: varwidth 2009/03/30 ver 0.92; Variable-width minipages
+\@vwid@box=\box57
+\sift@deathcycles=\count282
+\@vwid@loff=\dimen181
+\@vwid@roff=\dimen182
+)
+\c@taburow=\count283
+\tabu@nbcols=\count284
+\tabu@cnt=\count285
+\tabu@Xcol=\count286
+\tabu@alloc=\count287
+\tabu@nested=\count288
+\tabu@target=\dimen183
+\tabu@spreadtarget=\dimen184
+\tabu@naturalX=\dimen185
+\tabucolX=\dimen186
+\tabu@Xsum=\dimen187
+\extrarowdepth=\dimen188
+\abovetabulinesep=\dimen189
+\belowtabulinesep=\dimen190
+\tabustrutrule=\dimen191
+\tabu@thebody=\toks31
+\tabu@footnotes=\toks32
+\tabu@box=\box58
+\tabu@arstrutbox=\box59
+\tabu@hleads=\box60
+\tabu@vleads=\box61
+\tabu@cellskip=\skip63
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/threeparttable/threeparttable.sty
+Package: threeparttable 2003/06/13 v 3.0
+\@tempboxb=\box62
+)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/threeparttablex/threeparttablex.sty
+Package: threeparttablex 2013/07/23 v0.3 by daleif
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/environ/environ.sty
+Package: environ 2014/05/04 v0.3 A new way to define environments
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/trimspaces/trimspaces.sty
+Package: trimspaces 2009/09/17 v1.1 Trim spaces around a token list
+))
+\TPTL@width=\skip64
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/ulem/ulem.sty
+\UL@box=\box63
+\UL@hyphenbox=\box64
+\UL@skip=\skip65
+\UL@hook=\toks33
+\UL@height=\dimen192
+\UL@pe=\count289
+\UL@pixel=\dimen193
+\ULC@box=\box65
+Package: ulem 2019/11/18
+\ULdepth=\dimen194
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/makecell/makecell.sty
+Package: makecell 2009/08/03 V0.1e Managing of Tab Column Heads and Cells
+\rotheadsize=\dimen195
+\c@nlinenum=\count290
+\TeXr@lab=\toks34
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/xcolor/xcolor.sty
+Package: xcolor 2016/05/11 v2.12 LaTeX color extensions (UK)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/graphics-cfg/color.cfg
+File: color.cfg 2016/01/02 v1.6 sample color configuration
+)
+Package xcolor Info: Driver file: pdftex.def on input line 225.
+LaTeX Info: Redefining \color on input line 709.
+Package xcolor Info: Model `cmy' substituted by `cmy0' on input line 1348.
+Package xcolor Info: Model `hsb' substituted by `rgb' on input line 1352.
+Package xcolor Info: Model `RGB' extended on input line 1364.
+Package xcolor Info: Model `HTML' substituted by `rgb' on input line 1366.
+Package xcolor Info: Model `Hsb' substituted by `hsb' on input line 1367.
+Package xcolor Info: Model `tHsb' substituted by `hsb' on input line 1368.
+Package xcolor Info: Model `HSB' substituted by `hsb' on input line 1369.
+Package xcolor Info: Model `Gray' substituted by `gray' on input line 1370.
+Package xcolor Info: Model `wave' substituted by `hsb' on input line 1371.
+) (/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/l3backend/l3backend-pdfmode.def
+File: l3backend-pdfmode.def 2020-06-29 L3 backend support: PDF mode
+\l__kernel_color_stack_int=\count291
+\l__pdf_internal_box=\box66
+) (./cleaningFeederData_allData.aux
+
+LaTeX Warning: Label `tab:bestDaysProp' multiply defined.
+
+
+LaTeX Warning: Label `tab:bestDaysProp' multiply defined.
+
+
+LaTeX Warning: Label `tab:bestDaysProp' multiply defined.
+
+)
+\openout1 = `cleaningFeederData_allData.aux'.
+
+LaTeX Font Info: Checking defaults for OML/cmm/m/it on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for OMS/cmsy/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for OT1/cmr/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for T1/cmr/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for TS1/cmr/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for OMX/cmex/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for U/cmr/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for PD1/pdf/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+LaTeX Font Info: Checking defaults for PU/pdf/m/n on input line 134.
+LaTeX Font Info: ... okay on input line 134.
+\AtBeginShipoutBox=\box67
+Package hyperref Info: Link coloring OFF on input line 134.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/hyperref/nameref.sty
+Package: nameref 2019/09/16 v2.46 Cross-referencing by name of section
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/refcount/refcount.sty
+Package: refcount 2019/12/15 v3.6 Data extraction from label references (HO)
+)
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/generic/gettitlestring/gettitlestring.sty
+Package: gettitlestring 2019/12/15 v1.6 Cleanup title references (HO)
+)
+\c@section@level=\count292
+)
+LaTeX Info: Redefining \ref on input line 134.
+LaTeX Info: Redefining \pageref on input line 134.
+LaTeX Info: Redefining \nameref on input line 134.
+(./cleaningFeederData_allData.out) (./cleaningFeederData_allData.out)
+\@outlinefile=\write4
+\openout4 = `cleaningFeederData_allData.out'.
+
+*geometry* driver: auto-detecting
+*geometry* detected driver: pdftex
+*geometry* verbose mode - [ preamble ] result:
+* driver: pdftex
+* paper: <default>
+* layout: <same size as paper>
+* layoutoffset:(h,v)=(0.0pt,0.0pt)
+* modes:
+* h-part:(L,W,R)=(72.26999pt, 469.75502pt, 72.26999pt)
+* v-part:(T,H,B)=(72.26999pt, 650.43001pt, 72.26999pt)
+* \paperwidth=614.295pt
+* \paperheight=794.96999pt
+* \textwidth=469.75502pt
+* \textheight=650.43001pt
+* \oddsidemargin=0.0pt
+* \evensidemargin=0.0pt
+* \topmargin=-37.0pt
+* \headheight=12.0pt
+* \headsep=25.0pt
+* \topskip=10.0pt
+* \footskip=30.0pt
+* \marginparwidth=65.0pt
+* \marginparsep=11.0pt
+* \columnsep=10.0pt
+* \skip\footins=9.0pt plus 4.0pt minus 2.0pt
+* \hoffset=0.0pt
+* \voffset=0.0pt
+* \mag=1000
+* \@twocolumnfalse
+* \@twosidefalse
+* \@mparswitchfalse
+* \@reversemarginfalse
+* (1in=72.27pt=25.4mm, 1cm=28.453pt)
+
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty
+Package: epstopdf-base 2020-01-24 v2.11 Base part for package epstopdf
+Package epstopdf-base Info: Redefining graphics rule for `.eps' on input line 4
+85.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg
+File: epstopdf-sys.cfg 2010/07/13 v1.3 Configuration of (r)epstopdf for TeX Liv
+e
+))
+LaTeX Font Info: Trying to load font information for OT1+lmr on input line 1
+36.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/ot1lmr.fd
+File: ot1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info: Trying to load font information for OML+lmm on input line 1
+36.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/omllmm.fd
+File: omllmm.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info: Trying to load font information for OMS+lmsy on input line
+136.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/omslmsy.fd
+File: omslmsy.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info: Trying to load font information for OMX+lmex on input line
+136.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/omxlmex.fd
+File: omxlmex.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <12> on input line 136.
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <8> on input line 136.
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <6> on input line 136.
+LaTeX Font Info: Trying to load font information for U+msa on input line 136
+.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsfonts/umsa.fd
+File: umsa.fd 2013/01/14 v3.01 AMS symbols A
+)
+LaTeX Font Info: Trying to load font information for U+msb on input line 136
+.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/amsfonts/umsb.fd
+File: umsb.fd 2013/01/14 v3.01 AMS symbols B
+)
+
+Package xcolor Warning: Incompatible color definition on input line 137.
+
+LaTeX Font Info: Trying to load font information for T1+lmtt on input line 1
+38.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/t1lmtt.fd
+File: t1lmtt.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+LaTeX Font Info: Font shape `T1/lmtt/bx/n' in size <10> not available
+(Font) Font shape `T1/lmtt/b/n' tried instead on input line 140.
+
+Package xcolor Warning: Incompatible color definition on input line 153.
+
+
+Package xcolor Warning: Incompatible color definition on input line 153.
+
+
+Package xcolor Warning: Incompatible color definition on input line 164.
+
+
+Package xcolor Warning: Incompatible color definition on input line 175.
+
+
+Package xcolor Warning: Incompatible color definition on input line 175.
+
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <10> on input line 189.
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <7> on input line 189.
+LaTeX Font Info: External font `lmex10' loaded for size
+(Font) <5> on input line 189.
+
+Package xcolor Warning: Incompatible color definition on input line 199.
+
+[1
+
+{/home/ba1e12/.TinyTeX/texmf-var/fonts/map/pdftex/updmap/pdftex.map}]
+
+Package xcolor Warning: Incompatible color definition on input line 208.
+
+
+Package xcolor Warning: Incompatible color definition on input line 208.
+
+
+Overfull \hbox (179.63316pt too wide) in paragraph at lines 214--231
+ [][]
+ []
+
+
+Package xcolor Warning: Incompatible color definition on input line 236.
+
+
+Package xcolor Warning: Incompatible color definition on input line 252.
+
+
+Package xcolor Warning: Incompatible color definition on input line 252.
+
+
+Package xcolor Warning: Incompatible color definition on input line 264.
+
+[2pdfTeX warning (ext4): destination with the same identifier (name{table.1}) h
+as been already used, duplicate ignored
+
+\AtBegShi@Output ...ipout \box \AtBeginShipoutBox
+ \fi \fi
+l.274 \end{Shaded}
+ ]
+
+Package xcolor Warning: Incompatible color definition on input line 274.
+
+
+Package xcolor Warning: Incompatible color definition on input line 274.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/kwProfiles-1.pdf, id=98, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/kwProfiles-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/kwProfiles-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/kwProfiles-1.pdf used on input line 27
+6.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 286.
+
+
+Package xcolor Warning: Incompatible color definition on input line 298.
+
+
+Package xcolor Warning: Incompatible color definition on input line 298.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/basicCountTile-1.pdf, id=99, 578.16pt x 722.7pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/basicCountTile-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/basicCountTile-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/basicCountTile-1.pdf used on input lin
+e 300.
+(pdftex.def) Requested size: 469.75385pt x 587.1923pt.
+[3 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/kwProfiles-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 309.
+
+[4 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/basicCountTile-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 323.
+
+
+Package xcolor Warning: Incompatible color definition on input line 323.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/aggVisN-1.pdf, id=120, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/aggVisN-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/aggVisN-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/aggVisN-1.pdf used on input line 325.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 332.
+
+
+Package xcolor Warning: Incompatible color definition on input line 343.
+
+
+Package xcolor Warning: Incompatible color definition on input line 343.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/basickWTile-1.pdf, id=121, 578.16pt x 722.7pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/basickWTile-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/basickWTile-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/basickWTile-1.pdf used on input line 3
+45.
+(pdftex.def) Requested size: 469.75385pt x 587.1923pt.
+[5 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/aggVisN-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 351.
+
+[6 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/basickWTile-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 363.
+
+
+Package xcolor Warning: Incompatible color definition on input line 363.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/aggViskW-1.pdf, id=142, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/aggViskW-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/aggViskW-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/aggViskW-1.pdf used on input line 365.
+
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 374.
+
+
+Package xcolor Warning: Incompatible color definition on input line 384.
+
+
+Package xcolor Warning: Incompatible color definition on input line 384.
+
+[7 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/aggViskW-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 408.
+
+
+Package xcolor Warning: Incompatible color definition on input line 415.
+
+
+Package xcolor Warning: Incompatible color definition on input line 415.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/tileFeeders-1.pdf, id=154, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/tileFeeders-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/tileFeeders-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/tileFeeders-1.pdf used on input line 4
+17.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 424.
+
+
+Package xcolor Warning: Incompatible color definition on input line 436.
+
+
+Package xcolor Warning: Incompatible color definition on input line 436.
+
+
+Package xcolor Warning: Incompatible color definition on input line 436.
+
+[8 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/tileFeeders-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 436.
+
+
+Package xcolor Warning: Incompatible color definition on input line 436.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/missingProfiles-1.pdf, id=166, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/missingProfiles-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/missingProfiles-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/missingProfiles-1.pdf used on input li
+ne 438.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 444.
+
+
+Package xcolor Warning: Incompatible color definition on input line 452.
+
+
+Package xcolor Warning: Incompatible color definition on input line 452.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/kWProfiles-1.pdf, id=167, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/kWProfiles-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/kWProfiles-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/kWProfiles-1.pdf used on input line 45
+4.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+[9 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_
+files/figure-latex/missingProfiles-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 462.
+
+
+Package xcolor Warning: Incompatible color definition on input line 470.
+
+
+Package xcolor Warning: Incompatible color definition on input line 470.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/compareProfiles-1.pdf, id=178, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/compareProfiles-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/compareProfiles-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/compareProfiles-1.pdf used on input li
+ne 472.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+[10 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/kWProfiles-1.pdf>]
+LaTeX Font Info: Trying to load font information for TS1+lmr on input line 4
+82.
+(/home/ba1e12/.TinyTeX/texmf-dist/tex/latex/lm/ts1lmr.fd
+File: ts1lmr.fd 2009/10/30 v1.6 Font defs for Latin Modern
+)
+
+Package xcolor Warning: Incompatible color definition on input line 503.
+
+
+Package xcolor Warning: Incompatible color definition on input line 508.
+
+
+Package xcolor Warning: Incompatible color definition on input line 508.
+
+[11 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/compareProfiles-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 544.
+
+
+Package xcolor Warning: Incompatible color definition on input line 563.
+
+
+Package xcolor Warning: Incompatible color definition on input line 563.
+
+[12]
+
+Package xcolor Warning: Incompatible color definition on input line 584.
+
+
+Package xcolor Warning: Incompatible color definition on input line 590.
+
+
+Package xcolor Warning: Incompatible color definition on input line 590.
+
+
+Package xcolor Warning: Incompatible color definition on input line 601.
+
+
+Package xcolor Warning: Incompatible color definition on input line 606.
+
+
+Package xcolor Warning: Incompatible color definition on input line 606.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/bestDaysMean-1.pdf, id=202, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/bestDaysMean-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/bestDaysMean-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/bestDaysMean-1.pdf used on input line
+608.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 617.
+
+
+Package xcolor Warning: Incompatible color definition on input line 624.
+
+
+Package xcolor Warning: Incompatible color definition on input line 624.
+
+</home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData_fil
+es/figure-latex/bestDaysProp-1.pdf, id=203, 578.16pt x 325.215pt>
+File: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allDat
+a_files/figure-latex/bestDaysProp-1.pdf Graphic file (type pdf)
+<use /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/bestDaysProp-1.pdf>
+Package pdftex.def Info: /home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleani
+ngFeederData_allData_files/figure-latex/bestDaysProp-1.pdf used on input line
+626.
+(pdftex.def) Requested size: 469.75385pt x 264.23653pt.
+
+Package xcolor Warning: Incompatible color definition on input line 628.
+
+[13 </home/ba1e12/git.Soton/ba1e12/datacleaning/docs/cleaningFeederData_allData
+_files/figure-latex/bestDaysMean-1.pdf>]
+
+Package xcolor Warning: Incompatible color definition on input line 635.
+
+
+Package xcolor Warning: Incompatible color definition on input line 635.
+
+
+Package xcolor Warning: Incompatible color definition on input line 660.
+
+
+Package xcolor Warning: Incompatible color definition on input line 666.
+
+
+Package xcolor Warning: Incompatible color definition on input line 666.
+
+
+Package xcolor Warning: Incompatible color definition on input line 691.
+
+
+Package xcolor Warning: Incompatible color definition on input line 697.
+
+
+Package xcolor Warning: Incompatible color definition on input line 697.
+
+
+Package xcolor Warning: Incompatible color definition on input line 722.
+
+
+Package xcolor Warning: Incompatible color definition on input line 728.
+
+
+Package xcolor Warning: Incompatible color definition on input line 728.
+
+[14pdfTeX warning (ext4): destination with the same identifier (name{table.2})
+has been already used, duplicate ignored
+
+\AtBegShi@Output ...ipout \box \AtBeginShipoutBox
+ \fi \fi
+l.757 \section
+ {Runtime}\label{runtime} </home/ba1e12/git.Soton/ba1e12/dataclean
+ing/docs/cleaningFeederData_allData_files/figure-latex/bestDaysProp-1.pdf>] [15
+pdfTeX warning (ext4): destination with the same identifier (name{table.3}) has
+ been already used, duplicate ignored
+
+\AtBegShi@Output ...ipout \box \AtBeginShipoutBox
+ \fi \fi
+l.757 \section
+ {Runtime}\label{runtime}pdfTeX warning (ext4): destination with t
+he same identifier (name{table.4}) has been already used, duplicate ignored
+
+\AtBegShi@Output ...ipout \box \AtBeginShipoutBox
+ \fi \fi
+l.757 \section
+ {Runtime}\label{runtime}pdfTeX warning (ext4): destination with t
+he same identifier (name{table.5}) has been already used, duplicate ignored
+
+\AtBegShi@Output ...ipout \box \AtBeginShipoutBox
+ \fi \fi
+l.757 \section
+ {Runtime}\label{runtime}] [16]
+
+Package xcolor Warning: Incompatible color definition on input line 854.
+
+
+Package xcolor Warning: Incompatible color definition on input line 862.
+
+
+Package xcolor Warning: Incompatible color definition on input line 862.
+
+
+Package xcolor Warning: Incompatible color definition on input line 866.
+
+
+Package xcolor Warning: Incompatible color definition on input line 872.
+
+
+Package xcolor Warning: Incompatible color definition on input line 872.
+
+
+Package xcolor Warning: Incompatible color definition on input line 877.
+
+
+Package xcolor Warning: Incompatible color definition on input line 924.
+
+
+Package xcolor Warning: Incompatible color definition on input line 924.
+
+
+Package xcolor Warning: Incompatible color definition on input line 924.
+
+[17]
+
+Package xcolor Warning: Incompatible color definition on input line 924.
+
+
+Package xcolor Warning: Incompatible color definition on input line 924.
+
+
+Package xcolor Warning: Incompatible color definition on input line 928.
+
+
+Package xcolor Warning: Incompatible color definition on input line 932.
+
+
+Package xcolor Warning: Incompatible color definition on input line 932.
+
+
+Package xcolor Warning: Incompatible color definition on input line 934.
+
+
+Package xcolor Warning: Incompatible color definition on input line 947.
+
+
+Package xcolor Warning: Incompatible color definition on input line 947.
+
+
+Package xcolor Warning: Incompatible color definition on input line 949.
+
+[18]
+
+Package xcolor Warning: Incompatible color definition on input line 956.
+
+
+Package xcolor Warning: Incompatible color definition on input line 956.
+
+
+Package xcolor Warning: Incompatible color definition on input line 960.
+
+
+Package xcolor Warning: Incompatible color definition on input line 964.
+
+
+Package xcolor Warning: Incompatible color definition on input line 964.
+
+
+Package xcolor Warning: Incompatible color definition on input line 966.
+
+
+Package xcolor Warning: Incompatible color definition on input line 980.
+
+
+Package xcolor Warning: Incompatible color definition on input line 980.
+
+
+Package xcolor Warning: Incompatible color definition on input line 984.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1001.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1001.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1006.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1056.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1056.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1056.
+
+[19]
+
+Package xcolor Warning: Incompatible color definition on input line 1056.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1056.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1060.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1064.
+
+
+Package xcolor Warning: Incompatible color definition on input line 1064.
+
+[20]
+Package atveryend Info: Empty hook `BeforeClearDocument' on input line 1112.
+[21]
+Package atveryend Info: Empty hook `AfterLastShipout' on input line 1112.
+(./cleaningFeederData_allData.aux)
+Package atveryend Info: Executing hook `AtVeryEndDocument' on input line 1112.
+Package atveryend Info: Executing hook `AtEndAfterFileList' on input line 1112.
+
+Package rerunfilecheck Info: File `cleaningFeederData_allData.out' has not chan
+ged.
+(rerunfilecheck) Checksum: 919355C365AB546FED2F255D26066E68;2438.
+
+LaTeX Warning: There were multiply-defined labels.
+
+Package atveryend Info: Empty hook `AtVeryVeryEnd' on input line 1112.
+ )
+Here is how much of TeX's memory you used:
+ 12880 strings out of 482419
+ 187735 string characters out of 5948430
+ 452933 words of memory out of 5000000
+ 28331 multiletter control sequences out of 15000+600000
+ 583313 words of font info for 68 fonts, out of 8000000 for 9000
+ 14 hyphenation exceptions out of 8191
+ 37i,9n,37p,968b,461s stack positions out of 5000i,500n,10000p,200000b,80000s
+{/home/ba1e12/.TinyTeX/texmf-dist/fonts/enc/dvips/lm/lm-ts1.enc}{/home/ba1e12
+/.TinyTeX/texmf-dist/fonts/enc/dvips/lm/lm-ec.enc}</home/ba1e12/.TinyTeX/texmf-
+dist/fonts/type1/public/lm/lmbx10.pfb></home/ba1e12/.TinyTeX/texmf-dist/fonts/t
+ype1/public/lm/lmbx12.pfb></home/ba1e12/.TinyTeX/texmf-dist/fonts/type1/public/
+lm/lmr10.pfb></home/ba1e12/.TinyTeX/texmf-dist/fonts/type1/public/lm/lmr12.pfb>
+</home/ba1e12/.TinyTeX/texmf-dist/fonts/type1/public/lm/lmr17.pfb></home/ba1e12
+/.TinyTeX/texmf-dist/fonts/type1/public/lm/lmri10.pfb></home/ba1e12/.TinyTeX/te
+xmf-dist/fonts/type1/public/lm/lmtk10.pfb></home/ba1e12/.TinyTeX/texmf-dist/fon
+ts/type1/public/lm/lmtt10.pfb></home/ba1e12/.TinyTeX/texmf-dist/fonts/type1/pub
+lic/lm/lmtti10.pfb>
+Output written on cleaningFeederData_allData.pdf (21 pages, 13892852 bytes).
+PDF statistics:
+ 332 PDF objects out of 1000 (max. 8388607)
+ 270 compressed objects within 3 object streams
+ 57 named destinations out of 1000 (max. 500000)
+ 208 words of extra memory for PDF output out of 10000 (max. 10000000)
+
diff --git a/_drakeCleanFeeders.R b/_drakeCleanFeeders.R
index d914fad43611a6b5a84ddc855076e820725c0381..f58757e31956474fd898a3e4759af9b4323e5b3b 100644
--- a/_drakeCleanFeeders.R
+++ b/_drakeCleanFeeders.R
@@ -1,12 +1,15 @@
# basic _drake.R style file
-# but adapted for use in a project with multiple plans
+# but adapted for use in a project where there might be multiple plans in the same folder
+# called using r_make() from make_cleanFeeders.R
+# see https://books.ropensci.org/drake/projects.html#usage for explanation
+
+# Libraries/Packages ----
+# the drake book suggests putting this in packages.R but...
-# Libraries ----
library(dataCleaning) # remember to build it first :-)
dataCleaning::setup() # load env.R set up the default paths etc
makeLibs <- c("data.table", # data munching
- "drake", # for plans
"here", # here
"lubridate", # dates and times
"hms", # times
@@ -17,7 +20,7 @@ makeLibs <- c("data.table", # data munching
dataCleaning::loadLibraries(makeLibs)
# Parameters ----
-updateData <- "yep" # edit this in any way (at all) to get drake to re-load the data
+updateData <- "rerun" # edit this in any way (at all) to get drake to re-load the data
updateReport <- "yes" # edit this in any way (at all) to get drake to re-load the data
# Some data to play with:
@@ -32,6 +35,7 @@ authors <- "Ben Anderson & Ellis Ridett"
# Functions ----
# for use in drake
+# the drake book suggests putting this in functions.R but...
addSeason <- function(dt,dateVar,h){
dt <- dt[, tmpM := lubridate::month(get(dateVar))] # sets 1 (Jan) - 12 (Dec). May already exist but we can't rely on it
@@ -79,7 +83,10 @@ getData <- function(f,updateData){
makeUniq <- function(dt){
# we suspect there may be duplicates by feeder_ID, dateTime & kW
# remove them (report this in the .Rmd)
- uniq <- unique(dt, by = c("rDateTime", "feeder_ID", "kW"))
+ uniq <- unique(dt, by = c("rDateTime", # dateTime
+ "feeder_ID", # our constructed unique feeded ID
+ "kW") # kW
+ )
return(uniq)
}
@@ -150,6 +157,8 @@ makeReport <- function(f,version, type = "html", updateReport){
# Set the drake plan ----
+# the drake book suggests putting this in plan.R but...
+# I had expected r_make() to load drake() in the new clean R session but it doesn't
my_plan <- drake::drake_plan(
origData = getData(dFile, updateData), # returns data as data.table. If you edit 'update' in any way it will reload - drake is watching you!
uniqData = makeUniq(origData), # remove duplicates
@@ -162,4 +171,5 @@ my_plan <- drake::drake_plan(
)
# see https://books.ropensci.org/drake/projects.html#usage
-drake_config(my_plan, verbose = 2)
\ No newline at end of file
+# I had expected r_make() to load drake() in the new clean R session but it doesn't
+drake::drake_config(my_plan, verbose = 2)
\ No newline at end of file
diff --git a/docs/cleaningFeederData_allData.html b/docs/cleaningFeederData_allData.html
index 92aca205c594e41143da788420897e439d825ad7..955d5694fa48e53065a3d23ad4cf159368a3e7f8 100644
--- a/docs/cleaningFeederData_allData.html
+++ b/docs/cleaningFeederData_allData.html
@@ -181,7 +181,7 @@ summary {
<h1 class="title toc-ignore">Testing electricity substation/feeder data</h1>
<h3 class="subtitle">Outliers and missing data...</h3>
<h4 class="author">Ben Anderson & Ellis Ridett</h4>
-<h4 class="date">Last run at: 2020-07-09 00:56:06</h4>
+<h4 class="date">Last run at: 2020-07-09 09:48:01</h4>
</div>
@@ -214,7 +214,7 @@ dataCleaning::loadLibraries(rmdLibs)</code></pre>
<h1>Intro</h1>
<p>We have some electricity substation feeder data that has been cleaned to give mean kW per 15 minutes.</p>
<p>There seem to be some NA kW values and a lot of missing time stamps. We want to select the 'best' (i.e most complete) days within a day-of-the-week/season/year sampling frame. If we can't do that we may have to resort to seasonal mean kW profiles by hour & day of the week...</p>
-<p>Code used to generate this report: <a href="https://git.soton.ac.uk/ba1e12/spatialec/-/blob/master/isleOfWight/cleaningFeederData.Rmd" class="uri">https://git.soton.ac.uk/ba1e12/spatialec/-/blob/master/isleOfWight/cleaningFeederData.Rmd</a></p>
+<p>The code used to generate this report is in: <a href="https://git.soton.ac.uk/ba1e12/dataCleaning/Rmd/" class="uri">https://git.soton.ac.uk/ba1e12/dataCleaning/Rmd/</a></p>
</div>
<div id="data-prep" class="section level1">
<h1>Data prep</h1>
@@ -226,11 +226,11 @@ dataCleaning::loadLibraries(rmdLibs)</code></pre>
uniqDataDT <- drake::readd(uniqData) # readd the drake object
kableExtra::kable(head(origDataDT), digits = 2,
- caption = "Counts per feeder (long table)") %>%
+ caption = "First 6 rows of data") %>%
kable_styling()</code></pre>
<table class="table" style="margin-left: auto; margin-right: auto;">
<caption>
-Counts per feeder (long table)
+First 6 rows of data
</caption>
<thead>
<tr>
@@ -487,14 +487,16 @@ Winter
message("Unique data nrows: ", tidyNum(nrow(uniqDataDT)))
-message("So we have ", tidyNum(nrow(origDataDT) - nrow(uniqDataDT)), " duplicates...")
+nDups <- tidyNum(nrow(origDataDT) - nrow(uniqDataDT))
+
+message("So we have ", tidyNum(nDups), " duplicates...")
pc <- 100*((nrow(origDataDT) - nrow(uniqDataDT))/nrow(origDataDT))
message("That's ", round(pc,2), "%")
feederDT <- uniqDataDT[!is.na(rDateTime)] # use dt with no duplicates
origDataDT <- NULL # save memory</code></pre>
-<p>There were duplicates - that's 0.38 % of the observations loaded.</p>
+<p>There were 83,606 duplicates - that's ~ 0.38 % of the observations loaded.</p>
<p>So we remove the duplicates...</p>
</div>
</div>
@@ -1500,7 +1502,7 @@ Fri
</div>
<div id="runtime" class="section level1">
<h1>Runtime</h1>
-<p>Analysis completed in 196.02 seconds ( 3.27 minutes) using <a href="https://cran.r-project.org/package=knitr">knitr</a> in <a href="http://www.rstudio.com">RStudio</a> with R version 3.6.0 (2019-04-26) running on x86_64-redhat-linux-gnu.</p>
+<p>Analysis completed in 218.48 seconds ( 3.64 minutes) using <a href="https://cran.r-project.org/package=knitr">knitr</a> in <a href="http://www.rstudio.com">RStudio</a> with R version 3.6.0 (2019-04-26) running on x86_64-redhat-linux-gnu.</p>
</div>
<div id="r-environment" class="section level1">
<h1>R environment</h1>
@@ -1539,8 +1541,8 @@ Fri
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
-## [1] kableExtra_1.1.0 skimr_2.1.2 ggplot2_3.3.2 hms_0.5.3
-## [5] lubridate_1.7.9 here_0.1 drake_7.12.4 data.table_1.12.0
+## [1] kableExtra_1.1.0 drake_7.12.4 skimr_2.1.2 ggplot2_3.3.2
+## [5] hms_0.5.3 lubridate_1.7.9 here_0.1 data.table_1.12.0
## [9] dataCleaning_0.1.0
##
## loaded via a namespace (and not attached):
diff --git a/docs/cleaningFeederData_allData.pdf b/docs/cleaningFeederData_allData.pdf
index 6268d3469501440f1ee512c5b142717b1624b0ef..f35c9dcc5876b5611e31e47b0cd6875d89d891bb 100644
Binary files a/docs/cleaningFeederData_allData.pdf and b/docs/cleaningFeederData_allData.pdf differ
diff --git a/make_cleanFeeders.R b/make_cleanFeeders.R
index fbf91a625f79c9e8031f9396b045b2fbc14ae1bb..11702eab02002bb7b61c9e4129df538f7f2a28f2 100644
--- a/make_cleanFeeders.R
+++ b/make_cleanFeeders.R
@@ -4,9 +4,9 @@
# Set up ----
startTime <- proc.time()
-library(drake)
# use r_make to run the plan inside a clean R session so nothing gets contaminated
drake::r_make(source = "_drakeCleanFeeders.R") # where we keep the drake plan etc
+
# we don't keep this in /R because that's where the package functions live
# we don't use "_drake.R" because we have lots of different plans