IoW stub & initial BEIS data load

isleOfWight/beisIoWData.Rmd

+---
+title: 'Exploring Isle of Wight Data'
+subtitle: 'Code and notes'
+author: "Ben Anderson (b.anderson@soton.ac.uk), [SERG, Energy & Climate Change](http://www.energy.soton.ac.uk/), University of Southampton"
+date: 'Last run at: `r Sys.time()`'
+output:
+  bookdown::html_document2:
+    code_folding: hide
+    fig_caption: yes
+    number_sections: yes
+    self_contained: no
+    toc: yes
+    toc_depth: 3
+    toc_float: yes
+  bookdown::pdf_document2:
+    fig_caption: yes
+    keep_tex: yes
+    number_sections: yes
+    toc: yes
+    toc_depth: 2
+  bookdown::word_document2:
+    fig_caption: yes
+    toc: yes
+    toc_depth: 2
+always_allow_html: yes
+bibliography: '`r path.expand("~/bibliography.bib")`'
+---
+
+```{r setup, include=FALSE}
+# Knitr setup ----
+knitr::opts_chunk$set(echo = TRUE)
+knitr::opts_chunk$set(warning = FALSE) # for final tidy run
+knitr::opts_chunk$set(message = FALSE) # for final tidy run
+
+# Set start time ----
+startTime <- proc.time()
+
+# Libraries ----
+
+library(data.table) # cos we like data.table (you may not in which case dplyr is fine :-)
+library(lubridate) # for data & time manip
+library(hms) # for hh:mm:ss if we need it
+library(ggplot2) # fancy plots
+library(kableExtra) # for extra kable
+library(skimr) # for skim (data description)
+library(XML)
+
+# Parameters ----
+dPath <- "~/Data" # edit for your set up
+
+gisPath <- paste0(dPath, "/")
+
+censusLutsPath <- paste0(dPath, "/UK_Census/Census_LUTS/2011/EngWales/OA11_LSOA11_MSOA11_LAD11_EW_LU.zip Folder")
+
+oaCensusPath <- paste0(dPath, "/UK_Census/2011Data/EW/oa/processed")
+lsoaCensusPath <- paste0(dPath, "/UK_Census/2011Data/EW/lsoa/processed")
+oacPath <- paste0(dPath, "/UK OAC/2011/CDRC/Tables")
+beisPath <- paste0(dPath, "/DECC/SubnationalEnergyData/2010-2018")
+```
+
+
+# Intro
+
+Developing a model of residential electricity demand for IoW.
+
+# Data prep
+## Load spine of OAs and filter to IoW
+
+```{r loadOA}
+oaSpine <- paste0(censusLutsPath, "/OA11_LSOA11_MSOA11_LAD11_EW_LUv2.csv")
+dtOA <- data.table::fread(oaSpine)
+head(dtOA)
+
+dtOA_IoW <- dtOA[LAD11NM %like% "Wight"]
+
+```
+
+How many OAs, LSOAs etc?
+
+
+```{r lsoaTables}
+t <- dtOA_IoW[, .(nOAs = uniqueN(OA11CD),
+                  nLSOAs = uniqueN(LSO11ANM),
+                  nMSOAs = uniqueN(MSOA11NM)), keyby = .(LAD11NM)]
+
+kableExtra::kable(t, caption = "Number of LSOAs & MSOAs")
+```
+
+This is important because various datasets can be linked at these levels:
+
+ * OA
+    * urban/rural indicator
+ * LSOA
+   * BEIS domestic gas/elec (annual kWh)
+ * MSOA
+   * BEIS non-domestic gas/elec (annual kWh)
+
+SSEN datasets for the LV network could be at about MSOA level
+
+## Add elec/gas
+
+LSOA level - domestic
+```{r loadBeisLSOA}
+dtLsoaElec <- data.table::fread(paste0(beisPath, "/LSOA_DOM_ELEC_csv/LSOA_ELEC_2018.csv"))
+# to prevent confusion
+dtLsoaElec[, nDElecMeters := METERS]
+dtLsoaElec[, kwhDElec := KWH]
+dtLsoaElec[, kwhDMeanElec := MEAN]
+dtLsoaElec[, kwhDMedianElec := MEDIAN]
+
+
+dtLsoaGas <- data.table::fread(paste0(beisPath, "/LSOA_DOM_GAS_csv/LSOA_GAS_2018.csv"))
+dtLsoaGas[, nDGasMeters := METERS]
+dtLsoaGas[, kwhDGas := KWH]
+dtLsoaGas[, kwhDMeanGas := MEAN]
+dtLsoaGas[, kwhDMedianGas := MEDIAN]
+
+```
+
+Now MSOA level - domestic & non-domestic
+
+```{r loadBeisMSOA}
+dtMsoaDElec <- data.table::fread(paste0(beisPath, "/MSOA_DOM_ELEC_csv/MSOA_ELEC_2018.csv"))
+
+dtMsoaDElec[, nDElecMeters := METERS]
+dtMsoaDElec[, kwhDElec := KWH]
+dtMsoaDElec[, kwhDMeanElec := MEAN]
+dtMsoaDElec[, kwhDMedianElec := MEDIAN]
+
+
+dtMsoaDGas <- data.table::fread(paste0(beisPath, "/MSOA_DOM_GAS_csv/MSOA_GAS_2018.csv"))
+dtMsoaDGas[, nDGasMeters := METERS]
+dtMsoaDGas[, kwhDGas := KWH]
+dtMsoaDGas[, kwhDMeanGas := MEAN]
+dtMsoaDGas[, kwhDMedianGas := MEDIAN]
+
+
+dtMsoaNDElec <- data.table::fread(paste0(beisPath, "/MSOA_ND_ELEC_csv/MSOA_NONDOM_ELEC_2018.csv"))
+
+dtMsoaNDElec[, nNDElecMeters := METERS]
+dtMsoaNDElec[, kwhNDElec := KWH]
+dtMsoaNDElec[, kwhNDMeanElec := MEAN]
+dtMsoaNDElec[, kwhNDMedianElec := MEDIAN]
+
+
+dtMsoaNDGas <- data.table::fread(paste0(beisPath, "/MSOA_ND_GAS_csv/MSOA_NONDOM_GAS_2018.csv"))
+dtMsoaNDGas[, nNDGasMeters := METERS]
+dtMsoaNDGas[, kwhNDGas := KWH]
+dtMsoaNDGas[, kwhNDMeanGas := MEAN]
+dtMsoaNDGas[, kwhNDMedianGas := MEDIAN]
+
+```
+
+If we wanted to include non-domestic we'd need towork at the MSOA level.
+
+For now what we want to do is flag the MSOAs where most energy demand is residential - we can then use these for validation if we can get smart meter or SSEN data for roughly those areas.
+
+```{r linkBeisMSOA}
+setkey(dtMsoaDElec, MSOACode)
+setkey(dtMsoaNDElec, MSOACode)
+setkey(dtMsoaDGas, MSOACode)
+setkey(dtMsoaNDGas, MSOACode)
+
+dtMsoaSpine <- dtMsoaDElec[dtMsoaDGas]
+dtMsoaSpine <- dtMsoaSpine[dtMsoaDGas]
+dtMsoaSpine <- dtMsoaSpine[dtMsoaNDGas]
+dtMsoaSpine <- dtMsoaSpine[dtMsoaNDElec]
+dtMsoaSpine[, elecRatio := kwhDElec/(kwhDElec + kwhNDElec)]
+dtMsoaSpine[, gasRatio := kwhDGas/(kwhNDGas + kwhDGas)]
+
+dtMsoaSpineIoW <- dtMsoaSpine[MSOAName %like% "Wight"]
+t <- dtMsoaSpineIoW[order(-elecRatio)]
+kableExtra::kable(head(t[, .(MSOAName, MSOACode, 
+                                       elecRatio, kwhDElec, kwhNDElec,
+                             gasRatio, kwhDGas, kwhNDGas)],10)) %>%
+  kable_styling()
+
+```
+
+# Annexes
+
+## Beis data
+
+### LSOA
+Domestic elec
+```{r skimLSOAElec}
+# Elec
+skimr::skim(dtLsoaElec)
+```
+
+Domestic gas
+```{r skimLSOAGas}
+# Elec
+skimr::skim(dtLsoaGas)
+```
+
+### MSOA
+Domestic elec
+```{r skimMSOADElec}
+# Elec
+skimr::skim(dtMsoaDElec)
+```
+
+Domestic gas
+```{r skimMSOADGas}
+# gas
+skimr::skim(dtMsoaDGas)
+```
+
+Non-domestic elec
+```{r skimMSOANDElec}
+# gas
+skimr::skim(dtMsoaNDElec)
+```
+
+Non-domestic gas
+```{r skimMSOANDGas}
+# gas
+skimr::skim(dtMsoaNDGas)
+```
+
+# Runtime
+
+```{r check runtime, include=FALSE}
+t <- proc.time() - startTime
+elapsed <- t[[3]]
+```
+
+Analysis completed in `r round(elapsed,2)` seconds ( `r round(elapsed/60,2)` minutes) using [knitr](https://cran.r-project.org/package=knitr) in [RStudio](http://www.rstudio.com) with `r R.version.string` running on `r R.version$platform`.
+
+# R environment
+
+## R packages used
+
+ * base R [@baseR]
+ * bookdown [@bookdown]
+ * data.table [@data.table]
+ * ggplot2 [@ggplot2]
+ * kableExtra [@kableExtra]
+ * knitr [@knitr]
+ * lubridate [@lubridate]
+ * rmarkdown [@rmarkdown]
+ * skimr [@skimr]
+ * XML [@XML]
+
+## Session info
+
+```{r sessionInfo, echo=FALSE}
+sessionInfo()
+```
+
+# References
+