Skip to content
Snippets Groups Projects
Commit 6e6c1515 authored by Ben Anderson's avatar Ben Anderson
Browse files

IoW stub & initial BEIS data load

parent 2487a041
Branches
No related tags found
No related merge requests found
Showing
with 12026 additions and 0 deletions
---
title: 'Exploring Isle of Wight Data'
subtitle: 'Code and notes'
author: "Ben Anderson (b.anderson@soton.ac.uk), [SERG, Energy & Climate Change](http://www.energy.soton.ac.uk/), University of Southampton"
date: 'Last run at: `r Sys.time()`'
output:
bookdown::html_document2:
code_folding: hide
fig_caption: yes
number_sections: yes
self_contained: no
toc: yes
toc_depth: 3
toc_float: yes
bookdown::pdf_document2:
fig_caption: yes
keep_tex: yes
number_sections: yes
toc: yes
toc_depth: 2
bookdown::word_document2:
fig_caption: yes
toc: yes
toc_depth: 2
always_allow_html: yes
bibliography: '`r path.expand("~/bibliography.bib")`'
---
```{r setup, include=FALSE}
# Knitr setup ----
knitr::opts_chunk$set(echo = TRUE)
knitr::opts_chunk$set(warning = FALSE) # for final tidy run
knitr::opts_chunk$set(message = FALSE) # for final tidy run
# Set start time ----
startTime <- proc.time()
# Libraries ----
library(data.table) # cos we like data.table (you may not in which case dplyr is fine :-)
library(lubridate) # for data & time manip
library(hms) # for hh:mm:ss if we need it
library(ggplot2) # fancy plots
library(kableExtra) # for extra kable
library(skimr) # for skim (data description)
library(XML)
# Parameters ----
dPath <- "~/Data" # edit for your set up
gisPath <- paste0(dPath, "/")
censusLutsPath <- paste0(dPath, "/UK_Census/Census_LUTS/2011/EngWales/OA11_LSOA11_MSOA11_LAD11_EW_LU.zip Folder")
oaCensusPath <- paste0(dPath, "/UK_Census/2011Data/EW/oa/processed")
lsoaCensusPath <- paste0(dPath, "/UK_Census/2011Data/EW/lsoa/processed")
oacPath <- paste0(dPath, "/UK OAC/2011/CDRC/Tables")
beisPath <- paste0(dPath, "/DECC/SubnationalEnergyData/2010-2018")
```
# Intro
Developing a model of residential electricity demand for IoW.
# Data prep
## Load spine of OAs and filter to IoW
```{r loadOA}
oaSpine <- paste0(censusLutsPath, "/OA11_LSOA11_MSOA11_LAD11_EW_LUv2.csv")
dtOA <- data.table::fread(oaSpine)
head(dtOA)
dtOA_IoW <- dtOA[LAD11NM %like% "Wight"]
```
How many OAs, LSOAs etc?
```{r lsoaTables}
t <- dtOA_IoW[, .(nOAs = uniqueN(OA11CD),
nLSOAs = uniqueN(LSO11ANM),
nMSOAs = uniqueN(MSOA11NM)), keyby = .(LAD11NM)]
kableExtra::kable(t, caption = "Number of LSOAs & MSOAs")
```
This is important because various datasets can be linked at these levels:
* OA
* urban/rural indicator
* LSOA
* BEIS domestic gas/elec (annual kWh)
* MSOA
* BEIS non-domestic gas/elec (annual kWh)
SSEN datasets for the LV network could be at about MSOA level
## Add elec/gas
LSOA level - domestic
```{r loadBeisLSOA}
dtLsoaElec <- data.table::fread(paste0(beisPath, "/LSOA_DOM_ELEC_csv/LSOA_ELEC_2018.csv"))
# to prevent confusion
dtLsoaElec[, nDElecMeters := METERS]
dtLsoaElec[, kwhDElec := KWH]
dtLsoaElec[, kwhDMeanElec := MEAN]
dtLsoaElec[, kwhDMedianElec := MEDIAN]
dtLsoaGas <- data.table::fread(paste0(beisPath, "/LSOA_DOM_GAS_csv/LSOA_GAS_2018.csv"))
dtLsoaGas[, nDGasMeters := METERS]
dtLsoaGas[, kwhDGas := KWH]
dtLsoaGas[, kwhDMeanGas := MEAN]
dtLsoaGas[, kwhDMedianGas := MEDIAN]
```
Now MSOA level - domestic & non-domestic
```{r loadBeisMSOA}
dtMsoaDElec <- data.table::fread(paste0(beisPath, "/MSOA_DOM_ELEC_csv/MSOA_ELEC_2018.csv"))
dtMsoaDElec[, nDElecMeters := METERS]
dtMsoaDElec[, kwhDElec := KWH]
dtMsoaDElec[, kwhDMeanElec := MEAN]
dtMsoaDElec[, kwhDMedianElec := MEDIAN]
dtMsoaDGas <- data.table::fread(paste0(beisPath, "/MSOA_DOM_GAS_csv/MSOA_GAS_2018.csv"))
dtMsoaDGas[, nDGasMeters := METERS]
dtMsoaDGas[, kwhDGas := KWH]
dtMsoaDGas[, kwhDMeanGas := MEAN]
dtMsoaDGas[, kwhDMedianGas := MEDIAN]
dtMsoaNDElec <- data.table::fread(paste0(beisPath, "/MSOA_ND_ELEC_csv/MSOA_NONDOM_ELEC_2018.csv"))
dtMsoaNDElec[, nNDElecMeters := METERS]
dtMsoaNDElec[, kwhNDElec := KWH]
dtMsoaNDElec[, kwhNDMeanElec := MEAN]
dtMsoaNDElec[, kwhNDMedianElec := MEDIAN]
dtMsoaNDGas <- data.table::fread(paste0(beisPath, "/MSOA_ND_GAS_csv/MSOA_NONDOM_GAS_2018.csv"))
dtMsoaNDGas[, nNDGasMeters := METERS]
dtMsoaNDGas[, kwhNDGas := KWH]
dtMsoaNDGas[, kwhNDMeanGas := MEAN]
dtMsoaNDGas[, kwhNDMedianGas := MEDIAN]
```
If we wanted to include non-domestic we'd need towork at the MSOA level.
For now what we want to do is flag the MSOAs where most energy demand is residential - we can then use these for validation if we can get smart meter or SSEN data for roughly those areas.
```{r linkBeisMSOA}
setkey(dtMsoaDElec, MSOACode)
setkey(dtMsoaNDElec, MSOACode)
setkey(dtMsoaDGas, MSOACode)
setkey(dtMsoaNDGas, MSOACode)
dtMsoaSpine <- dtMsoaDElec[dtMsoaDGas]
dtMsoaSpine <- dtMsoaSpine[dtMsoaDGas]
dtMsoaSpine <- dtMsoaSpine[dtMsoaNDGas]
dtMsoaSpine <- dtMsoaSpine[dtMsoaNDElec]
dtMsoaSpine[, elecRatio := kwhDElec/(kwhDElec + kwhNDElec)]
dtMsoaSpine[, gasRatio := kwhDGas/(kwhNDGas + kwhDGas)]
dtMsoaSpineIoW <- dtMsoaSpine[MSOAName %like% "Wight"]
t <- dtMsoaSpineIoW[order(-elecRatio)]
kableExtra::kable(head(t[, .(MSOAName, MSOACode,
elecRatio, kwhDElec, kwhNDElec,
gasRatio, kwhDGas, kwhNDGas)],10)) %>%
kable_styling()
```
# Annexes
## Beis data
### LSOA
Domestic elec
```{r skimLSOAElec}
# Elec
skimr::skim(dtLsoaElec)
```
Domestic gas
```{r skimLSOAGas}
# Elec
skimr::skim(dtLsoaGas)
```
### MSOA
Domestic elec
```{r skimMSOADElec}
# Elec
skimr::skim(dtMsoaDElec)
```
Domestic gas
```{r skimMSOADGas}
# gas
skimr::skim(dtMsoaDGas)
```
Non-domestic elec
```{r skimMSOANDElec}
# gas
skimr::skim(dtMsoaNDElec)
```
Non-domestic gas
```{r skimMSOANDGas}
# gas
skimr::skim(dtMsoaNDGas)
```
# Runtime
```{r check runtime, include=FALSE}
t <- proc.time() - startTime
elapsed <- t[[3]]
```
Analysis completed in `r round(elapsed,2)` seconds ( `r round(elapsed/60,2)` minutes) using [knitr](https://cran.r-project.org/package=knitr) in [RStudio](http://www.rstudio.com) with `r R.version.string` running on `r R.version$platform`.
# R environment
## R packages used
* base R [@baseR]
* bookdown [@bookdown]
* data.table [@data.table]
* ggplot2 [@ggplot2]
* kableExtra [@kableExtra]
* knitr [@knitr]
* lubridate [@lubridate]
* rmarkdown [@rmarkdown]
* skimr [@skimr]
* XML [@XML]
## Session info
```{r sessionInfo, echo=FALSE}
sessionInfo()
```
# References
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
File added
File added
File added
File added
File added
File added
File added
File added
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment