A place to store hints, tips and examples for data cleaning. We use a lot of very dirty data which often has outliers and missing observations. Since most of this data is large scale 'sensor' data with time stamps we make a lot of use of these R packages to process and visualise the data so we can see what is odd and what is missing:
*[ggplot2](https://ggplot2.tidyverse.org/)'s[geom_tile()](https://ggplot2.tidyverse.org/reference/geom_tile.html) with time of day on the date on the x axis, time on the y and 'fill' set to the sensor value that _should_ be there. This shows up non-random (and random) data holes like [these](https://git.soton.ac.uk/SERG/datacleaning/-/blob/master/rmd/cleaningFeederData_files/figure-latex/missingVis-1.pdf) very nicely.
*[data.table](https://rdatatable.gitlab.io/data.table/) - very fast data loading and wrangling
*[lubridate](https://lubridate.tidyverse.org/) - _the_ way to do dates and dateTimes in R
*[hms](https://hms.tidyverse.org/) - deals with time (HH:MM:SS)
*[ggplot2](https://ggplot2.tidyverse.org/) - plots, especially using[geom_tile()](https://ggplot2.tidyverse.org/reference/geom_tile.html) with time of day on the date on the x axis, time on the y and 'fill' set to the sensor value that _should_ be there. This shows up non-random (and random) data holes like [these](https://git.soton.ac.uk/SERG/datacleaning/-/blob/master/rmd/cleaningFeederData_files/figure-latex/missingVis-1.pdf) very nicely.
