1 Report Purpose

To use:

  • NZ Census 2013 data (from ~/Data/NZ_Census/data/) and
  • NZ GREENGrid household power demand data (from (Anderson et al. 2018))

To develop initial local area estimates of temporal (half-hourly) power demand in mesh blocks and/or unit areas in the Hawkes Bay and Taranaki areas. These areas have been chosen as they match the original sampling sites for the NZ GREENGrid household power data.

2 Abstract

testing ipf

3 Load data

2013 NZ Census data from NZ Stats at area unit level (pre-processed long form file).

## Loading ~/Data/NZ_Census/data//processed/2013IpfInput.csv
## Loading ~/Data/NZ_GREENGrid/safe/survey/ggHouseholdAttributesSafe_2019-10-20.csv.gz
## Loading ~/Data/NZ_GREENGrid/safe/survey/ggIpfInput.csv
## Loading ~/Data/NZ_GREENGrid/safe/ipf/nonZeroWeightsAu2013.csv
## Filtered areas to 142 areas from 2 regions

Load area labels

We focus on households/families/dwellings not individuals as the spatial microsimulation will operate at the household level.

Table 3.1: First few rows/columns of ipf weights table
AU2013_code AU2013_label REGC2013_label nMBs linkID ipfWeight
541503 Taharua Hawke’s Bay Region 6 rf_06 0.0293534
541503 Taharua Hawke’s Bay Region 6 rf_07 0.1578527
541503 Taharua Hawke’s Bay Region 6 rf_08 0.0293534
541503 Taharua Hawke’s Bay Region 6 rf_09 0.0000000
541503 Taharua Hawke’s Bay Region 6 rf_10 0.0000000
541503 Taharua Hawke’s Bay Region 6 rf_12 1.7374338
Table 3.1: Summary of ipf weights table
AU2013_code AU2013_label REGC2013_label nMBs linkID ipfWeight
Min. :541503 Length:5964 Length:5964 Min. : 2 Length:5964 Min. : 0.0
1st Qu.:545913 Class :character Class :character 1st Qu.:11 Class :character 1st Qu.: 2.8
Median :548916 Mode :character Mode :character Median :21 Mode :character Median : 8.0
Mean :549371 NA NA Mean :24 NA Mean : 16.9
3rd Qu.:552100 NA NA 3rd Qu.:35 NA 3rd Qu.: 16.8
Max. :554800 NA NA Max. :79 NA Max. :545.7 
## Dimensions of ipf weights table: 5,964 rows x 6 cols

The IPF process produces a long form data file of nrow = n(linkID) x n(areaCode) (!) so that linkIDs are repeated and the weights are held in a single column (variable). This makes everything else a lot easier later. The dataset comprises 42 GREENGrid households replicated (with weights) across 0 unit areas.

We now need to add the survey-based attributes back (from the GREENGrid survey).

Table 3.2: N bedrooms vs n people (all households, all areas selected, weighted ipf results)
0 1
0 72093.08 15026.72
1 2543.92 10923.28

4 Test IPF outcomes

## Calculating weighted household counts - can take some time...
Table 4.1: Summary of weights by region
REGC2013_label nGGHouseholds meanWeight minWeight maxWeight
Hawke’s Bay Region 42 17.58 0 545.71
Taranaki Region 42 16.00 0 398.45

5 Test household counts

Figure @ref(fig:checkTotalCounts_nPeople) shows the actual and simulated number of households using the nPeople constraint.

(#tab:checkTotalCounts_nPeople)Summary of n households (n people)
REGC2013_label mean_nHHs min_nHHs max_nHHs
Hawke’s Bay Region 738.8846 24 2730
Taranaki Region 672.3750 36 2025

Figure @ref(fig:checkTotalCounts_nRooms) shows the actual and simulated number of households using the nRooms constraint. This was the penultimate constraint to be fitted.

(#tab:checkTotalCounts_nRooms)Summary of n households (n rooms)
REGC2013_label mean_nHHs min_nHHs max_nHHs
Hawke’s Bay Region 743.3462 24 2733
Taranaki Region 675.2812 33 2031

Can’t repeat for n kids as the base doesn’t match

Figure @ref(fig:checkTotalCounts_ggHHs) shows the number of GREEN Grid households who are being ‘upweighted’ to match the given number of census households.

6 Constrained variables

Table ?? shows the weighted household counts for the number of people. This was the last constraint so it fitted last.

Table ?? shows the weighted household counts for the number of rooms This was the penultimate constraint to be fitted.

Table ?? shows the weighted household counts for the presence of children.

7 Unconstrained variables

Table ?? shows the weighted household counts for main heat source. Remember that this was not used in the final constraints.

8 Data Annexe

8.1 Weights at AU level

Table 8.1: Summary of weights by UA)
AU2013_code nGGHouseholds meanWeight minWeight maxWeight
541503 42 0.5000000 0.0000000 5.805419
545201 42 2.0000000 0.1278216 10.687572
545202 42 2.2857143 0.1814222 18.046863
545204 42 6.0000000 0.0379763 71.654177
545205 42 2.5000000 0.2698154 15.049122
545301 42 5.7142857 0.3190087 47.867673
545302 42 6.3571429 0.7432360 33.160485
545303 42 2.2857143 0.2584516 8.949721
545304 42 7.1428571 0.3204557 68.871228
545500 42 36.2857143 8.6859523 142.506475
545611 42 17.6428571 0.8943544 187.487578
545621 42 16.0000000 0.0743253 289.229621
545631 42 14.6428571 0.4292126 191.376602
545632 42 3.0000000 0.2541373 21.900286
545710 42 8.7857143 0.1839310 88.373340
545721 42 12.4285714 0.7733584 110.452948
545722 42 6.5000000 0.5901935 30.120612
545730 42 15.6428571 1.7681027 108.438648
545740 42 20.8571429 1.5034442 158.793991
545750 42 4.7142857 0.0479715 60.765119
545761 42 3.7857143 0.0915019 45.571166
545762 42 8.2857143 0.1098840 93.049681
545811 42 6.6428571 0.3802246 62.539453
545812 42 6.1428571 0.2038960 81.949747
545821 42 22.9285714 1.0621912 264.750708
545822 42 2.9285714 0.0183115 19.182844
545831 42 3.2142857 0.1198888 33.830123
545832 42 4.3571429 0.3202752 48.039210
545841 42 10.7142857 0.8056353 110.066872
545842 42 6.0714286 0.4649196 49.742280
545851 42 11.3571429 0.3919505 138.393140
545852 42 5.5714286 0.1612011 62.312079
545860 42 9.7142857 0.3432163 104.602012
545911 42 5.5714286 0.2167543 58.516648
545912 42 3.3571429 0.1859009 28.249761
545913 42 2.2142857 0.0712175 27.397306
546100 42 11.5714286 0.3179598 112.883136
546200 42 12.0714286 0.6544744 62.041584
546300 42 13.2857143 1.2354339 80.512843
546400 42 3.0000000 0.1570479 21.600605
546500 42 38.0000000 8.5415421 112.325492
546600 42 45.2142857 9.6755142 155.843505
546700 42 22.9285714 6.6703246 61.490427
546801 42 27.2142857 0.7062207 314.751120
546802 42 26.7857143 0.6130388 322.242631
546901 42 22.4285714 3.7186545 82.895412
546902 42 23.2142857 3.5098849 83.203674
547001 42 23.5714286 7.7167005 75.224984
547002 42 24.2142857 3.1796031 121.206237
547100 42 64.7857143 3.5737949 545.709700
547200 42 51.1428571 4.0840254 353.524281
547300 42 51.0714286 3.5402755 381.129083
547400 42 31.9285714 6.5231505 112.262142
547600 42 31.7857143 5.9704310 115.826362
547700 42 27.0714286 4.9025320 100.267809
547800 42 19.1428571 1.6463882 155.066776
547900 42 36.5000000 9.1027541 118.921589
548000 42 35.8571429 11.9339611 100.130373
548100 42 30.7142857 5.9258478 102.551512
548200 42 29.8571429 2.2935396 152.820431
548300 42 41.2857143 15.7270292 142.870586
548500 42 14.4285714 3.8207591 47.040559
548611 42 20.0000000 3.3661392 72.808725
548612 42 19.1428571 5.1685590 68.103842
548620 42 26.6428571 5.0910148 99.285744
548810 42 12.2857143 3.5689808 40.479047
548821 42 27.5714286 1.3883136 284.754522
548822 42 8.2857143 0.0938502 131.290041
548831 42 33.9285714 1.4887069 319.991873
548832 42 29.1428571 0.8086994 349.169480
548833 42 9.7142857 0.0980684 182.379301
549000 42 4.7142857 0.3929028 33.780745
549100 42 19.1428571 1.9459072 114.175575
549200 42 24.3571429 1.1216312 290.149372
549400 42 5.2142857 0.6294355 25.047348
549500 42 37.5714286 2.7032514 261.970884
549601 42 1.9285714 0.1388464 15.140567
549602 42 28.2142857 0.6627377 414.176448
550600 42 4.2857143 0.1857299 38.802236
550700 42 19.0714286 1.1323776 183.644339
550800 42 12.0714286 0.6292876 122.503248
550901 42 45.8571429 4.5524862 317.718292
550902 42 1.6428571 0.0387832 17.531495
551012 42 8.2857143 0.1820803 92.479728
551013 42 4.0000000 0.0419868 50.557711
551014 42 7.5714286 0.2075396 100.371844
551021 42 6.8571429 0.1530966 98.633090
551022 42 26.9285714 0.4691343 398.450688
551023 42 7.2142857 0.0000124 115.952550
551024 42 20.3571429 0.3311885 276.536066
551030 42 4.7142857 0.0650928 66.088558
551111 42 20.0714286 0.6185918 240.307156
551112 42 24.7142857 0.9910270 279.477766
551120 42 5.0000000 0.7629521 23.152853
551301 42 35.7857143 4.0767467 170.038672
551302 42 26.1428571 3.1824677 134.899174
551400 42 33.0714286 2.1820214 230.742004
551501 42 21.6428571 1.5654291 150.157572
551502 42 19.2857143 1.6686889 127.604332
551503 42 18.8571429 3.1964973 74.465817
551600 42 14.8571429 0.6743209 141.267163
551700 42 35.2142857 2.3775970 217.391119
551800 42 7.8571429 0.3414628 48.336343
551900 42 19.5714286 1.2504926 122.244272
552001 42 8.2857143 0.5148108 50.390376
552002 42 14.2857143 0.3855247 132.031803
552100 42 37.0000000 2.4750913 260.769310
552200 42 16.5714286 1.2629810 111.069438
552300 42 27.4285714 1.6399966 222.827178
552400 42 31.7857143 3.9926988 161.862968
552500 42 48.3571429 3.9617116 299.067230
552701 42 5.0714286 0.2874266 54.585083
552702 42 23.7142857 1.1711838 254.434157
552800 42 31.2857143 2.7192015 204.301883
552900 42 2.4285714 0.3838621 12.261250
553002 42 6.1428571 0.3582630 56.345325
553003 42 10.3571429 0.6715432 94.264453
553004 42 10.2857143 0.8047909 88.834574
553101 42 31.3571429 1.8436426 224.087361
553102 42 22.2142857 1.6723922 154.627822
553200 42 13.3571429 1.0138471 86.455247
553301 42 2.4285714 0.1023269 20.494475
553302 42 22.0714286 1.7097382 164.883893
553400 42 3.0000000 0.2001224 20.017898
553500 42 14.0000000 0.6120396 140.813536
553601 42 18.9285714 1.7009284 120.104049
553700 42 10.6428571 1.1042813 74.049071
553800 42 9.0714286 1.6277307 39.435813
553900 42 7.8571429 0.7846024 47.827858
554010 42 36.7857143 3.3947648 236.928986
554020 42 37.1428571 2.7248477 246.504656
554111 42 6.2857143 0.4347708 54.111812
554113 42 6.4285714 0.2370765 83.972804
554114 42 3.7142857 0.0968076 57.188141
554115 42 7.2857143 0.0358343 131.179698
554120 42 5.0000000 0.0938164 61.315657
554130 42 4.7142857 0.1542146 36.979021
554300 42 0.7857381 0.0000000 11.525600
554400 42 9.0714286 0.7026226 74.067893
554500 42 7.7857143 0.1115762 108.810693
554700 42 11.3571429 0.6607632 74.452220
554800 42 8.6428571 0.3089099 82.028018

9 About

Analysis completed in 5.294 seconds ( 0.09 minutes) using knitr in RStudio with R version 3.5.2 (2018-12-20) running on x86_64-apple-darwin15.6.0.

Additional R packages used in this report (data.table, dkUtils, ggplot2, hms, ipfp, GREENGridData, kableExtra, lubridate, plotly, readr, skimr, survey, viridis):

  • data.table - for fast data munching (Dowle et al. 2015)
  • dkUtils - utilities (Anderson 2018)
  • ggplot2 - for slick graphics (Wickham 2009)
  • GREENGridData - manipulating GREENGrid demand data (Anderson and Eyers 2018)
  • hms - HH:MM:SS conversions (Müller 2018)
  • ipfp - for IPF process (Blocker 2016)
  • kableExtra - for pretty tables (Zhu 2018)
  • knitr - to create this document (Xie 2016)
  • lubridate - date and time conversions (Grolemund and Wickham 2011)
  • plotly - for interactive graphics (Sievert et al. 2016)
  • readr - for data loading (Wickham, Hester, and Francois 2016)
  • skimr - for data summaries (Arino de la Rubia et al. 2017)
  • survey - for weighted data summaries (Lumley 2016)
  • viridis - for color palettes (Garnier 2018)

References

Anderson, Ben. 2018. DkUtils: A Bunch of Useful Little Functions. https://github.com/dataknut/dkUtils.

Anderson, Ben, and David Eyers. 2018. GREENGridData: Processing Nz Green Grid Project Data to Create a ’Safe’ Version for Data Archiving and Re-Use. https://github.com/CfSOtago/GREENGridData.

Anderson, Ben, David Eyers, Rebecca Ford, Diana Giraldo Ocampo, Rana Peniamina, Janet Stephenson, Kiti Suomalainen, Lara Wilcocks, and Michael Jack. 2018. “New Zealand GREEN Grid Household Electricity Demand Study 2014-2018,” September. https://doi.org/10.5255/UKDA-SN-853334.

Arino de la Rubia, Eduardo, Hao Zhu, Shannon Ellis, Elin Waring, and Michael Quinn. 2017. Skimr: Skimr. https://github.com/ropenscilabs/skimr.

Blocker, Alexander W. 2016. Ipfp: Fast Implementation of the Iterative Proportional Fitting Procedure in c. https://CRAN.R-project.org/package=ipfp.

Dowle, M, A Srinivasan, T Short, S Lianoglou with contributions from R Saporta, and E Antonyan. 2015. Data.table: Extension of Data.frame. https://CRAN.R-project.org/package=data.table.

Garnier, Simon. 2018. Viridis: Default Color Maps from ’Matplotlib’. https://CRAN.R-project.org/package=viridis.

Grolemund, Garrett, and Hadley Wickham. 2011. “Dates and Times Made Easy with lubridate.” Journal of Statistical Software 40 (3): 1–25. http://www.jstatsoft.org/v40/i03/.

Lumley, Thomas. 2016. Survey: Analysis of Complex Survey Samples.

Müller, Kirill. 2018. Hms: Pretty Time of Day. https://CRAN.R-project.org/package=hms.

Sievert, Carson, Chris Parmer, Toby Hocking, Scott Chamberlain, Karthik Ram, Marianne Corvellec, and Pedro Despouy. 2016. Plotly: Create Interactive Web Graphics via ’Plotly.js’. https://CRAN.R-project.org/package=plotly.

Wickham, Hadley. 2009. Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. http://ggplot2.org.

Wickham, Hadley, Jim Hester, and Romain Francois. 2016. Readr: Read Tabular Data. https://CRAN.R-project.org/package=readr.

Xie, Yihui. 2016. Knitr: A General-Purpose Package for Dynamic Report Generation in R. https://CRAN.R-project.org/package=knitr.

Zhu, Hao. 2018. KableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.