90-appendixA.Rmd

`r if(knitr:::is_latex_output()) '\\startappendices'`

`r if(!knitr:::is_latex_output()) '# (APPENDIX) Appendix {-}'` 

```{r}
resource_version <- list.files("resources/beyond-borders")
```

# Models for areal spatial structure

## Comparison of AGHQ to NUTS {#aghq-nuts}

```{r}
aghq_nuts_comparison_diagnostics <- readr::read_csv("figures/beyond-borders/check_aghq-nuts/diagnostics.csv")
aghq_nuts_comparison_diagnostics_iid <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "IID")
aghq_nuts_comparison_diagnostics_besag <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "Besag")
aghq_nuts_comparison_diagnostics_bym2 <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "BYM2")
aghq_nuts_comparison_diagnostics_fck <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "FCK")
aghq_nuts_comparison_diagnostics_ck <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "CK")
aghq_nuts_comparison_diagnostics_fik <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "FIK")
aghq_nuts_comparison_diagnostics_ik <- dplyr::filter(aghq_nuts_comparison_diagnostics, inf_model == "IK")
```

(ref:time-taken) A comparison of time taken to fit AGHQ via `aghq` as compared with NUTS via `tmbstan` for each inferential model. For the models run using NUTS via `tmbstan` there was significant variation in time taken depending on initial random seed. As such, these timings and more broadly the inferences obtained from NUTS in Appendix \@ref(aghq-nuts) should be interpreted with appropriate skepticism.

```{r time-taken, fig.cap="(ref:time-taken)"}
path <- "figures/beyond-borders/check_aghq-nuts/time-taken.png"
knitr::include_graphics(path)
```

(ref:iid-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting an IID inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_iid["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_iid["max_rhat"])`.

```{r iid-aghq-nuts, fig.cap="(ref:iid-aghq-nuts)"}
# Sometimes Stan hung up when I ran these: likely because it depends on the initial warm-up
# As a result I ran them one-by-one to supervise. This isn't good for reproducibility
# I moved the resulting figures into this folder rather than use orderly for this reason
# Given more time I'd like to do a better job here!
path <- "figures/beyond-borders/check_aghq-nuts/iid-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:besag-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a Besag inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_besag["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_besag["max_rhat"])`.

```{r besag-aghq-nuts, fig.cap="(ref:besag-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/besag-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:bym2-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a BYM2 inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_bym2["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_bym2["max_rhat"])`.

```{r bym2-aghq-nuts, fig.cap="(ref:bym2-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/bym2-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:fck-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a FCK inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_fck["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_fck["max_rhat"])`.

```{r fck-aghq-nuts, fig.cap="(ref:fck-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/fck-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:ck-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a CK inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_ck["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_ck["max_rhat"])`.

```{r ck-aghq-nuts, fig.cap="(ref:ck-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/ck-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:fik-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a FIK inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_fik["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_fik["max_rhat"])`.

```{r fik-aghq-nuts, fig.cap="(ref:fik-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/fik-aghq-nuts.png"
knitr::include_graphics(path)
```

(ref:ik-aghq-nuts) A comparison of the posterior means and standard deviations obtained with AGHQ via `aghq` as compared with NUTS via `tmbstan` fitting a IK inferential model to IID synthetic data on the grid geometry (Panel \@ref(fig:geometries)E). For NUTS, the minimum ESS was `r round(aghq_nuts_comparison_diagnostics_ik["min_ess"])`, and the maximum value of the potential scale reduction factor was `r sprintf("%.2f", aghq_nuts_comparison_diagnostics_ik["max_rhat"])`.

```{r ik-aghq-nuts, fig.cap="(ref:ik-aghq-nuts)"}
path <- "figures/beyond-borders/check_aghq-nuts/ik-aghq-nuts.png"
knitr::include_graphics(path)
```

\newpage

## Lengthscale prior sensitivity {#lengthscale-prior}

| Description | Prior | Additional details  |
|:-------|:------|:--------------------|
| Gamma | $l \sim \text{Gamma}(1, 1)$ |$-$ |
| Geometry-informed inverse-gamma | $l \sim \text{IG}(a, b)$ | The parameters $a$ and $b$ chosen such that 5\% of the prior mass was below and above the 5% and 95% quantile for distance between points [@betancourt2017robust] |
| Geometry-informed normal | $l \sim \mathcal{N}^{+}(0, \sigma)$ | The parameter $\sigma$ set as one third the difference between the minimum and maximum distance between points [@betancourt2017robust] |
| Log-normal | $l \sim \text{Log-normal}(0, 1)$ | $-$ |
| Non-informative | $p(l) = 1$ | This is an improper prior in that it does not integrate to one |
| Oracle normal | $l \sim \mathcal{N}^{+}(2.5, 1)$ | The mean of this prior was set to the true value of the lengthscale |

Table: (\#tab:lengthscale-prior) Six lengthscale prior distributions were considered for use in the simulation (Section \@ref(simulation-study)) and HIV prevalence (Section \@ref(hiv-study)) studies.

(ref:lengthscale-prior) The probability density for each lengthscale prior distribution as given in Table \@ref(tab:lengthscale-prior).

```{r lengthscale-prior, fig.cap="(ref:lengthscale-prior)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/lengthscale-prior.png")
knitr::include_graphics(path)
```

(ref:lengthscale-posterior) Lengthscale posterior distributions obtained using NUTS to fit a centroid kernel model to integrated kernel data. The true value, 2.5, is shown as a dashed vertical line. Six different lengthscale prior distributions were considered as given in Table \@ref(tab:lengthscale-prior). The geometry used was the grid (Panel \@ref(fig:geometries)E). 

```{r lengthscale-posterior, fig.cap="(ref:lengthscale-posterior)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/lengthscale-posterior.png")
knitr::include_graphics(path)
```

## Simulation study {#simulation-appendix}

### Lengthscale recovery {#lengthscale}

(ref:lengthscale-recovery) The lengthscale posterior mean and 95% credible interval obtained using the centroid kernel model on integrated kernel data for the first 40 simulation replicates on each geometry. The true lengthscale, and lengthscale obtained using the heuristic method of @best1999bayesian, are shown as dashed horizontal lines.

```{r lengthscale-recovery, fig.cap="(ref:lengthscale-recovery)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/lengthscale-recovery.png")
knitr::include_graphics(path)
```

### BYM2 proportion {#bym2-proportion}

(ref:proportion-recovery) The BYM2 proportion parameter posterior mean and 95% credible interval obtained for the first 40 simulation replicates for the realistic geometries. When the simulated data is IID, the BYM2 proportion parameter is in the majority of cases below 0.5, corresponding to have inferred that the noise is mostly IID (spatially unstructured) When the simulated data is either Besag or IK, the BYM2 proportion parameter is in the majority of cases above 0.5, corresponding to have inferred that the noise is mostly Besag (spatially structured).

```{r proportion-recovery, fig.cap="(ref:proportion-recovery)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/proportion-recovery.png")
knitr::include_graphics(path)
```

### Mean squared error

(ref:mse-table) The average mean squared error (MSE) of each inferential model in estimating $\rho$, under different simulation and geometry settings. Entries for FCK and CK on geometry 2 are empty because model was undefined in that case. The units used in this table are expressed in thousandths.

```{r mse-table}
gt_mse <- readRDS(paste0("resources/beyond-borders/", resource_version, "/depends/gt_mse.rds"))

as_latex_with_caption <- function(gt, chunk_label) {
  gt <- gt::as_latex(gt)
  caption <- paste0(
    "\\caption{\\label{tab:", chunk_label, "}(ref:", chunk_label, ")}\\\\")
  latex <- strsplit(gt[1], split = "\n")[[1]]
  latex <- c(latex[1], caption, latex[-1])
  latex <- paste(latex, collapse = "\n")
  gt[1] <- latex
  return(gt)
}

if(knitr::is_html_output()) {
  gt_mse %>%
    gt::tab_caption("(ref:mse-table)")
}

if(knitr::is_latex_output()) {
  gt_mse %>%
    as_latex_with_caption("mse-table")
}
```

### Continuous ranked probability score

(ref:crps-table) The average continuous ranked probability score (CRPS) of each inferential model in estimating $\rho$, under different simulation and geometry settings. Entries for FCK and CK on geometry 2 are empty because model was undefined in that case. The units used in this table are thousandths.

```{r crps-table}
gt_crps <- readRDS(paste0("resources/beyond-borders/", resource_version, "/depends/gt_crps.rds"))

if(knitr::is_html_output()) {
  gt_crps %>%
    gt::tab_caption("(ref:crps-table)")
}

if(knitr::is_latex_output()) {
  gt_crps %>%
    as_latex_with_caption("crps-table")
}
```

(ref:crps-mean-se-1) The mean CRPS with 95% credible interval in estimating $\rho$ using each inferential model and simulation model on the first vignette geometry (Panel \@ref(fig:geometries)A). Credible intervals were generated using 1.96 times the standard error.

```{r crps-mean-se-1, fig.cap="(ref:crps-mean-se-1)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-1.png")
knitr::include_graphics(path)
```

(ref:crps-mean-se-2) The mean CRPS with 95% credible interval in estimating $\rho$ using each inferential model and simulation model on the second vignette geometry (Panel \@ref(fig:geometries)B). Credible intervals were generated using 1.96 times the standard error.

```{r crps-mean-se-2, fig.cap="(ref:crps-mean-se-2)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-2.png")
knitr::include_graphics(path)
```

(ref:crps-mean-se-3) The mean CRPS with 95% credible interval in estimating $\rho$ using each inferential model and simulation model on third vignette geometry (Panel \@ref(fig:geometries)C). Credible intervals were generated using 1.96 times the standard error.

```{r crps-mean-se-3, fig.cap="(ref:crps-mean-se-3)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-3.png")
knitr::include_graphics(path)
```

(ref:crps-mean-se-4) The mean CRPS with 95% credible interval in estimating $\rho$ using each inferential model and simulation model on the fourth vignette geometry (Panel \@ref(fig:geometries)D). Credible intervals were generated using 1.96 times the standard error.

```{r crps-mean-se-4, fig.cap="(ref:crps-mean-se-4)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-4.png")
knitr::include_graphics(path)
```

(ref:map-crps-1) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the first vignette geometry (Panel \@ref(fig:geometries)A).

```{r map-crps-1, fig.cap="(ref:map-crps-1)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-1.png")
knitr::include_graphics(path)
```

(ref:map-crps-2) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the second vignette geometry (Panel \@ref(fig:geometries)B).

```{r map-crps-2, fig.cap="(ref:map-crps-2)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-2.png")
knitr::include_graphics(path)
```

(ref:map-crps-3) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the third vignette geometry (Panel \@ref(fig:geometries)C).

```{r map-crps-3, fig.cap="(ref:map-crps-3)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-3.png")
knitr::include_graphics(path)
```

(ref:map-crps-4) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the fourth vignette geometry (Panel \@ref(fig:geometries)D).

```{r map-crps-4, fig.cap="(ref:map-crps-4)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-4.png")
knitr::include_graphics(path)
```

(ref:map-crps-grid) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the grid geometry (Panel \@ref(fig:geometries)E).

```{r map-crps-grid, fig.cap="(ref:map-crps-grid)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-grid.png")
knitr::include_graphics(path)
```

(ref:map-crps-civ) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the Côte d’Ivoire geometry (Panel \@ref(fig:geometries)F).

```{r map-crps-civ, fig.cap="(ref:map-crps-civ)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-civ.png")
knitr::include_graphics(path)
```

(ref:map-crps-tex) Choropleths showing the mean value of the CRPS in estimating $\rho$, under each inferential model and simulation model, at each area of the Texas geometry (Panel \@ref(fig:geometries)G).

```{r map-crps-tex, fig.cap="(ref:map-crps-tex)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/map-crps-tex.png")
knitr::include_graphics(path)
```

### Calibration

(ref:histogram-ecdf-diff-1) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the first vignette geometry (Panel \@ref(fig:geometries)A).

```{r histogram-ecdf-diff-1, fig.cap="(ref:histogram-ecdf-diff-1)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-1.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-2) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the second vignette geometry (Panel \@ref(fig:geometries)B).

```{r histogram-ecdf-diff-2, fig.cap="(ref:histogram-ecdf-diff-2)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-2.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-3) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the third vignette geometry (Panel \@ref(fig:geometries)C).

```{r histogram-ecdf-diff-3, fig.cap="(ref:histogram-ecdf-diff-3)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-3.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-4) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the fourth vignette geometry (Panel \@ref(fig:geometries)D).

```{r histogram-ecdf-diff-4, fig.cap="(ref:histogram-ecdf-diff-4)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-4.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-grid) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the grid geometry (Panel \@ref(fig:geometries)E).

```{r histogram-ecdf-diff-grid, fig.cap="(ref:histogram-ecdf-diff-grid)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-grid.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-civ) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the Côte d’Ivoire geometry (Panel \@ref(fig:geometries)F).

```{r histogram-ecdf-diff-civ, fig.cap="(ref:histogram-ecdf-diff-civ)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-civ.png")
knitr::include_graphics(path)
```

(ref:histogram-ecdf-diff-tex) Probability integral transform histograms and empirical cumulative distribution function difference plots for $\rho$, under each inferential model and simulation model, for the Texas geometry (Panel \@ref(fig:geometries)G).

```{r histogram-ecdf-diff-tex, fig.cap="(ref:histogram-ecdf-diff-tex)"}
path <- paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-tex.png")
knitr::include_graphics(path)
```

## HIV study {#hiv-appendix}

### Lengthscale

(ref:lengthscale-posteriors) The lengthscale hyperparameter prior and posterior distributions for each of the four considered PHIA surveys (Table \@ref(tab:hiv-surveys)), using both the CK and IK inferential models.

```{r lengthscale-posteriors, fig.cap="(ref:lengthscale-posteriors)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/lengthscale-posteriors.png"))
```

### BYM2 proportion

(ref:proportion-posteriors) The BYM2 proportion hyperparameter prior and posterior distributions for each of the four considered PHIA surveys (Table \@ref(tab:hiv-surveys)). A value of zero corresponds to IID noise. A value of one corresponds to Besag noise. For each survey, excluding the Côte d’Ivoire 2017 PHIA, the posterior distribution for the BYM2 proportion is concentrated towards a value of one. This result can be interpreted as suggesting that the variation in HIV prevalence from these surveys is spatially structured.

```{r proportion-posteriors, fig.cap="(ref:proportion-posteriors)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/proportion-posteriors.png"))
```

### Estimates

(ref:ladder-civ2017phia) The HIV prevalence posterior mean and 95% credible interval for each area of Côte d’Ivoire, based on the 2017 PHIA survey. Direct estimates obtained from the survey are as shown in Panel \@ref(fig:hiv-surveys)A.

```{r ladder-civ2017phia, fig.cap="(ref:ladder-civ2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/ladder-civ2017phia.png"))
```

(ref:ladder-mwi2016phia) The HIV prevalence posterior mean and 95% credible interval for each area of Malawi, based on the 2016 PHIA survey. Direct estimates obtained from the survey are as shown in Panel \@ref(fig:hiv-surveys)B.

```{r ladder-mwi2016phia, fig.cap="(ref:ladder-mwi2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/ladder-mwi2016phia.png"))
```

(ref:ladder-tza2017phia) The HIV prevalence posterior mean and 95% credible interval for each area of Tanzania, based on the 2017 PHIA survey. Direct estimates obtained from the survey are as shown in Panel \@ref(fig:hiv-surveys)C.

```{r ladder-tza2017phia, fig.cap="(ref:ladder-tza2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/ladder-tza2017phia.png"))
```

(ref:ladder-zwe2016phia) The HIV prevalence posterior mean and 95% credible interval for each area of Zimbabwe, based on the 2016 PHIA survey. Direct estimates obtained from the survey are as shown in Panel \@ref(fig:hiv-surveys)D.

```{r ladder-zwe2016phia, fig.cap="(ref:ladder-zwe2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/ladder-zwe2016phia.png"))
```

### Cross-validation

#### Mean squared error

(ref:cv-mse) The mean pointwise leave-one-out and spatial leave-one-out MSE in estimating $\rho_i$, with standard errors, for each inferential model across the four considered PHIA surveys. The units used in this table are thousandths.

```{r cv-mse}
gt_cv_mse <- readRDS(paste0("resources/beyond-borders/", resource_version, "/depends/gt_cv-mse.rds"))

as_latex_with_caption <- function(gt, chunk_label) {
  gt <- gt::as_latex(gt)
  caption <- paste0(
    "\\caption{\\label{tab:", chunk_label, "}(ref:", chunk_label, ")}\\\\")
  latex <- strsplit(gt[1], split = "\n")[[1]]
  latex <- c(latex[1], caption, latex[-1])
  latex <- paste(latex, collapse = "\n")
  gt[1] <- latex
  return(gt)
}

if(knitr::is_html_output()) {
  gt_cv_mse %>%
    gt::tab_caption("(ref:cv-mse)")
}

if(knitr::is_latex_output()) {
  gt_cv_mse %>%
    as_latex_with_caption("cv-mse")
}
```

#### Continuous ranked probability score

(ref:crps-mean-se-civ2017phia) The pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation, with mean and 95% credible interval for the Côte d’Ivoire 2017 PHIA survey (Panel \@ref(fig:hiv-surveys)A).

```{r crps-mean-se-civ2017phia, fig.cap="(ref:crps-mean-se-civ2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-civ2017phia.png"))
```

(ref:crps-mean-se-mwi2016phia) The pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation, with mean and 95% credible interval, for the Malawi 2016 PHIA survey \@ref(fig:hiv-surveys)B.

```{r crps-mean-se-mwi2016phia, fig.cap="(ref:crps-mean-se-mwi2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-mwi2016phia.png"))
```

(ref:crps-mean-se-tza2017phia) The pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation, with mean and 95% credible interval, for the Tanzania 2017 PHIA survey \@ref(fig:hiv-surveys)C.

```{r crps-mean-se-tza2017phia, fig.cap="(ref:crps-mean-se-tza2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-tza2017phia.png"))
```

(ref:crps-mean-se-zwe2016phia) The pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation, with mean and 95% credible interval, for the Zimbabwe 2016 PHIA survey \@ref(fig:hiv-surveys)D.

```{r crps-mean-se-zwe2016phia, fig.cap="(ref:crps-mean-se-zwe2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/crps-mean-se-zwe2016phia.png"))
```

(ref:map-civ2017phia) Choropleth showing the pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation for the Côte d’Ivoire 2017 PHIA survey (Panel \@ref(fig:hiv-surveys)A).

```{r map-civ2017phia, fig.cap="(ref:map-civ2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/map-civ2017phia.png"))
```

(ref:map-mwi2016phia) Choropleth showing the pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation for the Malawi 2016 PHIA survey (Panel \@ref(fig:hiv-surveys)B).

```{r map-mwi2016phia, fig.cap="(ref:map-mwi2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/map-mwi2016phia.png"))
```

(ref:map-tza2017phia) Choropleth showing the pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation for the Tanzania 2017 PHIA survey (Panel \@ref(fig:hiv-surveys)C).

```{r map-tza2017phia, fig.cap="(ref:map-tza2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/map-tza2017phia.png"))
```

(ref:map-zwe2016phia) Choropleth showing the pointwise CRPS in estimating $\rho_i$ using either leave-one-out or spatial leave-one-out cross-validation for the Zimbabwe 2016 PHIA survey (Panel \@ref(fig:hiv-surveys)D).

```{r map-zwe2016phia, fig.cap="(ref:map-zwe2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/map-zwe2016phia.png"))
```

(ref:histogram-ecdf-diff-civ2017phia) Probability integral transform histograms and empirical cumulative distribution function difference plots in estimating $\rho$ for the Côte d’Ivoire 2017 PHIA survey (Panel \@ref(fig:hiv-surveys)A).

```{r histogram-ecdf-diff-civ2017phia, fig.cap="(ref:histogram-ecdf-diff-civ2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-civ2017phia.png"))
```

(ref:histogram-ecdf-diff-mwi2016phia)  Probability integral transform histograms and empirical cumulative distribution function difference plots in estimating $\rho$ for the Malawi 2016 PHIA survey (Panel \@ref(fig:hiv-surveys)B).

```{r histogram-ecdf-diff-mwi2016phia, fig.cap="(ref:histogram-ecdf-diff-mwi2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-mwi2016phia.png"))
```

(ref:histogram-ecdf-diff-tza2017phia)  Probability integral transform histograms and empirical cumulative distribution function difference plots in estimating $\rho$ for the Tanzania 2017 PHIA survey (Panel \@ref(fig:hiv-surveys)C).

```{r histogram-ecdf-diff-tza2017phia, fig.cap="(ref:histogram-ecdf-diff-tza2017phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-tza2017phia.png"))
```

(ref:histogram-ecdf-diff-zwe2016phia)  Probability integral transform histograms and empirical cumulative distribution function difference plots in estimating $\rho$ for the Zimbabwe 2016 PHIA survey (Panel \@ref(fig:hiv-surveys)D).

```{r histogram-ecdf-diff-zwe2016phia, fig.cap="(ref:histogram-ecdf-diff-zwe2016phia)"}
knitr::include_graphics(paste0("resources/beyond-borders/", resource_version, "/depends/histogram-ecdf-diff-zwe2016phia.png"))
```