---
title: "Building reproducible data capsules"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Building reproducible data capsules}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

`rmoriebricklayer` turns "the data was downloaded from the portal at some
point" into a verifiable, self-contained record: a **capsule**. A capsule
pins where the data came from, what its bytes hashed to, what shape it must
have, and — when the portal is unreachable — how to synthesize a stand-in so
the pipeline still runs end-to-end.

This vignette walks the essential flow with a small, fully offline example:

1. pin a source in a provenance record,
2. validate data against the pinned schema,
3. verify file integrity with SHA256,
4. fall back to schema-driven synthetic data,
5. account for every step in a manifest and plain-language summary.

```{r setup}
library(rmoriebricklayer)
```

## 1. The provenance record

Everything starts from a `data_provenance.json` file: the machine-readable
pin of a project's data source. It records the portal endpoint, a pattern
identifying the right resource, the expected SHA256, the schema the data
must satisfy, and a recipe for synthesizing a stand-in.

```{r provenance}
prov_json <- '{
  "dataset": {
    "title": "Demo library statistics",
    "ckan_api_endpoint": "https://data.ontario.ca/api/3/action/package_show?id=ontario-public-library-statistics"
  },
  "resource": { "name_match_pattern": "2014" },
  "schema": {
    "expected_columns": ["year", "visits", "alert"],
    "structural_invariants": { "min_data_rows": 5 },
    "expected_value_sets": { "year": [2024, 2025] },
    "synthetic_recipe": {
      "n_rows": 25,
      "seed": 42,
      "columns": {
        "year":   { "type": "sample",    "values": [2024, 2025] },
        "visits": { "type": "poisson",   "lambda": 3, "min": 1 },
        "alert":  { "type": "bernoulli", "p": 0.2 },
        "id":     { "type": "id_pattern", "pattern": "p-{seq:05d}" }
      }
    }
  }
}'

prov_path <- file.path(tempdir(), "data_provenance.json")
writeLines(prov_json, prov_path)

prov <- load_provenance(prov_path)
prov$dataset$title
```

## 2. Resolving the live source (network)

On a machine with network access, the resolvers turn the pinned endpoint
into a current download URL — surviving portal-side resource-UUID churn.
CKAN powers data.ontario.ca, data.gov.uk, and data.gov; Socrata and ArcGIS
resolvers cover the Chicago/NYC/Calgary and Toronto Police portals.

```{r resolve, eval = FALSE}
url <- resolve_via_ckan(prov)            # package_show + name match
if (is.null(url))                        # slug-change fallback
  url <- resolve_via_ckan_search(prov)

path <- file.path(tempdir(), "data.csv")
friendly_download(url, path)             # plain-language failure diagnosis
                                         # + automatic Wayback fallback
```

This vignette stays offline, so we go straight to the fallback path.

## 3. Synthetic fallback: the pipeline always runs

When the real source is down, `make_synthetic_csv()` generates a
reproducible stand-in from the recipe pinned in the provenance itself.
Results from synthetic data are marked as such all the way through (see the
manifest below) — the point is to keep the *pipeline* testable, never to
pass synthetic output off as real.

```{r synthetic}
data_path <- file.path(tempdir(), "data.csv")
gen <- make_synthetic_csv(prov$schema$synthetic_recipe, data_path)
gen$rows

df <- read.csv(data_path)
head(df, 3)
```

## 4. Integrity and schema validation

Every file gets a SHA256; every data frame is checked against the pinned
schema. `validate_schema()` reports issues without raising, so callers
choose the severity response; `apply_schema_validation()` is the strict
wrapper (fatal issues stop, drift warns).

```{r validate}
digest <- sha256_file(data_path)
substr(digest, 1, 16)

chk <- verify_sha256(data_path, digest)
chk$match

issues <- validate_schema(df, prov)
length(issues)   # 0 = clean against the pinned schema
```

## 5. The manifest: account for everything

`make_manifest()` + `record()` accumulate named cross-checks (each PASS /
DIFFER / INFO), and the writers produce the two capsule artifacts: a
machine-readable `manifest.json` and a human-readable `SUMMARY.txt`.

```{r manifest}
man <- make_manifest(
  list(project = "demo-study", author = "A. Author", synthetic = TRUE),
  environment = FALSE  # TRUE also snapshots R/OS/package versions
)

man <- record(man, "rows_generated", observed = gen$rows, expected = 25)
man <- record(man, "mean_visits",
              observed = mean(df$visits), expected = 3,
              tol = 1, synthetic = TRUE)

out_dir <- file.path(tempdir(), "capsule-demo")
dir.create(out_dir, showWarnings = FALSE)

write_manifest_json(man, file.path(out_dir, "manifest.json"))
summary_path <- write_summary_txt(
  man, out_dir,
  paths = list(input = data_path, results = out_dir),
  what_was_done = c("* generated synthetic stand-in (real source offline)",
                    "* validated schema and recorded cross-checks")
)

cat(readLines(summary_path)[7:12], sep = "\n")
```

The capsule directory now contains everything a reviewer needs to trace the
run: the exact inputs, their digests, every cross-check with its status,
and a plain-language account — reproducible years later even if the portal
has moved on.

## Where this sits in the MORIE family

`rmoriebricklayer` is the provenance/reproducibility layer under
[rmorie](https://github.com/rootcoder007/rmorie) and
[rmoriedata](https://github.com/rootcoder007/rmoriedata); both link against
its compiled core (`LinkingTo: rmoriebricklayer`) for shared SHA-256 and
summary-statistic kernels (`core_sha256()`, `core_mean()`, `core_var()`,
`core_cor()`).