# Data library — District source notes (knowledge layer)

Reporter notebook for the district knowledge layer. This layer surfaces
attributed public-record statements, official board-meeting excerpts, and
quoted testimony alongside district profile pages and, where budget-relevant
notes exist, the generated district budget report. It is a **separate,
complementary layer** — see `district-source-notes/complementary-not-ufb-revision`
for exactly what that does and doesn't mean.

Unlike the other pages in this library, this is not a scraped government
file: it's an internally curated knowledge layer, hand-loaded into a Python
`SEED` list and rebuilt into a warehouse table on every run. The publisher
of record is this project, not NJDOE — every row still traces to a public
source (a board video, a filing, a news article), but the curation,
attribution, and rights-gating are ours.

```toml ergo
[dataset]
ergo = "0.4"
slug = "district-source-notes"
title = "District source notes (knowledge layer)"
publisher = "njschooldata (curated from public records)"
subject = "https://njschooldata.fyi/data/district-source-notes/"
source_url = "https://njschooldata.fyi/data/district-source-notes/"
pitfall = "Only rows with rights = public_record or link_quote may publish — anything else is rejected at build time, by the table's CHECK constraint, and by the check_notes_rights.py gate before any profile ships."
status = "live"
confidence = "B"
updated = "2026-07-10"
implementation = "https://github.com/lavallee/njschooldata"

[dataset.coverage]
entities = "South Orange-Maplewood SD only (district_code 4900) as of this writing; the layer is designed to extend to any NJ district"
years = "2026 (dates of the underlying statements/documents)"
vintage = "SEED list curated 2026-07"

[dataset.access]
keys = ["district_code"]
```

## Purpose

The UFB budget module on district profiles and full budget reports shows
statewide NJDOE data. The knowledge layer adds district-specific context that
is only available in primary documents: board resolutions, public budget
presentations, meeting minutes, and linked press quotes. For SOMSD, the
initial focus is budget structure and arts funding (commissioned by edie,
2026).

## Schema

Warehouse table: `district_source_notes`
Built by the notes builder from a curated seed list — see
`district-source-notes/db-rebuilt-from-seed-each-run` for the rebuild
behavior.

| Column | Type | Notes |
|---|---|---|
| `id` | INTEGER PK | auto-assigned, resets on every rebuild |
| `district_code` | TEXT | NJDOE district code (e.g. `4900` for SOMSD) |
| `date` | TEXT | ISO-8601 date of the source document or meeting |
| `source_type` | TEXT | e.g. `budget_document`, `board_meeting`, `board_video`, `news_article`, `state_data`, `public_filing` |
| `source_url` | TEXT | direct URL to the primary source, or a local warehouse path — see `district-source-notes/source-url-non-http-not-linked` |
| `speaker` | TEXT | name of person quoted (null for documents without a named author) |
| `speaker_role` | TEXT | role/title at time of statement |
| `quote_or_claim` | TEXT | the attributed text or summarised claim; may carry an `internal_only` marker — see `district-source-notes/internal-only-marker-is-freetext` |
| `rights` | TEXT | **must be** `public_record` or `link_quote` — see `district-source-notes/invalid-rights-blocks-publish` |
| `confidence` | TEXT | designed to match the `soma-student-population/SOURCES.md` A/B/C scale — see `district-source-notes/confidence-values-are-not-letters` for what's actually loaded |
| `storyline` | TEXT | slug identifying the editorial thread (e.g. `budget`, `arts-funding`) |

### Rights values

| Value | Meaning |
|---|---|
| `public_record` | Official government document, board resolution, budget filing, or public meeting minutes/video |
| `link_quote` | Third-party article or publication; quote attributed to a named speaker; source URL points to the article |

### Confidence scale

Designed to match `soma-student-population/SOURCES.md`:

| Level | Meaning |
|---|---|
| A | Authoritative primary data we extracted directly |
| B | Official document; figures read directly from it |
| C | Reporting or secondary source |

The 18 SOMSD rows currently loaded don't use these letters — see
`district-source-notes/confidence-values-are-not-letters`. This page's
manifest rates the layer `B` as a rough single-line read (14 of 18 rows are
`public_record`, closer to B; 4 are `link_quote` via secondary news
reporting, closer to C) — treat a row's own `confidence` and `rights`
values as the operative signal for that row, not the manifest rating.

## Issues

### Notes render beside UFB budget figures but are not a revision to them

```toml ergo
[issue]
id = "complementary-not-ufb-revision"
title = "Notes render beside UFB budget figures on district pages but are not a revision to them"
effect = "context"
core = true
type = "definitional"
status = "open"
discovered = "2026-07"
misuse = "Treating a quoted board statement or linked news claim as verified, audited budget data, or reading this layer's presence as a correction to NJDOE User-Friendly Budget figures — it is a separate, complementary evidentiary layer, a record-to-UFB bridge rather than an independent verification of UFB numbers."

[issue.scope]
all = true
```

Real, not a bug: the layer is additive by design.
`build_district_notes_db.py` only touches the `district_source_notes` table;
`build_profiles.py` attaches notes in `attach_district_notes()` and renders
them in `mod_district_notes()` plus the full-report
`budget_local_record_context_panel()`. The UFB warehouse and budget figures
stay separate; district source notes are contextual records used to verify
or complicate budget-story claims, not to restate them. The gated "Local
public-record context" panel on full budget reports selects one strongest
note per record thread, keeps the source and rights/confidence flags
visible, and states how the note should be checked against the generated UFB
evidence — it is a record-to-UFB bridge, not a full transcript, complete cut
list, audited forecast, or independent verification beyond the cited source.

### A rights value outside public_record/link_quote is rejected before it can publish

```toml ergo
[issue]
id = "invalid-rights-blocks-publish"
title = "A rights value outside public_record/link_quote is rejected before it can publish"
effect = "breaks"
type = "coding"
status = "mitigated"
discovered = "2026-07"
detection = "A SEED row with rights not in {public_record, link_quote} raises ValueError in validate_seed() before any DB write; if one somehow reached the table anyway, the CHECK(rights IN ('public_record','link_quote')) constraint rejects the INSERT, and check_notes_rights.py exits non-zero at publish time, printing the offending row ids."
misuse = "Assuming any rights value is fine as long as a source_url is present — rights is the publish gate itself, checked three separate times (SEED validation, table constraint, publish-time query), and only public_record and link_quote pass."

[issue.scope]
tables = ["district_source_notes"]
columns = ["rights"]
```

Three independent gates, verified against the current source (2026-07-10):
`validate_seed()` walks the SEED list and raises `ValueError` before
`build()` ever touches the database; the `CREATE TABLE` statement in
`build()` carries `CHECK(rights IN ('public_record','link_quote'))` so a
direct `INSERT` bypassing the Python loader still fails; and
a standalone publish-time rights check, run before any
profile rebuild ships, queries the live table and exits non-zero — printing
`id`, `district_code`, `rights`, `storyline`, and `date` for every offending
row — if anything slipped through. All 18 currently loaded SOMSD rows
conform: 14 `public_record`, 4 `link_quote`.

### Loaded rows use confidence values 'high'/'medium', not the documented A/B/C letters

```toml ergo
[issue]
id = "confidence-values-are-not-letters"
title = "Loaded rows use confidence values 'high'/'medium', not the documented A/B/C letters"
effect = "corrupts"
type = "coding"
status = "open"
discovered = "2026-07"
detection = "SELECT DISTINCT confidence FROM district_source_notes (or grep the SEED list) returns only 'high' and 'medium' for all 18 loaded SOMSD rows — never 'A', 'B', or 'C'."
misuse = "Filtering or comparing district_source_notes.confidence against the page's documented A/B/C scale, e.g. a query for confidence = 'A' — it silently returns zero rows instead of erroring, because no loaded row uses a letter grade. the profile generator's confidence-rank lookups work around this by mapping both {high,medium,low} and {a,b,c} to the same rank; new code should not assume the documented scale is what's actually stored."

[issue.scope]
tables = ["district_source_notes"]
columns = ["confidence"]
```

Found while verifying the rights gate above: the SEED list's 18 rows split
10 `"high"` / 8 `"medium"`, not a single `A`/`B`/`C`. This page's own
"Confidence scale" table (matching `soma-student-population/SOURCES.md`) is
aspirational for this layer, not descriptive of what's loaded today. The two
places that currently rank confidence (the profile
generator’s two confidence-rank lookups)
happen to be defensive — their lookup dict maps both vocabularies to the
same 3/2/1 rank — so rendering isn't broken, but that's a property of those
two functions, not of the schema.

### district_source_notes is dropped and rebuilt from the SEED list on every run

```toml ergo
[issue]
id = "db-rebuilt-from-seed-each-run"
title = "district_source_notes is dropped and rebuilt from the SEED list on every run"
effect = "context"
type = "revision"
status = "mitigated"
discovered = "2026-07"
detection = "id (AUTOINCREMENT) resets and any hand-edited row disappears after the notes builder runs again — the build always does DROP TABLE IF EXISTS, then CREATE, then INSERT from the seed list."
misuse = "Editing a row directly in the SQLite district_source_notes table (or in a copy of the warehouse file) instead of the curated seed list the builder loads from — the edit is silently discarded the next time the builder runs."

[issue.scope]
all = true
```

Deliberate, not an oversight: rebuilding from scratch every run is what
guarantees the SEED list — reviewable in a diff, attributable in git blame —
stays the single source of truth, rather than the warehouse table quietly
accumulating hand edits nobody can trace. Re-running with identical SEED
rows is byte-equivalent.

### The internal_only flag is a substring inside quote_or_claim, not a column

```toml ergo
[issue]
id = "internal-only-marker-is-freetext"
title = "The internal_only flag is a substring inside quote_or_claim, not a column"
effect = "context"
type = "coverage"
status = "open"
discovered = "2026-07"
detection = "grep district_source_notes.quote_or_claim (or the SEED list) for the literal substring 'internal_only'. 1 of 18 loaded SOMSD rows carries it, in a parenthetical flagging parent reactions as internal_only."
misuse = "Querying district_source_notes directly and treating every row as public-safe. The public profile and budget-report renderers (the district-notes module, the budget local-record panel, and the note-candidate filter in the profile generator) skip rows whose quote_or_claim contains 'internal_only', but the reporter source table keeps those rows, and nothing in the schema marks them structurally — rename or rephrase that parenthetical and the row silently becomes publishable."

[issue.scope]
tables = ["district_source_notes"]
rows = "quote_or_claim rows containing the substring 'internal_only'"
```

Rows whose claim text is marked `internal_only` stay in the reporter source
table but are filtered before public profile or budget-report rendering.
There is no dedicated boolean column for this — it's a text convention
inside `quote_or_claim`, checked by three separate string searches in
`build_profiles.py`, so it lives or dies with the exact substring.

### Local warehouse paths in source_url render without a link

```toml ergo
[issue]
id = "source-url-non-http-not-linked"
title = "Local warehouse paths in source_url render without a link"
effect = "context"
type = "format"
status = "open"
discovered = "2026-07"
detection = "source_url does not start with http:// or https://. 3 of 18 loaded SOMSD rows (all source_type = state_data) carry the local path ed/data/kg/ed_kg.db; the other 15 are http(s) URLs (YouTube, Village Green)."
misuse = "Assuming every loaded note has a clickable public source. The public template’s public-URL test only linkifies values starting http:// or https://, so state_data rows citing the internal warehouse path render as an unlinked source line."

[issue.scope]
tables = ["district_source_notes"]
columns = ["source_url"]
```

Source URLs are only linked publicly when they are `http://` or `https://`
URLs; local warehouse paths are not rendered as links. This is deliberate —
`state_data` rows cite an internal file, not a public page — but it means
`source_url` is not a uniformly clickable column.

## Source attribution format

The public-facing section (id `district-notes`, title "District knowledge")
on a district profile renders each note as:

- metadata line (date · source type · storyline · confidence)
- the attributed text or claim
- speaker byline (if a named speaker)
- source link

Filtering (`district-source-notes/internal-only-marker-is-freetext`) and
link-eligibility (`district-source-notes/source-url-non-http-not-linked`)
rules apply before this renders. The section links to
`/data/district-source-notes/` for methodology detail.

Full district budget reports also render a gated "Local public-record
context" panel when budget-relevant notes exist — see
`district-source-notes/complementary-not-ufb-revision` for what it is and
isn't. The strongest selected note is also promoted into the full report's
front-brief verification card and budget-machine first read so the local
journalism handoff is visible before the long record table.

## Currently loaded districts

| District | Code | Notes |
|---|---|---|
| South Orange-Maplewood SD | 4900 | 18 rights-checked rows (14 `public_record`, 4 `link_quote`) covering budget structure (15 rows, storyline `budget`) and arts-funding cuts (3 rows, storyline `arts-funding`) |

## Validation

```toml ergo
[validation]
date = "2026-07-10"
method = "confirmed from the builder source — seed validation, the CREATE TABLE CHECK constraint, and the publish-time rights check"
result = "rights is constrained to {public_record, link_quote} at three independent points: validate_seed() raises ValueError before any DB write, CHECK(rights IN ('public_record','link_quote')) rejects the INSERT at the table level, and check_notes_rights.py exits non-zero if a bad value ever reaches the live table."
```

```toml ergo
[validation]
date = "2026-07-10"
method = "confirmed by loading the builder's seed list directly (18 rows, all district_code 4900)"
result = "rights: 14 public_record / 4 link_quote. confidence: 10 'high' / 8 'medium' — no row uses the page's documented A/B/C scale. source_url: 3 rows (all source_type=state_data) carry the local path ed/data/kg/ed_kg.db rather than an http(s) URL; the other 15 are http(s) links. storyline: 15 budget / 3 arts-funding. 1 row's quote_or_claim contains the literal substring 'internal_only'."
```

## Changelog

```toml ergo
[change]
date = "2026-07-10"
note = "Converted to ergo format: 6 issues registered — including the discovery that loaded confidence values are high/medium, not the documented A/B/C letters."
issues = ["confidence-values-are-not-letters"]
```