p-066 — DevOps: CI coverage, GitHub Pages, releases¶

Status: done

Created: 2026-08-05

Refs: pi-container devops practices (CI, GitHub Pages, releases, coverage badges, release skill).

Context¶

This repository has a working CI (ci.yml — lint + format + mypy + tests + generated-artifact checks), a Makefile, and a pre-commit hook installed via core.hooksPath. What it is missing, by comparison with pi-container's devops baseline, is:

Coverage measurement and badge publishing. CI runs pytest but never measures coverage, never enforces a floor, and never publishes a badge.
GitHub Pages documentation site. The README is the only hosted documentation; there is no MkDocs Material site behind github.io.
Release workflow. There is no CHANGELOG.md, no version tag convention, no GitHub Release creation on tag push.
Release skill. pi-container has a .pi/skills/release/SKILL.md that lets pi drive the release; this repo has nothing.
Semantic versioning. pyproject.toml declares a version and a [build-system] with hatchling. The tag is the release identifier; the version in pyproject.toml must stay in sync.
Multi-page documentation site. docs/ already contains substantial reference material (mapping docs, debugging guide, validation strategy, observability) that should be accessible from the hosted site, not just the README.
Dependency automation. Python dependencies in pyproject.toml and the third_party/marc2bibframe2 submodule both need update mechanisms — dependabot for the former, manual review for the latter.

The BFFI pipeline's constraints shape the design:

No paid services. Everything must run on GitHub-hosted runners with free-tier minutes.
Pro bono. Minimal operational overhead — the NLF reviewer reads the mapping docs, not the badge.
Generated artifacts are committed. The mapping docs and field-coverage corpus live in the repo and are --check-guarded. Submodule bumps of third_party/marc2bibframe2 also trigger doc regeneration.
No Docker images to publish. Unlike pi-container, this repo ships Python + XSLT + RDF; there is no registry to publish to. Git tags are the release boundary.

Design decisions¶

1. CI: coverage job + badge auto-publish¶

What: A new coverage job in ci.yml that: - Runs pytest --cov --cov-report=xml --cov-report=term-missing. - Generates a Shields.io-style coverage.svg badge into docs/assets/coverage.svg. - On push to main, auto-commits the badge SVG and pushes it back to main.

Why this matches pi-container: identical pattern — badge SVG committed to docs/assets/coverage.svg on every main push, with github-actions[bot] as the committer.

Differences from pi-container: - Uses coverage.py directly (no separate generate_coverage_badge.py dependency beyond what pytest-cov already brings in). - The badge SVG is hand-crafted (same Shields.io look), not generated by a library.

What to skip: pi-container's validate_versions.py cross-checks git tag / pyproject.toml / schema config. This repo has no schema_version (no seeded config template), so the remaining check — "does the git tag match pyproject.toml version?" — is thin enough to enforce via the release skill's all-or-nothing commit (version bump + tag in one atomic operation). If someone manually breaks that invariant, the next release run catches it. Don't build it.

Badge generation: Copied from pi-container (.github/scripts/generate_coverage_badge.py). Uses coverage library to read .coverage file and generate Shields.io-style SVG. No need to implement from scratch.

2. GitHub Pages: MkDocs Material¶

What: A gh-pages.yml workflow that: - Triggers on push to main. - Generates docs/index.md from README.md (link rewriting for MkDocs-relative paths). - Builds with mkdocs build --site-dir site. - Deploys to GitHub Pages via actions/deploy-pages@v4.

Why this matches pi-container: identical pattern — README-as-index, MkDocs Material, actions/deploy-pages@v4, github-pages environment.

Differences from pi-container: - The site is multi-page: docs/index.md (auto-generated from README) is the home page, and existing docs/*.md references are hand-added to the nav:. - The generate_index.py script rewrites docs/assets/… → assets/… and root-level file links → blob:main GitHub URLs. This repo has fewer such paths to rewrite.

File: mkdocs.yml (minimal, single-page), .github/scripts/generate_index.py, .github/workflows/gh-pages.yml.

3. Release workflow¶

What: A release.yml workflow that: - Triggers on push matching refs/tags/v*. - Waits for check and coverage jobs to pass. - Creates a GitHub Release via softprops/action-gh-release@v2 with auto-generated release notes.

Why this matches pi-container: identical pattern — softprops/action-gh-release, auto-notes from merged commits since last tag, triggers on v* tags.

Differences from pi-container: - No pyproject.toml ↔ schema_version cross-check (only one version source). - No Docker image publishing step (no images to build). - The release is purely a git tag + GitHub Release annotation.

4. CHANGELOG.md + semantic versioning¶

What: A CHANGELOG.md following Keep a Changelog format with an [Unreleased] section. Versions follow Semantic Versioning: vMAJOR.MINOR.PATCH.

Why this matches pi-container: identical pattern — Keep a Changelog format, [Unreleased] block, date-stamped version sections. Plus semantic versioning with the same three-place bump as pi-container (git tag ↔ pyproject.toml version sync).

Differences from pi-container: no schema_version to sync (no seeded config template). Only two places stay in sync: git tag and pyproject.toml version.

5. Pre-commit config¶

What: Formalize the ad-hoc .githooks/ hook into a proper .pre-commit-config.yaml plus a pyproject.toml [tool.pre-commit] or just rely on the Makefile-based hook.

Decision: Keep the Makefile-based approach. The existing hook runs make lint && make test (ruff + mypy + pytest) on *.py changes. Adding .pre-commit-config.yaml on top of that would duplicate work. The pi-container pattern uses .pre-commit-config.yaml because it has multiple hook groups (src, proxy, docs) — this repo has one group. Stay with make lint && make test as the gate.

6. Release skill + semantic version bump¶

What: A .pi/skills/release/SKILL.md (matching pi-container) that lets pi drive the release process, including the version bump.

Why this matches pi-container: identical concept — pi drives version bump, changelog, tag, push.

How it differs from pi-container's release skill: - pi-container's skill bumps pyproject.toml version + schema_version in two config files + regenerates uv.lock + runs validate_versions.py. - This repo's skill bumps pyproject.toml version + updates CHANGELOG.md + regenerates uv.lock + runs validate_versions.py + tags. No schema_version (no seeded config template). - Steps: (1) summarize recent commits into a changelog block, (2) move [Unreleased] to a dated version, (3) bump pyproject.toml [project].version, (4) uv lock, (5) lint + test, (6) amend commit, (7) git tag -a v<X.Y.Z> -m "Release v<X.Y.Z>", (8) git push origin main && git push origin v<X.Y.Z>.

7. `.gitattributes`¶

What: LF normalization for text files, binary for SVG/lockfiles.

Why this matches pi-container: identical pattern — text=auto, explicit eol=lf for .py/.sh/.yaml/.yml/.md/.toml, binary for .svg and .lock.

Phases¶

Phase A: Coverage + badge¶

[ ] Add pytest-cov to [dependency-groups].dev.
[ ] Add coverage job to .github/workflows/ci.yml:
Run pytest --cov --cov-report=xml --cov-report=term-missing.
Generate docs/assets/coverage.svg.
Auto-commit + push badge on push to main.
[ ] Add docs/assets/.gitignore to exclude the badge from normal .gitignore (it is committed by CI, not by developers).
[ ] Add coverage badge URL to README.md (top, alongside existing shields).
[ ] Set fail_under in pyproject.toml [tool.coverage.report].

Phase B: GitHub Pages (multi-page MkDocs)¶

[ ] Add mkdocs-material to [dependency-groups].dev.
[ ] Create mkdocs.yml with nav: entries for the existing docs/*.md references:
Home: index.md (auto-generated from README)
Mapping references: bf_to_bffi_mapping.md, bffi_to_marc_mapping.md, marc_to_bibframe_mapping.md
Debugging: roundtrip-debugging.md
Validation: validation-strategy.md
Observability: observability.md
Plans: plans/index.md (links to docs/plans/README.md)
Vocabulary: a note page pointing to vocab/lkd.rdf
[x] Copy .github/scripts/generate_index.py from pi-container (with REPO_URL and FRONTMATTER adapted for BFFI).
[ ] Create .github/workflows/gh-pages.yml (MkDocs build + actions/deploy-pages@v4).
[ ] Add site/ to .gitignore (generated, committed by CI).
[ ] Update README.md with GitHub Pages badge.

Phase C: Release workflow + CHANGELOG + semantic versioning¶

[ ] Create CHANGELOG.md with [Unreleased] and seed first version block from git history.
[ ] Create .github/workflows/release.yml (triggers on v* tags, creates GitHub Release).
[ ] Update CI ci.yml to include release job gated on tag push (only after check + coverage pass).
[ ] Update release skill to bump pyproject.toml version in addition to changelog + tag.
[ ] Add fail_under to pyproject.toml [tool.coverage.report].

Phase D: Release skill + version bump¶

[ ] Create .pi/skills/release/SKILL.md.
[ ] Create .pi/skills/release/scripts/release.sh:
Summarize recent commits into a changelog block.
Move [Unreleased] to a dated version header.
Bump pyproject.toml [project].version.
Regenerate uv.lock (uv lock).
Run validate_versions.py + lint + tests.
Amend commit with changelog + version bump.
Tag v<X.Y.Z> and push (main + tag).
[ ] Create .github/dependabot.yml:
python ecosystem for pyproject.toml dependencies.
github-actions ecosystem for action versions.
[ ] Configure submodule update discipline for third_party/marc2bibframe2:
Operator reviews bump diff manually (not automated via dependabot).
After bump: regenerate mapping docs (--check catches drift), update field-coverage corpus if XSLT surface changed, commit together.

Phase E: `.gitattributes` + dependabot submodule discipline + polish¶

[ ] Create .gitattributes (LF normalization, binary for SVG/lock).
[ ] Document submodule update discipline in CLAUDE.md (under "Conventions"):
third_party/marc2bibframe2 is vendored via git submodule; never modify it.
Submodule bumps are a separate release concern: regenerate docs, run --check, commit together.
[ ] Update README.md with all new badges (coverage, pages, release, semver).
[ ] Update CLAUDE.md operating constraints with semantic versioning convention.
[ ] Add badges to README: coverage, GitHub Pages, license, Python version, semver.

Multi-page MkDocs site (expanded scope)¶

The README-as-index is the landing page, but docs/ already contains substantial reference material that should be accessible from the site:

docs/bf_to_bffi_mapping.md — forward-direction routing decisions
docs/bffi_to_marc_mapping.md — reverse-direction emit + known limitations
docs/marc_to_bibframe_mapping.md — marc2bibframe2 XSLT coverage
docs/roundtrip-debugging.md — failure-pattern catalogue
docs/validation-strategy.md — three validation boundaries
docs/observability.md — local metrics stack
docs/plans/README.md — plans index
vocab/lkd.rdf — the BFFI ontology (rendered as a reference, not a doc page)

generate_index.py keeps generating docs/index.md from README.md for the home page. The deeper docs are added to the MkDocs nav: as hand-maintained entries (they are not auto-generated — the mapping docs already exist as standalone Markdown). CI still regenerates only docs/index.md.

Semantic versioning (expanded scope)¶

This repo does publish to PyPI via the [project.scripts] entry point (bffi-pipeline), and the [build-system] declares hatchling. Even though the primary distribution channel today is uv sync from git, the version in pyproject.toml is the canonical release identifier and should follow Semantic Versioning:

vMAJOR.MINOR.PATCH
 ^      ^     ^
 |      |     └─ Patch: bug fix, no new features, no breaking changes
 |      └─────── Minor: backwards-compatible feature addition
 └────────────── Major: breaking change

0.x.y is pre-release. 1.x.y is stable. The release skill bumps the version in pyproject.toml, updates the CHANGELOG, and tags v<X.Y.Z> — matching pi-container's three-place sync (git tag, pyproject version, schema version).

This repo has no schema_version (no seeded config template to ship), so the sync point is two places: git tag ↔ pyproject.toml version. The validate_versions.py check from pi-container collapses to "does the tag match pyproject.toml?", which is still worth running in CI.

Dependabot / Renovate (expanded scope)¶

Automated dependency updates for uv.lock and third_party/ submodule pins. Two mechanisms, both enabled:

Dependabot (.github/dependabot.yml) for Python dependencies declared in pyproject.toml. Opens PRs against main with uv lock regeneration.
Manual submodule management for third_party/marc2bibframe2. Dependabot can bump git submodules, but the submodule is a vendored LoC stylesheet with its own release cadence and breaking-change surface. Treat submodule bumps as a separate concern: the operator reviews the diff, regenerates mapping docs (--check will catch drift), and lands them in their own commit.

Outcome¶

All phases shipped. Deviations from the plan are noted per phase.

Phase A: Coverage + badge¶

Shipped: pytest-cov in dev deps, coverage job in ci.yml (runs on push to main, gated on lint-and-test), docs/assets/coverage.svg committed, fail_under = 80 in pyproject.toml, badge linked in README.

Deviations: No docs/assets/.gitignore — the badge is committed normally alongside other assets. The plan called for excluding it from normal tracking; that distinction turned out to be unnecessary.

Phase B: GitHub Pages¶

Shipped: mkdocs-material + mermaid2 plugin in dev deps, full mkdocs.yml with grouped navigation, .github/scripts/generate_index.py, .github/workflows/gh-pages.yml, site/ in .gitignore, README badge linking to coverage section.

Deviations from plan: - mkdocs.yml nav is more elaborate than the plan's "minimal, single-page" — grouped into Runbook (Getting Started, Development, Debugging, Observability), Pipeline docs (Overview, Mapping references, Schema), and Plans. Includes pipeline-overview.md with mermaid architecture diagrams. - mermaid2 plugin added for interactive architecture diagrams in the pipeline overview. - Logo set to assets/mark-dark.svg, favicon to assets/apple-touch-icon.png — not in the plan.

Phase C: Release workflow + CHANGELOG¶

Shipped: CHANGELOG.md with [Unreleased] block, .github/workflows/release.yml (triggers on v* tags, runs its own check job then creates GitHub Release via softprops/action-gh-release), fail_under = 80 in pyproject.toml.

Deviations from plan: The release workflow is a separate release.yml triggered on tag push, not a release job inside ci.yml gated on tag push (as the plan described). The release.yml has its own check job that runs lint + format + mypy + artifact checks + tests, then a release job that creates the GitHub Release. This is cleaner separation — the main CI runs on every push, the release CI only on tags.

Phase D: Release skill¶

Shipped: .pi/skills/release/SKILL.md, .pi/skills/release/scripts/release.sh (summarizes commits, bumps version, regenerates uv.lock, lints, tests, amends, tags, pushes), .github/dependabot.yml (python + github-actions ecosystems), submodule discipline documented in CLAUDE.md Conventions.

Deviations: None significant. Matches the plan.

Phase E: .gitattributes + dependabot + polish¶

Shipped: .gitattributes (LF normalization, binary for SVG/lock), submodule discipline in CLAUDE.md Conventions, coverage badge in README.

Deviations from plan: - GitHub Pages badge not added to README — the plan called for it but it wasn't implemented. - Semver badge not added to README — same. - CLAUDE.md updated with semantic versioning convention and submodule discipline (as planned).

What the plan got wrong¶

The validate_versions.py callout in the plan was correct — it's intentionally thin (only tag ↔ pyproject sync, no schema_version) and the release skill handles it inline. The plan's call to "don't build it" held.

Out of scope (noted, not deferred)¶

Docker image publishing. This repo has no container images to publish (contrast: pi-container publishes pi-coding-agent:local etc.).
Renovate. Dependabot covers all dependency update needs; no need for a second tool.

p-066 — DevOps: CI coverage, GitHub Pages, releases¶

Context¶

Design decisions¶

1. CI: coverage job + badge auto-publish¶

2. GitHub Pages: MkDocs Material¶

3. Release workflow¶

4. CHANGELOG.md + semantic versioning¶

5. Pre-commit config¶

6. Release skill + semantic version bump¶

7. .gitattributes¶

Phases¶

Phase A: Coverage + badge¶

Phase B: GitHub Pages (multi-page MkDocs)¶

Phase C: Release workflow + CHANGELOG + semantic versioning¶

Phase D: Release skill + version bump¶

Phase E: .gitattributes + dependabot submodule discipline + polish¶

Multi-page MkDocs site (expanded scope)¶

Semantic versioning (expanded scope)¶

Dependabot / Renovate (expanded scope)¶

Outcome¶

Phase A: Coverage + badge¶

Phase B: GitHub Pages¶

Phase C: Release workflow + CHANGELOG¶

Phase D: Release skill¶

Phase E: .gitattributes + dependabot + polish¶

What the plan got wrong¶

Out of scope (noted, not deferred)¶

References¶

7. `.gitattributes`¶

Phase E: `.gitattributes` + dependabot submodule discipline + polish¶