Changelog

Added

• Standalone binding assays and typed endpoints with explicit units, censored and qualitative results, provenance and preparation links, separate from degradation.
• Public binding API and browse page, dedicated dataset/release tables, and revision/withdrawal handling that preserves existing degradation labels.
• Binding vocabulary and adsorption-isotherm, TIRF and FRAP methods.

Added

• Explicit reviewed parent-to-variant reclassification, with pinned source evidence, actor definitions, complete preparation coverage and exact assay conservation.
• Typed, durable reclassification audits and direct public redirects to the variant, including visibility checks and subsequent variant merges. Historical endpoints survive orphan cleanup and source-paper deletion.

Fixed

• Curator-reviewed parent consolidation can preserve explicitly mapped terminal constructs without broadening automatic sequence matching. Sequence hashes, source evidence and variant definitions are checked again before applying.
• Identical parent reference coordinates can retain existing fusion variants without inventing sequences or treating fusions as point mutants. Conflicted structures can remain on archived records with an explicit audit trail.
• Reviewed variant folds can compare tagged sequences in a common core frame, checking mutation coordinates while retaining each original sequence and offset.
• Paper revisions retain historical sequence-frame records rehomed from merged accessions, while still retracting dropped assertions about current records.

Fixed

• Parent protein merges leave previously retired variants on their original parents, preserving their redirects, sequences and merge audits. Retired variants no longer create active fold collisions or move to the survivor.
• A retired variant already on the survivor still reserves its name; incoming live collisions remain explicitly refused under the existing unique index.

Added

• Separate source-scoped protein constructs and enzyme preparations, with explicit reference/expressed/assayed sequence roles, coordinate frames, separate chains, tags/linkers/fusions, modifications and paper evidence. Missing sequences and unknown tag status remain explicit.
• Experiment-to-preparation associations preserve roles and amounts for multiple preparations, including standalone enzyme characterisations. Curator editors and extraction contracts retain the complete material descriptions.
• Public construct/preparation views, actor-page material panels, and four additive downloadable/snapshot tables, filtered by published actor/evidence.
• Exact legacy variant sequence snapshots before approved variant folds, with source attribution and explicit structure-reference associations.

Fixed

• Promotion, revision, withdrawal, orphan cleanup and merges preserve or retract material evidence with its source. Revising one ingestion no longer removes a shared interaction still owning another ingestion's experiments.

All new tables ship through the existing schema-init release step. No automatic corpus-wide reinterpretation or merge is performed. See docs/construct_preparation_layer.md for compatibility and migration policy.

Identity sources, sequence frames and curator-only correspondence. **Migration 0032_identity_sources** (three new tables; python -m app.db.init_schema then alembic upgrade head). Deploy before the producer starts sending the fields; the producer strips them until /openapi.json advertises them. See DECISIONS.md §9.1.

Added

• identity_sources on protein, protein_variant and protein_variant.parent_protein in the extraction contract: where an identity value (sequence, accession, parent, mutations) came from when it was not the paper — cited_paper, author_correspondence, reference_record, genome_recovery, database_lookup (plus paper/supplement), with who/when/reference, verified_against and an optional evidence file + sha256. Absent means "from the paper"; every existing payload stays valid. Vocabulary at GET /api/v1/vocabularies (identity_source) and the vocabulary catalog.
• sequence_frame (precursor / mature / construct / pdb_chain), construct_span and construct_tags on the same three objects. Recorded for curators; not used for dedup.
• Promotion records both per (entity, asserting ingestion) in identity_sources / sequence_frames, with the entity's accession and the asserted values frozen on the row. A replace-mode revision retires the superseded runs' rows; protein merges and variant folds re-home them onto the survivor; orphan cleanup removes them with their entity.
• Correspondence upload (correspondence on POST /ingestions/{id}/files) into ingestion_correspondence_files under a private prefix — never a supplement. Curator-only list and download (GET /ingestions/{id}/correspondence[/{file_id}], attachment, private, no-store). Linked from an entry by name and hash at approval.
• Audit: GET /api/v1/admin/identity-sources (curator; filter by source, accession — current or as asserted) and GET /api/v1/admin/sequence-frames.
• Curation form controls for every new leaf (121 → 157), and a small "Not from the paper: supplied by the corresponding author (sequence)" marker under the sequence on protein and variant pages — the source type only.

Security

• Identity provenance is redacted for non-curators. POST /ingestions, PATCH /ingestions/{id} and POST /ingestions/{id}/upload-markdown accept any signed-in account and echo extracted_data; every such response now goes through one helper (IngestionResponse.for_viewer → redact_identity_provenance), which cuts each identity_sources entry to {source, fields}. Curators still see everything, and the stored record is untouched. A structural test fails if a non-curator route builds an unredacted ingestion response. Follow-up, out of scope here: those producer routes still accept any CurrentUser; they should require an ingest-capable role.

Fixed (review)

• Linking and replacing evidence are serialised. Approval's evidence lookup, the correspondence replace and the delete all lock the file row (FOR UPDATE), so a replace can no longer swap bytes under a link being made. Replaced objects are deleted only after the request commits, so a failed multi-file upload never destroys an existing file. Every write gets its own object key (content hash plus a per-write version), cleanup re-checks after the commit that no row references a key before deleting it, and one request naming the same correspondence file twice is refused (422). A request that fails before committing (a later part's 413/409/422, a storage error) deletes what it wrote. A commit that raises may still have committed, so then nothing is deleted: the keys are logged and left for reconciliation. POST /ingestions/{id}/files sets 30 s SET LOCAL statement/lock/idle-in-transaction timeouts as its transaction opens and accepts at most 10 correspondence parts, so the window from the first correspondence write to the commit is at most (10 × 4 + 2) × 30 s = 21 min — under the reconcile script's 1 h minimum age. python -m scripts.reconcile_correspondence_objects (dry run by default, --apply, --older-than-hours ≥ 1, default 24, each key re-checked in a fresh query right before deletion) lists and removes any object under private/correspondence/ that no row references, e.g. after a crash. Key versions are a full uuid4 — so neither a duplicate in one request nor two interleaved replacements can delete the object the final row points to.
• Distinct assertions stay distinct. The per-payload dedup key includes the normalised asserted values, so two different accessions asserted for one sequence-folded protein are two audit rows.
• Linked evidence is immutable. Re-uploading a correspondence file under the same name with different bytes is a 409 once an approved entry links it; the same bytes again are a no-op. An unlinked file is replaced and its old object removed.
• DELETE /ingestions/{id}/correspondence/{file_id} (curator): removes an unlinked file and its object; a linked one is refused (409).
• Non-Latin-1 filenames download. Content-Disposition carries an RFC 5987 filename* and an ASCII fallback; an em dash in an email subject was a 500.
• Clearing a construct span in the form ({start: null, end: null} or {}) means "no span" instead of a 422; spans are bounded (≤ 100 000) so an overflow is a 422, not a database error.
• Approve reports evidence that did not link (identity_evidence_unlinked, plus a log line).
• A revision that drops an entry clears the public marker: the superseded rows' proteins are invalidated in the cache.
• 0032's downgrade documents and logs that it drops the provenance rows.

Security: the public chatbot's SQL runs as a least-privilege database role that can read only public-rows-only views, so the regex SQL guard is no longer the security boundary. Migration 0031_chat_public_views (creates schema chat_public and its views; touches no table). Needs a deploy (API + migrate job), then a one-time role provisioning and a new secret — until then the chat keeps today's path and warns loudly. See DECISIONS.md §11, "The chatbot reads views, as its own role".

Security

• Chat SQL runs as plasticdb_chat, a dedicated login role whose only privileges are USAGE on schema chat_public and SELECT on its views (via the NOLOGIN plasticdb_chat_reader). It has no privilege on any table in public, so tables added later — including the identity-provenance tables of #605 — are unreachable by default. The role is read-only by default, statement-time-limited, and pinned to search_path = chat_public.
• The guard had been bypassed through PostgreSQL lexical features (escape-string literals, unicode-escaped identifiers, schema-qualified relation names, and SQL-looking text inside string literals); one class allowed an anonymous chat request to read non-public rows. With the role in place a guard bypass reaches nothing private. The guard also now refuses those lexical forms outright, refuses quoted identifiers and the XML table functions, and scans with string-literal contents blanked.

Added

• app/db/chat_public.py: the chat_public views (security barrier; public rows only, mirroring app.services.query; only the columns the chat needs). Built by migration 0031, by create_schema on a fresh database, and rebuilt by python -m app.db.init_schema on every deploy.
• app/services/chat_db.py: a separate NullPool engine from CHATBOT_DATABASE_URL, read-only transaction per query. Chat SQL never uses the application session when it is set.
• CHATBOT_REQUIRE_RESTRICTED_ROLE (default false): when true, chat SQL is refused unless CHATBOT_DATABASE_URL is set. Flip it after the role is deployed. CHATBOT_STATEMENT_TIMEOUT_MS (default 15000).
• scripts/chat_restricted_role.sql (idempotent; superuser; after the release is deployed) and python -m scripts.provision_chat_role (dry run unless --apply; password from --password-stdin or CHATBOT_DB_PASSWORD, sent as a SCRAM verifier; connection errors are reported without the URL or exception text). Provisioning repairs unexpected memberships, PUBLIC/chat table, column and sequence grants, CREATE on schemas (revoked from PUBLIC), default ACLs, PUBLIC EXECUTE on application and extension functions, and the implicit PUBLIC EXECUTE default for new functions; grants SELECT on the approved views by name; and then verifies the boundary — memberships (direct and transitive), ownership, the exact chat_public inventory and each view's definition against app.db.chat_public, column and sequence privileges, catalog objects by provenance (stock or allowed extension only) plus an explicit sensitive-catalog deny list, executable functions, effective function defaults, large objects and default ACLs — failing the whole run on anything unexpected. Its output never includes anything derived from the connection URL.

Changed

• The chatbot's schema hint describes the chat_public views (same names and columns as the tables they front).
• With no CHATBOT_DATABASE_URL, chat SQL keeps the previous path and logs a SECURITY: warning at startup and on every query.

Removed

• scripts/chatbot_readonly_role.sql, which was never wired in, granted SELECT on base tables and carried a placeholder password. The new script disables its plasticdb_chatbot role if it exists.

Review fixes to the containment detector (1.53.0). Report-only, no migration, no write path. Needs a deploy (API).

Fixed

• Per-request all-pairs cost. The containment report is now cached, keyed by a digest of the report inputs (sequences, tags, stored cleavage sites), and the all-pairs search runs in a worker thread, off the event loop.
• TEV scar (ENLYFQ[GS]) is stripped only alongside a His run, so a real C-terminus is no longer trimmed.
• Leading Met is kept on post-SignalP chains; core_sequence agrees with frame_rows after a cut.
• Stored cleavage sites are used as residue coordinates on the normalised chain; offsets are computed on the normalised string.
• Containment plan nulls the survivor, classification and move fields it cannot honestly fill.

Containment detector for proteins stored in different sequence frames. Report-only: no migration, no write path, no change to identity, ingestion or merging. Needs a deploy (API) to show on the Duplicates page.

Added

• frame=containment on GET /api/v1/curation/duplicates and a "Containment (frames)" tab on /curation/duplicates: clusters of live proteins whose core chain (mature chain minus observed C-terminal tags and the leading Met) is a proper substring of another row's stored chain, each edge classified mature_equivalent (±3 aa of the SignalP site), tag_trim, construct_in_precursor or truncation, with the merge planner's refusals shown for context. Every cluster carries frame_unresolved and none is ever mergeable. Near-identical pairs (1–2 substitutions, ≥98%) are listed separately as never-duplicates.
• app/services/sequence_core.py: the pinned, evidenced tag list and strip_terminal_tags() / core_sequence() — pure and stdlib-only, so step (C) can move it into plasticdb_ml.frames. Strips (LE|GS)?H{6,12} only with ≥6 consecutive His, then a C-terminal ENLYFQ[GS] TEV scar, then one leading Met; capped; never writes back. Real C-termini such as #1264 …CPFALE and #665 …CPFAL are untouched.
• app/services/protein_containment.py (detector) and python -m scripts.report_protein_containment (--format json, --dump for an offline JSON export). Over the 2026-10-30 dump: 39 clusters / 89 rows, 84 edges, 59 near-identical pairs; all 67 edges of the frames investigation reproduced (differences explained in DECISIONS.md).

Changed

• protein_dedup.entity_key raises on the new FRAME_CONTAINMENT, so the containment frame cannot reach SequenceIndex, group_by_sequence or a merge planner.

CI only. No migration, no runtime change.

Fixed

• Deploy gate logs live. .github/wait-for-release.py printed through a block-buffered stdout on the Actions runner, so the v1.51.1 deploy (the first run that actually waited: a test release row held it for five polls) showed every "is running ... checking again in 60s" line at once, when the gate finished. Each status line is now flushed as it is written, so a held deploy shows that it is waiting, and on what, while it waits.

Calibration status on HMM screen results. No migration. Needs a deploy (API) and a release-worker image; hits show no badge until a release has published a calibration.json.

Added

• The release publishes calibrated/calibration.json beside PlasticDB.hmm (hmm_library.OPTIONAL_ARTEFACTS), in the same atomic loop and before the fingerprint, so a search never sees a library and a calibration record from different builds. hmm_publish.verify_emission checks it against the manifest checksum when present; a build without one still publishes and removes the previous library's copy. scripts/publish_hmm_library goes through the same publish() and reports it. The PlasticDB-ML bundle's own (stale) calibration.json is never used.
• app/services/hmm_calibration_status.py: loads calibration_status / calibration_status_reason per profile from the published file, cached per file version. An absent or unreadable file, rows without the fields (pre-0.11 format) or an unknown status all mean "unknown": no badge.
• HMM screen results: GET /api/v1/jobs/{id} hits carry calibration_status and calibration_status_reason (null when unknown); the job page shows a neutral "untestable — no negative set" label (reason as tooltip) and a warning "not separable" badge; separable shows nothing. Hits from not_separable profiles sort below the rest of the same query (API, page and TSV; otherwise the order is unchanged). The HMM TSV gains a trailing calibration_status column. There is no per-family or per-profile page in the app, so the hit table is the only surface.

Dependency pin. No migration. Needs a release-worker image for the next release to pick it up.

Fixed

• plasticdb-ml pinned to v0.11.1 in requirements.txt and requirements-notebooks.txt (PlasticDB-ML #35). Notebook 01's Matrix B example located its variant and parent by internal ids (VARIANT_ID, PARENT_ID = 329, 658), the same failure mode that killed release 2026.44.7 when a duplicate merge tombstoned protein 21. It now follows NylB'-SCY (PBDAPN) and NylB' (PB95X3) by accession and skips the example, with a note, if either is absent. The notebook-id lint now catches tuple, chained, annotated and walrus assignment.

Variant merge: "one sequence = one enzyme" (#576) for protein variants. Migration 0030_variant_tombstone (schema only): adds is_tombstoned (NOT NULL DEFAULT false) and merged_into_accession (indexed) to protein_variants; no row changes. Run python -m app.db.init_schema before alembic upgrade head, as usual (it creates the new variant_merges table). Nothing is merged by the release itself: run python -m scripts.merge_duplicate_variants (dry run) and read the plan first.

Added

• app/services/variant_merge.py. Groups live variants of one parent with one canonical sequence in one frame (protein_dedup.entity_key: precursor = the stored record, mature = after the variant's own SignalP cut). Identical sequences under different parents are reported separately and never merged. Built from the protein merge's parts rather than beside them: refusal vocabulary, classify_verdicts, the decisions file, the variant-to-variant fold (protein_merge.fold_variant_rows, extracted from _fold_variant), structure-link relinking (relink_folded_variant_structures, extracted from _relink_parked_structures) and EvidenceLostError.
  • Survivor: never a tombstone, then an accessioned row, then most evidence (interactions, experiments, observations, kinetics + properties, structures), then most complete (filled columns, mutation list, synonyms), then lowest id. A decision (or the admin endpoint) may name another.
  • Moves every row with a foreign key to the retired variant onto the survivor: interactions (with their experiments and observations), kinetics, properties, job hits, structures, enrichments, mutations. A structure or enrichment whose unique slot the survivor already fills, and a mutation list when the survivor has one, stay on the tombstone; nothing is deleted. The survivor fills its NULLs from the retired row and gains its name and synonyms as synonyms. The census over every referring table and the (interaction, paper, polymer, result) ledger must be identical before and after, or the group rolls back.
  • Tombstone: the retired row keeps its name, mutations and its (parent, name) slot; is_tombstoned + merged_into_accession, registry entry merged, one variant_merges audit row.
  • Refusals: verdict_conflict / verdict_disagreement (lifted by a decision's accept_verdict_disagreement), variant_type_conflict, mutation_conflict (mutation lists compared on the residue they name: position + numbering_offset − the frame's signal-peptide cut when both offsets are known, the paper's positions otherwise), numbering_conflict (the survivor has no list and moving the retired row's list would re-number it), sequence_mismatch / parent_mismatch (admin endpoint only), tombstoned, no_accession, excluded, decision_mismatch / decision_invalid (a variant decision takes only survivor, retire, accept_verdict_disagreement, note).
• scripts/merge_duplicate_variants.py: plan by default, --apply, --frame, --only, --exclude, --decisions, --limit; one group per transaction through commit_with_deadlock_retry; invalidates the parents' protein cache.
• variant_merges audit table (new table, ORM). Both variant columns keep their row alive in the orphan sweep.
• /api/v1/curation/duplicates → variants: the variant groups (V01…) with plan, survivor, refusals and moves; cross_parent (one sequence under several parents); recent variant merges. The /curation/duplicates page renders all three.

Changed

• POST /api/v1/admin/accessions/merge accepts variant accessions. Two variants of one parent run the variant merge; one variant and anything else, the same variant twice, or variants of different parents are 422; a refused plan is 409 with the plan. New body fields: dry_run (returns the plan, writes nothing; for proteins it reports and writes nothing), accept_verdict_disagreement and note (variants). Protein and organism merges are otherwise unchanged.
• A merged variant's accession redirects. GET /api/v1/variants/{accession|id} (and /fasta) answers 301 to the survivor; /api/v1/proteins/{accession} and /proteins/accession/{accession} 301 a merged variant's accession to /api/v1/variants/{survivor}; the /variants/{id} and /proteins/{accession} pages 301 to the survivor's page. A tombstone with no survivor is 410.
• Tombstoned variants are hidden everywhere public. query.variant_parent_public_filter and is_public_variant drop them (also under include_sequenceless), and so do the protein page's variant list, the engineering view and both search variant arms.
• Promotion resolves a tombstoned variant to its survivor: a paper naming the retired variant attaches its evidence to the survivor instead of the redirect.
• Paper deletion detaches a merged tombstone it owns (paper_id → NULL) instead of cascading it away (orphan_cleanup.hand_over_shared_variants).
• 0028_variant_unique_name's unknown-foreign-key guard lists variant_merges as an audit reference (it only matters when 0028 is replayed on a create_schema() database).
• A revision never deletes a merged variant's tombstone. delete_prior_ingestion_rows detaches a tombstoned variant from the retired ingestion (paper_ingestion_id → NULL) instead of deleting it with its parked mutations, structures and enrichments; before this, re-approving the retired variant's paper turned its accession's 301 into a 404.
• A protein merge never folds a live variant onto a tombstone. A same-name clash where either row is a merged variant's tombstone is a variant_conflict refusal (a rename decision resolves it). A tombstone whose survivor a protein merge folds away is re-pointed at the kept row.
• Variant merges re-plan under lock. apply_variant_plan takes SELECT … FOR UPDATE on the member variants and their interactions, plans again, and raises PlanChangedError (endpoint: 409; script: the group fails, nothing written) if any member's evidence, verdicts, mutations or state moved since the plan.
• /api/v1/changes reports a merged variant as merged with merged_into (a withdrawn one as removed), as for proteins; never for a staged parent.
• A revision of the survivor's own paper keeps the merged names: the retired variants' names and synonyms are re-unioned into synonyms after the payload rewrites them.
• The protein/organism dry_run reports the entities it would mark and 404s an unknown retired accession. The protein page no longer lists a structure parked on a merged variant's tombstone (it showed the survivor's PDB twice). Approving a payload that names a withdrawn variant tombstone is a 422 (RetiredVariantError), not a 500.

Dependency pin. No migration. Needs a release-worker image for the next release to pick it up.

Changed

• plasticdb-ml pinned to v0.11.0 in requirements.txt and requirements-notebooks.txt (PlasticDB-ML #33). Notebook 05 reports metagenome hits above both the published GA and a per-profile background threshold fitted on the pristine control runs (leave-one-out on the controls); the published search library and its GA cutoffs are unchanged. Notebook 01 prints what the analysis population excludes and why. calibration.json rows carry calibration_status (separable / not_separable / untestable) and a reason, for a later UI label.

Build and CI, plus one small public endpoint. No migration. Needs a deploy (API) for the endpoint; the deploy gate treats its absence as "unknown" and proceeds, so the release that ships it is not held.

Added

• GET /api/v1/releases/status → {"running": bool, "version": str|null, "started_at": datetime|null}, unauthenticated, nothing else exposed. "Running" is what the release task itself treats as in progress: a running row younger than STALE_RUN_HOURS (13 h); an older one is a dead worker's and is not running. The rule is now one function, dataset_releases.is_live_run, which claim_release uses too (behaviour unchanged). tests/test_release_status_endpoint.py.
• Deploy gate in Build and Push. After every image is pushed and before the gitops bump, a release waits while /api/v1/releases/status says a dataset release is running (.github/wait-for-release.py: every 60 s, at most 7 h). A deploy restarts the release worker, which kills a release (~4.5 h) mid-way. Past 7 h it emits a ::warning:: and deploys anyway; a 404 (endpoint not deployed yet) or an API unreachable three times in a row is "unknown" and deploys with a warning, so the gate cannot wedge a deploy. Dry runs skip it. The job timeout is 540 min (was 90).
• Weekly prune of CI tags (.github/workflows/prune-ci-tags.yml, .github/prune-ci-tags.py): Sundays 03:17 UTC, deletes tags of vmar559/plasticdb matching ^ci- (PR images ci-<sha>-<role>, dry runs ci-test-<sha>*) pushed more than 14 days ago, through the Docker Hub API with the login dockerhub-private.py uses. Release (<sha>, <sha>-api, <sha>-release, <sha>-chart), base-* and buildcache-* tags never match. workflow_dispatch defaults to a dry run, and a pull request that touches the pruner runs it dry against the real registry. A token without Delete scope fails the run with a message saying so. tests/test_prune_ci_tags.py covers the selection rule and the gate.

Changed

• Monthly base refresh. The UTC year-month is now an input of the base-image tag (.github/base-image-tag.sh, alongside BASE_REFRESH), so the first build of each month — PR checks or Build and Push, both through ensure-base.sh — rebuilds the base images. A new tag alone only re-tagged cached layers (measured: 15 s, no apt), so ensure-base.sh also passes the month as the BASE_REFRESH_MONTH build arg, which the toolchain and deepsig stages declare (with BASE_REFRESH) before their first RUN: apt and the installs re-run against the current python:3.12-slim (Debian and Python security updates). Before this, bumping BASE_REFRESH had the same flaw. This changes the base tag and its cache keys now, so this pull request builds the base cold once. BASE_REFRESH still forces a rebuild in between.
• The chart executor is vmar559/plasticdb:<sha>-chart, a tag in the one private repository, instead of the public vmar559/plasticdb-chart-executor:<sha> (CHART_EXECUTOR_REPO is gone; its layer cache keeps its own ref). The gitops chart selects it with chartExecutor.image.variant: chart (plasticdb-gitops), which must merge only after a release that publishes -chart has deployed.
• tests/test_deploy_contract.py covers the gate (release-only, last step before the bump, limits, timeouts), the monthly refresh (same month same tag, next month a new one, both workflows computing it through ensure-base.sh), the sidecar tag and the prune workflow's dry-run default.

Build only. No application behaviour changes, no migration.

Changed

• The API and release-worker images are tags in vmar559/plasticdb, not repositories of their own. Docker Hub's free plan allows one private repository and vmar559/plasticdb is it, so plasticdb-api and plasticdb-release (which 1.49.0 waited for) can never be created private on this account. Build and Push now publishes, under one release SHA: vmar559/plasticdb:<sha> (worker, unchanged), :<sha>-api (API, beat, migrate) and :<sha>-release (release worker), and no longer skips the two role builds. The private-repository guard is a single ensure on the one repository. The dry run measures the pushed -api / -release tags instead of re-building the API under a scratch tag. The gitops chart selects them with <role>.image.variant (plasticdb-gitops); until a variant is set, every role keeps running <sha>.

Build and CI: images are built from cached, content-addressed base images, one image per role, and pull requests build each image once. No application behaviour changes, no migration. Measured: the API image is 335 MiB compressed / 857 MiB unpacked (was the 1,024 / 3,223 MiB worker image); the worker is unchanged in content (1,021 MiB); a release re-ships ~4 MiB of code layers instead of ~65 MiB. Build and Push's build steps take 38 s warm and 150 s with no layer cache (after a ~2.5 min checkout), down from 12–20 min. PR checks take ~14 min warm and ~19 min when the base must be rebuilt (was 21–36 min); the HMM smoke (~8 min) is now the critical path. The base images are already published for this commit's inputs.

Changed

• Base images (Dockerfile.base). The bio toolchain (HMMER, T-Coffee, MAFFT, MMseqs2, foldseek, DIAMOND) and every Python dependency moved out of Dockerfile into base-api (toolchain + requirements.txt without torch) and base-worker (+ torch, the notebook toolchain, transformers, DeepSig). They are published as docker.io/vmar559/plasticdb:base-<hash>-api|worker, where <hash> (.github/base-image-tag.sh) covers Dockerfile.base and every file it COPYs, and are rebuilt only when that hash changes (.github/ensure-base.sh). Dockerfile builds api, worker and release FROM them and adds only the code, so a code change no longer re-downloads torch, TensorFlow or the toolchain. Each base records what it resolved to in /usr/local/share/plasticdb/base-*.freeze.txt. BASE_REFRESH in Dockerfile.base rebuilds a base on purpose (OS security updates are no longer picked up by every release).
• The API image no longer carries torch, transformers, jupyter or DeepSig. torch is excluded from its resolution (uv pip install --excludes); nothing app.main imports needs it, and the api build now fails if importing app.main loads torch, transformers, plasticdb_ml.ml, jupyter or nbconvert.
• Python installs use uv (bind-mounted, never in an image) with --index-strategy unsafe-best-match (pip's --extra-index-url semantics), bytecode compilation, and a BuildKit cache mount; apt and npm use cache mounts too. docker/pip-install.sh keeps the credential handling unchanged and turns uv's cache off if it is not on a cache mount.
• Registry layer cache. Every build reads and writes docker.io/vmar559/plasticdb:buildcache-<target> (mode=max). The runners are ephemeral ARC pods with a fresh docker-in-docker daemon per job, so no local builder can persist; the builder is still created by name, only if absent. Code COPYs are --link and come after the dependency layers.
• PR checks build once. A new image job builds the test, browser-test, api, HMM-smoke and worker images in parallel and pushes them as docker.io/vmar559/plasticdb:ci-<sha>-<role>; build (HMM smoke), tests, pg and e2e pull them and no longer check out or build. They run on !cancelled() and fail first unless image succeeded, so a failed build cannot pass the required checks by skipping them. Pull requests that touch only docs/ and top-level *.md files build nothing and report all four checks green. A superseded run is cancelled.
• Build and Push publishes one image per role, all under the release SHA: docker.io/vmar559/plasticdb:<sha> (the worker, unchanged name, still everything every role needs), docker.io/vmar559/plasticdb-api:<sha> and docker.io/vmar559/plasticdb-release:<sha>, plus the chart executor as before. The gitops bump is the same single image.tag sed and runs only for a v* tag.
• tests/test_deploy_contract.py covers the role → repository mapping, the shared tag, the bump, the base and its hash inputs, the credential reaching only the base build, the cache, the private-repository guard and the PR-check structure. The pin tests in test_bio_tools.py, test_signal_peptides.py and test_release_notebooks.py read Dockerfile.base.

Added

• workflow_dispatch on Build and Push: a dry run that pushes only ci-test-<sha> tags, never bumps gitops, and reports image sizes against the current release, the dependency diff and a scan of every image and cache blob for the git credential (.github/image-report.sh, .github/scan-registry-for-secret.py).
• .github/dockerhub-private.py, run before the first push: a push to a Docker Hub repository that does not exist yet creates it public, and these images carry private code. plasticdb-api and plasticdb-release do not exist yet and the CI token cannot create them (403), so until they are created private the release skips those two pushes with a warning and every role keeps running vmar559/plasticdb.
• make base builds the base images locally; docker compose builds FROM plasticdb-base:local (or PLASTICDB_BASE_IMAGE).

Dependency pin. No migration. Needs a worker image (release worker) for the next release to pick it up.

Fixed

• Release notebooks no longer hard-code IsPETase's database id. plasticdb-ml pinned to v0.10.1 in requirements.txt and requirements-notebooks.txt. Release 2026.44.7 failed in notebook 01 (ISPETASE = 21 → build_corpus[21], KeyError 21; then NameError pet_cell) because the 2026-10-28 duplicate cleanup (1.46–1.48) tombstoned protein 21 / PBG5RN into protein 3 / PBFCTW. v0.10.1 looks IsPETase up by accession through plasticdb_ml.ids.resolve_accession, which follows merged_into_accession when a bundle carries it (this exporter does not emit it yet), falls back to the former accession, and reports absence instead of raising. A notebook lint now refuses hard-coded integer protein ids. Verified on a bundle cut live from the 1.48.0 database: notebooks 01–04 exit 0.

Performance: the public protein page no longer downloads the whole record. For IsPETase (PBFCTW: 1,018 interactions, 865 variants) the page fetched 10.7 MB and took about 14 s cold. It now loads ?view=summary and pages the evidence in. No migration, no change to the full-record response. Needs a deploy (API only).

Changed

• /proteins/{accession} page requests ?view=summary (carrying include_sequenceless as before), renders at once, then loads interactions, variants, kinetics and properties 50 rows at a time, with a "Load more (n of N)" control and a retry on failure. Section headers show the totals from the summary's counts before any row arrives. The evidence card takes optional counts / moreHtml options (evidence_table.js); other pages are unaffected.
• ?view=summary is built lean. It used to build the whole record and strip it, so a cold summary cost nearly as much as the full record. It now never loads the lists (noload), reads the interactions with only plastic, paper and method, and takes the counts from COUNT queries. It has its own cache key (proteins:summary:{id}), cleared wherever the detail key is. The answer is identical to the old stripped record (a test compares them).

Added

• GET /api/v1/proteins/{accession}/variants, /interactions (kind=direct|indirect|all, indirect rows labelled with their variant), /kinetics and /properties: {total, items} with limit (default 25, max 200) and offset or page, same item shapes as the full record. Totals equal the summary's counts. They apply the v1.48.0 hide rule through variant_parent_public_filter and honour include_sequenceless; a hidden accession answers 404, a merged one 301.
• tests/test_pg_protein_page_lazy.py: pages add up to the full record, pagination bounds, hidden rows and hidden accessions, the opt-in, the full record unchanged, the lean summary equal to the stripped full record, and the page requesting the summary.

Feature (#576 decision 10): proteins and protein variants without a sequence are hidden from every public surface, and so are the interactions they are the actor of. No migration, no stored flag: it is a computed predicate, so a row reappears by itself once it gets a sequence. Needs a deploy (API and worker); the hourly DIAMOND rebuild then drops the two sequenced variants whose parent has no sequence. Measured on live V2 before the change: 424 of 1,511 live proteins, 331 of 2,948 variants (329 without a sequence of their own, 2 under a sequence-less parent) and 1,133 of 15,689 interactions (765 protein, 368 variant) are hidden. Organism-only interactions (8,782) are unaffected.

Changed

• The public gates carry the predicate. public_visibility_filter(Protein) now also requires a non-blank sequence (NULL and whitespace both count as none); variant_parent_public_filter() requires the variant's own sequence and a visible, sequenced parent ("a variant whose parent is hidden is hidden"); public_actor_interaction_filter() and measurement_actor_public_filter() route their protein and variant slots through those two. Their Python twins (visible_source_proteins, visible_step_protein, new is_public_protein / is_public_variant / is_public_interaction, has_sequence) follow. Every public query that was already gated for staged and tombstoned rows is gated for this by the same call: counts and the landing headline, the protein, organism, plastic, paper and engineering pages, global search and autocomplete, every bulk download and export, release snapshots, sitemaps, the change feed and the chat agent's SQL guard. A sequence-less accession answers 404 like a staged one.
• The DIAMOND/BLAST reference build uses the same gates. reference_db.protein_public_clauses / variant_public_clauses were a hand-written copy that checked a variant's parent for visibility but not for a sequence.
• The protein row of the CSV/JSON exports no longer counts a hidden variant's kinetic measurement in n_kinetic_measurements / best_kcat_km.
• Admins still see everything: the admin lists and the record preview never used the public gates. The preview now labels such a row "Public, but hidden — no sequence" and links its opt-in view.

Added

• "Show proteins without a sequence" checkbox in the protein search (off by default). It sends include_sequenceless=true to GET /api/v1/proteins, which lifts the predicate for proteins, variants and their evidence in that search, and the result links carry it on to /proteins/{accession} and /variants/{id} (and their API reads), so a listed row opens instead of 404ing. The opt-in detail is not cached. Every other route ignores it.
• tests/test_pg_sequenceless_hidden.py calls every registered GET route anonymously against a fixture world (a sequence-less protein, a blank-sequence protein, a sequence-less variant, a sequenced variant of a hidden parent, each with interactions, assays, kinetics and a pathway step) and fails on any response that carries one of them. On the previous release it finds 39 leaking route templates. It also checks the snapshot tarball and the DIAMOND build clauses, the opt-in, the admin view and that filling a sequence un-hides the row. tests/test_sequenceless_hidden_everywhere.py pins the gates and adds a ratchet: no module outside a reviewed list may compare visibility to "public" by hand.

Duplicate prevention (#576 step 5). No migration, no new table. Needs a deploy (API, worker and beat). Rule: one identical mature chain is one enzyme.

Added

• Auto-merge after SignalP. When a SignalP result writes a cleavage site, the protein's mature chain can become identical to another live protein's. The groups such a protein now belongs to are found with the detector report_duplicate_proteins --frame mature uses (protein_dedup.duplicate_groups_sync) and merged through the existing merge service (protein_merge.plan_group → apply_plan), one group per transaction with the deadlock retry. Hooked into the signal-peptide worker (predict_for_proteins: the on-promotion dispatch and the daily sweep) and the enrichment ladder (enrich_protein_sync, enrich_all_proteins_sync), after the result is committed. A merge problem is logged and never fails the SignalP task.
  • Survivor: protein_merge.survivor_sort_key, the merge script's rule: never a tombstone, then the most restrictive visibility, then the richest evidence, then the most complete accessions, then the lowest id. The protein whose prediction triggered the merge gets no preference.
  • Overridden refusals (#576): verdict_conflict and verdict_disagreement merge, and each paper's interactions move with their own verdict. organism_split merges, the survivor keeps its organism, and a curator entry offers the other one.
  • Still blocks: visibility_collapse (a staged row must not demote a public one), variant_conflict (needs a curator decision per variant), adjudicated_elsewhere and tombstoned. A blocked group gets a curator entry instead.
  • Audit: the protein_merges rows apply_plan writes, with ; auto-merged after SignalP (#576 step 5); overrode … appended to notes. One Slack line per batch lists what merged and what was blocked.
  • Setting DEDUP_AUTO_MERGE_AFTER_SIGNALP (default on) switches it off.
• Curator entries reuse the flags queue. Each entry is a pending suggestion annotation on the protein, so it appears in the curator inbox as "Flag or suggestion" and is worked at /dashboard/review. An organism entry carries field_name = source_microorganism_id with the other organism's id: accepting it switches the organism through the existing suggestion path, and rejecting it keeps the row's organism. Other entries carry no field, so accepting them only acknowledges. Entries are filed under the service account (the earliest active superuser if there is none) and are never filed twice for one event. With neither account, the entry is skipped with a warning and the merge or approval goes ahead.
• Weekly duplicate report to Slack. The beat job report-duplicate-proteins (Mondays 07:15 UTC, task report_duplicate_proteins_weekly) runs the mature-frame detector read-only. It posts only when at least one group exists: the group count, the row count and the first 10 groups (accessions, names, length, organism and visibility splits).

Changed

• An organism conflict at approval attaches instead of minting a duplicate. _get_or_create_protein used to refuse a sequence match whose source organism (compared by tax_id) differed, and create a second row. It now attaches the evidence to the existing row, which keeps its organism. The name is recorded as a synonym as usual, the organism_mismatch dedup note is still recorded, and a curator entry reads "Organism conflict on PB…: paper <doi> says X (tax_id N), row says Y (tax_id M) …". The entry is a one-click switch when an organism row for the paper's tax_id already exists. Otherwise it only acknowledges: promotion does not create an organism row with no evidence of its own. It is filed in a savepoint, so a failure cannot abort the approval.

Bug fix: approving a paper or a revision could attach evidence to a tombstoned (merged) protein. No migration. Needs a deploy (API and worker). Existing rows are repaired by a new script, run separately (scripts/repair_tombstone_evidence.py --apply, in the worker pod).

Fixed

• Promotion follows a merge to the survivor. A merged protein keeps its row as a tombstone (merged_into_accession set, is_tombstoned true), and that row keeps its uniprot_accession, genbank_id and name. _get_or_create_protein's accession match and its paper-scoped name match (_protein_named_in_this_paper) could both return it, so new interactions and variants were hung off a row every read path forwards past. On 2026-10-29 production held 24 interactions from 17 ingestions on 11 tombstones, plus 7 variants. For example, a revision of Cut190 (C7MVE8) landed on PBDUDQ, merged into PBRE2B the day before. Both matches now follow merged_into_accession to the live row at the end of the chain (_live_protein), so the evidence lands where the enzyme now lives. The accession match prefers a live holder of the accession over a tombstone, then the lowest id (it used to take whatever LIMIT 1 returned). A chain that ends at a withdrawn record, a missing accession or a loop counts as no match. The sequence match already skipped tombstones. The variant lookup is keyed on the parent this returns, so a variant actor lands on the survivor's variant, never on one parked on the tombstone.

Added

• scripts/repair_tombstone_evidence.py — re-homes what already points at a tombstone onto its live survivor, using the merge service's own move (protein_merge._rehome), so the unique-key rules match a merge's: enrichments, structures and variant names the survivor already holds stay parked on the tombstone. A variant whose name clashes with one of the survivor's and that carries evidence is refused and reported, never folded or overwritten. A tombstone with no live survivor that still carries evidence is refused too. Dry run by default (read-only session), --apply to write, --only to limit it to some tombstones. Each tombstone is one transaction, checked with the merge's evidence census before and after. It is idempotent. Live dry run on 2026-10-29: it would move 24 interactions, 7 variants, 10 properties, 1 kinetics row and 1 structure from 11 tombstones, with no refusals.

Curator decisions for the duplicate-protein merge pass (#576, #575 §2). No migration. The tool is scripts/merge_duplicate_proteins.py, run in the worker pod. Nothing changes for a run without --decisions, except that the variant auto-equivalence below now folds same-chain clashes that it used to refuse.

Added

• --decisions FILE. A JSON file of curator rulings, one entry per group (format in app/services/merge_decisions.py). Entries are matched by member accessions (survivor + retire), never by G-number. If the planned group's membership differs, the group is refused as decision_mismatch and nothing is lifted. A malformed file stops the run before anything is planned. Per group:
  • accept_verdict_disagreement: true lifts verdict_conflict, verdict_disagreement and adjudicated_elsewhere. Evidence still moves per paper: apply_plan now asserts that every interaction of the group keeps its paper, polymer and result across the merge (evidence_ledger), and rolls back otherwise.
  • organism: {tax_id, name, reason} sets the survivor's source_microorganism_id to the stored organism with that tax id, whose name or a synonym must match, and lifts organism_split. The organisms it replaces go into the protein_merges notes and cross_references["organism_decisions"]. It never creates an organism row: an unknown tax id is refused as decision_invalid.
  • variant_decisions resolve same-name variant clashes. fold moves the dropped row's evidence onto the kept row, which keeps its own sequence, frame, type and mutations. When the kept row is on a retired protein, it moves onto the survivor and the survivor's own row is parked on that tombstone. rename renames a row, which then moves as a separate variant. Every clash must be covered, or the group stays refused.
  • The survivor may be named by the decision when that doesn't change the resolved visibility.
  • visibility_collapse is never lifted.
• Variant auto-equivalence. Two same-named variants that conflict only on sequence/numbering_offset and hold one chain once each offset is applied are folded without a decision. "One chain" means the longer sequence minus exactly the N-terminal residues the shorter starts after, as in LCC's 293-aa precursor at offset 0 against the 258-aa mature chain at −35. Identical stored sequences also count. The fuller row is kept, and frame columns are never mixed between the two rows.

Fixed

• Protein merge moves protein_structure_papers links onto the survivor (#575 §1). A retired protein's structure row that parks because the survivor already holds that PDB id used to keep its paper links, so the survivor's copy under-reported which papers asserted it. Links now follow onto the survivor's row (or the kept variant's, for a folded variant). A link whose ingestion the target already records stays put, and none is deleted. The link count is part of the merge's conservation check.

#575 follow-ups. No migration. Needs a deploy (API and worker).

Fixed

• A duplicate variant name is a 409, not a 500. POST and PATCH /admin/protein-variants hit uq_protein_variants_parent_name (migration 0028) as an unhandled IntegrityError. The shared admin create/update now turns a violation of that index into 409 with "A variant with this name already exists on this protein". Any other integrity error is re-raised unchanged.
• GET /proteins/{accession} no longer re-validates its own cached payload. The cached detail is already jsonable_encoder(ProteinDetail), but returning the dict made FastAPI validate and re-encode it. For PBFCTW (10.7 MB, 1,018 indirect interactions, 865 variants) that was about 1.3 s of event-loop CPU against 0.2 s to dump. Measured against production before the change: 5.5 s warm uncompressed, 1.7-2.2 s gzip, 14 s on the cold request. The cold build is unchanged (see the issue for the open question).
• An approval now queues a lineage fill for the organisms it created. An organism is minted with a tax_id and no lineage, and only the Sunday 03:17 refresh filled one, so it sat in the curation report's no_lineage list (ingestion 925, Halomonas sp. G11, row 4115, tax_id 1684425, still lineage NULL two days after approval). The new fill_missing_lineages task runs the missing scope under the usual taxonomy-run lock; a broker failure is swallowed and the weekly pass remains the net.

Added

• report_duplicate_proteins reports live proteins sharing a v1 id (proteins.legacy_id, the verified PlasticDB v1 number; plasticdb_id is the V2 serial and already unique). It needs no sequence, so it finds an import-time split where one row's sequence is NULL. Tombstoned rows are ignored.

Bug fix (#576): the gaps that left proteins and protein variants without a sequence. No migration. Needs a deploy (API and worker). Existing rows are not backfilled here: the three known cases (TfH PBERS5, TfCut2 PB78JG, FsC PBDPN7) carry no accession, so neither the new rung nor a re-approval reaches them, and PB78JG would become a byte-identical duplicate of PB5QQN. That is a merge decision for a curator.

Fixed

• Promotion fills a missing sequence on a matched protein. _reconcile_protein_metadata left sequence "exactly as stored", on the grounds that a match means the two agree. That is false when the stored sequence is NULL. A sequence-bearing actor matched by accession onto a row minted without a sequence now writes it. A stored, non-empty sequence is never replaced. The write goes through app.services.sequence_fill.fill_missing_sequence, which also sets sequence_provenance back to unverified and clears has_signal_peptide / signal_peptide_cleavage_site, so the daily provenance and signal-peptide sweeps pick the row up. It refuses a value with no sequence_key, such as a note. The fill is recorded as a sequence_filled dedup note. The approval dispatches signal-peptide and provenance jobs for these rows as it does for new ones, and queues a protein-cache invalidation. Today only an accession match reaches this path: the paper-scoped name match runs only for an actor with no sequence.
• A reused variant fills a NULL sequence and numbering_offset. Promotion matches a variant on (parent, variant_name). On a first promotion the reused row took only synonyms and an unclaimed origin from the payload. A revision of a paper that does not own the row recorded the stated sequence as a conflict with NULL. So a variant created sequenceless by one paper stayed sequenceless whatever later papers stated: 28 variant entries in approved extractions carried a sequence and offset 0 against live rows holding NULL for both. _fill_variant_nulls now runs on both paths. sequence is written only when the stored one is NULL or blank, and the variant's signal-peptide columns are cleared with it. numbering_offset is written only when the stored one is NULL and the stored sequence is the payload's chain, because an offset stated against a different sequence would misdescribe it. Neither is ever overwritten, a fill is not reported as a revision conflict, and the row's ownership stamp does not move. A revision of the owning paper still rewrites its own row, as before.
• Enrichment retries a failed sequence fetch. Promotion fetches a sequence from the accession once and, on failure, logged "enrichment will retry later". Nothing did: _step_uniprot marks a row with no genbank_id and no sequence permanent: no_lookup_path. The ladder has a new first rung, sequence. It runs only for a protein with a UniProt or GenBank accession and no sequence, and uses the same helper promotion uses (fetch_sequence_for_accessions, over lookup_uniprot_sequence / lookup_genbank_sequence). Found: ok, the sequence is filled as above, and verdicts recorded while the row had no sequence (no_sequence, no_lookup_path) are dropped for that pass so those rungs run at once. Lookup failed or empty: transient, retried on the weekly pass THROTTLE_DAYS apart and given up at RETRY_CAP (5). The admin rescue can lift the cap (source: "sequence"). A row that has a sequence, or has no accession, gets no sequence row at all. The admin coverage table lists the new source.
• A nucleotide accession resolves through its CDS, and only when it links exactly one protein. U58990.1, AB038647.1 and FR727681 are nucleotide records. NCBI's protein database answers 400 for them, so the sequence was never fetched. accession_format.is_nucleotide_accession recognises the shape (one letter + 5 digits, two letters + 6 or 8, RefSeq NM_/NC_/…). It excludes a value that also fits the UniProt pattern, such as P12345. lookup_nucleotide_cds_proteins reads the record's CDS protein_ids (efetch nuccore rettype=fasta_cds_aa). With exactly one, that protein record's sequence is used, and enrichment records it in cross_references.cds_protein_id. The stored genbank_id is not replaced. With several CDS proteins, or none, nothing is filled: enrichment records permanent: nucleotide_ambiguous_cds: … or nucleotide_no_cds: …. An NCBI outage stays transient. Promotion and the curator form's GenBank lookup (POST /lookup-sequence) use the same path. On the form, an ambiguous record returns the reason as the error. One effect to note: R-1's load-time check now also applies to an actor that names a single-CDS nucleotide accession and supplies its own sequence. The CDS protein's record wins, as it already does for protein accessions.

Structure provenance and a sequence guard on PDB entries. Migration 0029_structure_provenance (schema + backfill): adds three nullable columns to protein_structures and backfills the new protein_structure_papers table. It deletes no rows. Run python -m app.db.init_schema before alembic upgrade head, as usual (it creates the table).

Added

• protein_structure_papers records which paper asserted each structure row. One row per (structure, asserting ingestion), with the ingestion's paper. Promotion writes it whenever a payload names a PDB id for a protein or variant, including when the row already exists (5LUI on protein 511 is asserted by two papers). protein_structures.created_by records which writer created a row: paper, enrichment, prediction or import; NULL means it predates the column.
• Enrichment checks that a PDB entry is the owner's protein. It reads the entry's polymer sequences from RCSB and requires one to align to the protein's (or variant's) sequence at ≥ 95% identity over the aligned region, covering ≥ 50% of the shorter sequence (local alignment, BLOSUM62). A variant without a sequence is checked against its parent. The verdict is stored on the row (sequence_check = match/mismatch, sequence_identity) and returned by the structure API. A mismatch row is kept for a curator, shown with a "sequence mismatch" badge on the protein page, and is never the primary structure: it does not count as "already has a measured structure", it does not stop AlphaFold, and structure_file is never pointed at it (one already pointing at it is cleared). Nothing is deleted on this basis. When RCSB cannot be reached or the owner has no sequence there is no verdict and nothing changes; during discovery that is recorded as transient and retried.
• The check runs on every way a PDB id arrives. Existing rows are checked once (paper-asserted, v1 import, earlier discoveries). Discovery checks its candidates: the id in structure_file, every entry that cross-references the UniProt accession (up to 20; it used to take RCSB's first, unranked hit), and the RCSB sequence search (which already requires identity 1.0). It adopts the best match, or nothing; the rejected candidates are listed in the enrichment detail.

Fixed

• A corrected paper's retracted PDB id is removed. Paper 10.1002/anie.202525215 asserted 6KYS on DuraPETase (variant 488), a typo for 6KY5; 6KYS is M. tuberculosis MazF-mt1. Revision 1632 dropped it and was approved in replace mode, but the structure row stayed. When an approval retires earlier runs, their structure links are now dropped. A row the new payload does not assert is deleted if no other ingestion links it and a paper created it. The owner's structure_file is cleared if it names that entry (otherwise enrichment would recreate the row), and the next best row becomes primary. Rows from the v1 import, enrichment, predictions, admin edits or of unknown origin only lose the link. The paper_superseded event lists each structure removed or kept (structures_retracted).
• NylB no longer has 2E8I as its primary structure. All 16 PDB entries that cross-reference P07061 are Hyb-24 NylB/NylB' hybrid mutants at 88.3–90.1% identity; enrichment took the first one. The guard flags all 16, and AlphaFold's model (AF-P07061, pLDDT 97) wins.
• upsert_protein_structure never re-points a flagged row at a new PDB id.

Migration

• 0029_structure_provenance adds created_by, sequence_check and sequence_identity to protein_structures (nullable; declared on the model). It backfills protein_structure_papers from approved (live) ingestions. An ingestion's paper is the one its interactions are stamped onto, or the paper matching reference.doi. Protein-level ids (protein.pdb_id, a variant's parent_protein.pdb_id) link the protein-owned rows with that id on the proteins the paper's interactions name. Variant-level ids link the row with that id on the paper's variant of that name. A linked experimental row whose owner is not a v1 record (legacy_id IS NULL) gets created_by = 'paper'; predicted rows get prediction; everything else stays NULL and is never removed automatically. Superseded ingestions are not read, so rows only a retired run asserted (6KYS, row 512) stay unlinked. Those are removed by hand. Idempotent. The downgrade drops the table and the three columns.

Every weekly release now cuts all of its ML bundle's record-derived inputs from the live database: Matrix B, identity.npz, features.json and mature.faa join the two corpora. Only the seed families and the Phase 1 partition stay frozen, on purpose. The release task gets its own 12 h time limit. No migration. Needs a deploy (worker image: plasticdb-ml v0.10.0) and the gitops branch release-time-limits (RELEASE_NOTEBOOK_TIMEOUT).

Changed

• Matrix B is cut live. export_ml_bundle --from-database now runs scripts/export_mutation_effects' own fetch and export --tier strict stages, with the frozen Phase 1 seeds, into its scratch directory. Its matrix-b/variants.json, mutation_effects.tsv and summary.json shadow the cached step2-2026-09-04 copy. Until now every bundle carried that copy: 745 variants, 332 triples over 42 parents, 282 variant rows. So T4 and every other variant-using cell in notebooks 01 and 03 ran on September data. On the 2026-10-28 database the live cut is 2,952 variants, 7,318 strict effect rows and 1,577 triples over 121 parents, giving 1,397 variant rows. Measured: re-running export on the September run's own variants.json, with the Phase 1 run it used, reproduces its mutation_effects.tsv and summary.json byte for byte.
• The positional artefacts are rebuilt over the live records and Matrix-B variant rows, every release. The release passes --refresh-positional always --identity-cache <ML_BUNDLE_IDENTITY_CACHE>. The identity matrix comes from plasticdb_ml.ml.identity, a pair cache keyed by both mature chains' SHA-256 and the aligner's Biopython version. Its result equals the old identity_matrix_parallel + extend_identity element for element: measured on the September set against the shipped identity.npz, and on the 2026-10-28 live set, which is 2,430 ids and 1.87M distinct pairs. A cold build of the live set is ~23 min on the pod's four CPUs. A warm one takes 0.4 s. The cache is pre-seeded on the shared volume (<bioinformatics_dir>/identity-cache/identity_pairs.npz).
• mature.faa is rebuilt too, in the cached run's own naming (>p<id>, variants p-<id>, the mature chain). Measured: this reproduces the September file byte for byte from that run's corpus and Matrix B. A protein curated this week is now in notebook 02's search set.
• Coverage counts the variant rows, and the export refuses a gap. positional_coverage now resolves ids with falsify.extended_records (records and Matrix-B variant rows) wherever it is given a Matrix B. Both the exporter and assert_bundle_covers_corpus give it one. With --apply, an export whose identity matrix or feature table misses any id, or whose coverage cannot be measured, now writes nothing and exits 1, naming the ids and the reason the rebuild could not run. It used to write the bundle with a named gap in missing_parts. The ESM entry was already refused on a missing or stale vector.
• weekly_dataset_release has its own limits: soft 12 h, hard 12 h 15 min. Every other task keeps 4 h / 4 h 15 min. The broker visibility_timeout goes from 5 h to 13 h. It is one number for the whole Redis broker (kombu's single unacked index), so it must clear the longest task. The cost: a message whose worker died waits up to 13 h, not 5 h, before it is redelivered, on every queue. STALE_RUN_HOURS goes from 8 to 13, so a live release younger than its ceiling is never "resumed" by a second run. docs/dataset-releases.md has the measured runtime.

Added

• --identity-cache FILE on export_ml_bundle (needs --from-database).
• ML_BUNDLE_IDENTITY_CACHE setting; release_notebooks.identity_cache_path().

Bug fix. Migration 0028_variant_unique_name (data): collapses duplicate protein variants, then adds a unique index. It refuses to run, and changes nothing, if two same-named variants on one parent disagree about the variant (see below); run the read-only query in the PR first.

Fixed

• Approving a paper that names a variant no longer 500s when its parent holds two variants of that name. Promotion found "variant X of protein P" by (parent_protein_id, variant_name) with scalar_one_or_none(), and nothing kept that pair unique. The lookup now takes the lowest id (ORDER BY id LIMIT 1). The interaction lookup that follows it (paper, polymer, actor) had the same shape and now does the same: a protein merge legitimately leaves two interactions of one paper on one actor and polymer, and the paper's next approval used to raise on them.
• Protein merge no longer duplicates variants. It moved every variant from the retired protein to the survivor, even when the survivor already had one of the same name; after a merge, the next approval naming that variant 500'd. A retired protein's variant whose name the survivor already holds now stays on the tombstone (counted in parked), and its evidence is folded onto the survivor's variant: interactions (with their experiments and observations), kinetics, properties, job hits, and the structure and enrichment rows the survivor's variant does not already hold. Its mutation list stays with it; the survivor's variant keeps its own values, fills its NULLs from it, and gains its synonyms. A fold is planned only when the two rows agree about what the variant is — sequence (whitespace and case ignored), variant_type, numbering_offset and the mutation list, compared wherever both state them. Otherwise the group is refused as variant_conflict, for a curator. apply_plan now also checks a variant-evidence census (variant-actor interactions carry no protein_id, so the protein census could not see them), and the audit row records each fold. The dry run lists the folds.

Migration

• 0028_variant_unique_name groups the variants of one parent with one exact variant_name (exact, as promotion compares; S238F and s238f stay two variants). If any group's rows disagree on sequence, type, numbering offset or mutation list, it raises before writing anything and lists the groups — folding them would keep one sequence and drop the other. Otherwise each group collapses onto its lowest id (the longest-citable accession). Before the duplicates are deleted, every foreign key onto them is re-pointed: interactions, enzyme_kinetics, enzyme_properties, job_results all move; structures move unless the survivor holds that PDB id (the copy then fills the survivor's row's NULL metadata, as 0027 does); enrichments move unless the survivor has that source; the survivor takes a mutation list only if it had none (the others are identical). It refuses to run if the live database has a foreign key onto protein_variants it does not know. The survivor fills its NULL columns from the duplicates and unions their synonyms; any other differing value a duplicate carried is written to the log. A removed accession is marked merged in accession_registry and pointed at the survivor (or moves to a survivor that had none). Referring rows are counted before and after and must match. It then creates uq_protein_variants_parent_name on (parent_protein_id, variant_name), declared on the ORM model for the same reason as 0027's indexes. The downgrade drops the index; it does not restore removed duplicates.

Bug fix. Migration 0027_structure_unique_pdb (data): collapses duplicate structure rows, then adds two unique indexes.

Fixed

• Approving a paper that names a protein's PDB id no longer 500s. Promotion asked "does this protein already have structure X?" with scalar_one_or_none(). Protein 511 (Thc_Cut1, PBHHTE) had four rows for 5LUI (two 5lui, two 5LUI), so approving ingestions 785 and 892 raised MultipleResultsFound and the whole approval failed. Both lookups (protein and variant) now ask an existence question (LIMIT 1), compare with upper(), and the protein lookup ignores variant-owned rows. A variant-owned row also carries its parent's protein_id, so a variant and its parent naming one PDB id in a single payload used to hit the same error.
• PDB ids are stored upper-case. Enrichment lower-cased the id it took from a structure_file name; extraction sends upper case. normalize_pdb_id (strip, upper, blank → NULL) is now applied by promotion, upsert_protein_structure and scripts/import_degraders.py.
• Enrichment no longer overwrites the wrong structure row. upsert_protein_structure updates the row that already holds the PDB id, if there is one, instead of re-pointing the first experimental row at it. For a protein it now looks only at the protein's own rows, so it can no longer overwrite a variant's structure with the parent's id.
• Protein merge no longer duplicates structures. It moved every structure row from the retired protein to the survivor, even ones the survivor already had; that is how protein 511 got its second pair. A protein-owned row whose PDB id the survivor already holds now stays on the tombstone, as enrichment rows do (reported as parked). Variant-owned rows still move.

Migration

• 0027_structure_unique_pdb sets blank pdb_id values to NULL, and collapses each group of rows with the same owner (the variant if protein_variant_id is set, otherwise the protein) and the same upper(trim(pdb_id)) to one row. The survivor is the is_primary row, then the row with the most of method/resolution/deposition date/source/confidence/model URL filled, then the lowest id. Its NULL columns are filled from the removed rows. No table references protein_structures.id. Each collapse is logged. It then upper-cases every pdb_id and creates uq_protein_structures_protein_pdb (protein_id, upper(pdb_id) where protein_variant_id IS NULL) and uq_protein_structures_variant_pdb (protein_variant_id, upper(pdb_id) where it is not NULL). A protein and its variant may still hold the same id. The indexes are declared on the ORM model, not in install_pg_extras, because the deploy runs create_schema() before the migration and would build them on the un-collapsed table. The downgrade drops the indexes; it does not restore removed duplicates or the original case.

Every weekly release now fits notebook 03 with ESM-2 embeddings and recomputes T4 and T5 from its own bundle. No migration. Needs a deploy (worker image gains transformers).

Added

• export_ml_bundle --with-esm builds the embeddings instead of copying them. build_esm_entry resolves the same record set the positional rebuild uses (records plus Matrix-B variant rows, falsify.extended_records), reuses every vector a persistent cache (--esm-cache) holds for the same mature sequence (SHA-256 checked), computes the rest with plasticdb_ml.ml.esm (ESM-2 650M, revision pinned), and writes exactly that id set with its digests. The manifest gains an esm block (ids, reused, computed, model, revision). The cached step-2 file is no longer read: it covered 879 of bundle-2026.44's 1,212 ids and carried no digests. --esm-threads sets torch's threads. A dry run plans and computes nothing; with --apply a failure to build stops the export, as a missing --with-esm input always did.
• The release passes --with-esm. The cache is ML_BUNDLE_ESM_CACHE, default <bioinformatics_dir>/esm-cache/esm_esm2_t33_650M_UR50D.json; the exporter child gets HF_HOME=<that dir>/huggingface unless one is set, so the 2.5 GB checkpoint is downloaded once onto the shared volume, and only when something needs computing. Measured on the research VM pinned to 4 cores: 2.31 s per sequence at the median length, ~20 s to load the model, 3.3 GB peak RSS; 8 new sequences cost 46 s end to end.
• release_notebooks.assert_bundle_carries_esm, before the first cell beside the coverage gate: refuses a bundle with no ESM entry, an id without a vector, or a vector whose digest does not match the record's sequence, because notebook 03 (plasticdb-ml 0.9.0) now refuses the same. A bundle pinned with ML_BUNDLE_PATH from before this release is therefore refused. RUN.json's bundle.esm records what was checked.

Changed

• plasticdb-ml v0.8.0 → v0.9.0, with the [esm] extra in requirements-notebooks.txt. Notebook 03 recomputes T4 and T5 from the bundle (falsify.t4_from_bundle / t5_from_bundle, the CLI protocol) instead of displaying the frozen 2026-09-04 t4.json/t5.json, which it now shows beside them; fits with ESM by default; and reports every head and T4/T5 with and without it. On bundle-2026.44 (930 records, VM, profile block live): T4 improved-direction 0.285 [0.124, 0.486] over 128 triples / 24 parents (frozen: 0.212, 120 / 21); T5 PET AUROC 0.668 [0.608, 0.778] (frozen: 0.685); PET forest AUROC at centroid25 0.660 with ESM, 0.531 without.

scripts/label_matrix_api.py reads what the API now publishes. No migration, no API change.

Changed

• Actor ids are read, not inferred. GET /papers/{id}/records has carried protein_id / protein_variant_id since 1.42.1 (#491). The script uses them directly and drops the public-page ground truth, the extraction matcher for identity, the group_permuted bijection, the self-audit and max_disagreements / max_impossible. Checked against the live API: all 3,141 protein-page foreign keys and all 2,555 variant ones agree with the published ids, and every one of the 3,162 protein-actor and 2,559 variant-actor interactions is placed (the 1.21.6-era run left 333 of 1,898 protein-actor rows placeable only inside a group).
• is_tombstoned is read, not probed. It is the admin protein list's state == "merged" (which covers proteins in no paper), cross-checked against FootprintProteinRead.is_tombstoned; the one-HTTP-status-per-protein resolver probe is gone.
• A missing field is an error. An API that does not publish protein_id, protein_variant_id, is_tombstoned or the list state raises ApiError naming the field; there is no fallback to the old inference.
• Still inferred: enzyme_preparation / enzyme_production_context are on no read schema, so /ingestions/{id} is still fetched for those two columns and matched on the now-known actor, plastic, verdict and note. Publishing them on a read schema would let that fetch go.

Vocabulary note only. No migration.

Fixed

• PEC vocabulary note now says what the row holds: poly(ester carbonate), v1's Mitsubishi Gas Chemical polyester-carbonate (Calabia & Tokiwa 2004). It used to say PEC "most likely denotes poly(ethylene carbonate)", which contradicted the row's own data and, since 1.41.1, a separate Poly(ethylene carbonate) row.

Dependency pin. No migration.

Changed

• plasticdb-ml v0.5.3 → v0.8.0. It brings the fresh-ID Phase 1 tooling: --first-family-id/--first-merge-id, the partition_id stamp, the crosswalk and stability checks. It also fixes parse_domtblout so a catalytic envelope that HMMER prints in two pieces of one model is completed, which scripts/export_ml_bundle.py picks up through its import. Order-unstable families are now split by a majority vote over 100 seeded orderings. Notebook 05 and plasticdb_ml.metagenomes come along too, but are excluded from release runs. The new Phase 1 partition is 67e0b2a6f4918232: 150 families, 11 profiles. It is staged at /app/bioinformatics/ml-work-2026-10-27, and the release reads it once ML_BUNDLE_WORK_DIR points there (plasticdb-gitops).

Publish safety ahead of the family-partition switch. No migration.

Added

• The family partition is part of the library fingerprint. hmm.fingerprint.json now carries partition: {partition_id, digest, source}, read from the build's families.json (also looked for as phase1/families.json) or Phase 2's profile_manifest.json (beside the calibration, in ../profiles/). partition_id is the stamp plasticdb-ml will write; until it does, the digest of the file is the fallback. A partition swap in a week where the corpus did not move now republishes, and the HMM screen TSV header gains # profile_library_partition: so a download says which partition produced it.
• Keeping old fingerprints comparable. The partition is its own key, not folded into digest, so digest is unchanged and still agrees with scripts/export_ml_bundle.corpus_fingerprint. A caller that cannot see a partition (the hourly check with no build directory) ignores it. One-time change: the library published before this release has no partition, so the first publish after it counts as stale once, republishes and records it.
• scripts/publish_hmm_library --allow-profile-set-change and --families (name the families.json when it is not inside the build directory).

Changed

• A changed profile set is never published automatically. hmm_publish.publish_from (the release) and hmm_build.check_hmm_library (the hourly beat) refuse when the new library's profile names differ from the published one's. The reason is logged and returned (so it lands on the release's notebooks["published_library"] and in the Slack line), with refused: "profile_set_changed"; the old library keeps serving. force does not override it. A person publishes a changed set with python -m scripts.publish_hmm_library --build-dir <dir> --apply --allow-profile-set-change; the script otherwise refuses with exit status 3. POST /admin/releases/publish-now publishes staged records, not the profile library, so it is unaffected and gains no flag.

Bug fix. No migration.

Fixed

• A failed notebook stage no longer loses its cause to a cleanup error. On release 2026.42 notebook 03 hit the 3600s timeout, and while the exception unwound TemporaryDirectory could not remove the scratch tree (OSError: [Errno 39] Directory not empty), so the release row recorded the OSError instead of NotebookRunError: ... did not finish within 3600s. The scratch directory is now removed best-effort (retried, then a warning naming the directory) and cleanup can never raise.
• A notebook timeout now kills the kernel, not just nbconvert. subprocess.run(timeout=) kills only the direct child; the kernel is a grandchild and kept writing into the directory being deleted, which is what made the cleanup fail. nbconvert now runs in its own session and the whole process group is killed on timeout or interruption.

Build fix. No migration, no runtime change.

Fixed

• A flaky runner link to github.com no longer fails the image build. docker/pip-install.sh checks that the private PlasticDB-ML repo is readable before pip starts, with a single git ls-remote. When the CI runner's connection to github.com dropped out, that check hung ~150s, failed, and blamed the token — the v1.41.0 build failed this way twice with a working token. The check now retries network failures (4 attempts, 10/20/30s back-off, each capped at 60s; PROBE_ATTEMPTS/PROBE_TIMEOUT override), stops at once on an authentication refusal, and prints what git actually said so a timeout is no longer reported as an expired token.

Vocabulary only. No migration.

Added

• Plastics: PBST — poly(butylene succinate-co-terephthalate) with no grade stated (copolymer, unparented like PBAT/PBS/PBSA). The legacy PBST55 row is one grade and stays the term for it; filing an unstated grade there would assert a composition. Unblocks 10.1016/j.jhazmat.2024.135895.
• Plastics: PET oligomer — PET oligomers (n ~ 3-8, e.g. a glycolysis by-product), parented to PET on the Nylon oligomer / Polyisoprene oligomer / PCL diol precedent. Not BHET or MHET, which are compounds. Unblocks 10.1016/j.wasman.2023.08.048.
• Compounds: Glutaric acid (C5H8O4; unblocks 10.1016/j.biortech.2025.133186) and N-methyldiethanolamine (C5H13NO2, pH-responsive PU chain extender; unblocks 10.1016/j.msec.2018.08.063).

Vocabulary only. No migration.

Added

• Plastics: Nylon 4 — polyamide 4, poly(4-aminobutyrate). A sibling of Nylon 6 and Nylon 66 (parent Nylon, polyamide). Unblocks 10.1016/j.polymdegradstab.2022.109868 and 10.1016/j.polymdegradstab.2023.110446.
• Plastics: Poly(ethylene carbonate) — the alternating ethylene oxide/CO2 aliphatic polycarbonate, declared as PPC is. It takes no abbreviation: PEC is already the legacy poly(ester carbonate) row (Tokiwa's PBS/C), so reusing it would merge two polymers.
• Plastics: Aliphatic polycarbonate (other) — a family anchor on the Aliphatic polyester (other) pattern (kind NULL, unparented, no abbreviation) for one-off epoxide/CO2 copolymers and terpolymers; comonomers go in substrate_composition. Not bisphenol-A PC. With the row above and the epoxides below, unblocks 10.1007/s002890050484.
• Plastics: PHA Copolymer — the covalent counterpart of PHA Blend, declared as PCL Copolymer is (parent PHA): PHA joined to a non-PHA partner by graft, block or end-cap, e.g. PEG-capped PHBV. Unblocks 10.1016/j.jbiotec.2007.03.012.
• Compounds: Trimethylolpropane (unblocks 10.1016/s0922-338x(97)83000-0), Ethylene oxide, Propylene oxide and Cyclohexene oxide (the epoxide comonomers of the epoxide/CO2 polycarbonates).

Bug fix. No migration.

Fixed

• Bulk approve/reject no longer 500s every revision after the first failure. A failed decision (e.g. an organism NCBI cannot resolve) rolls the session back, which expires every object in it — including the curator's User. The next revision's approve read current_user.id, a lazy load on an async session, and raised MissingGreenlet; on sweep-2026-10 one unresolved name turned the rest of each 10-revision chunk into 500s. The user is now re-loaded after each rollback. The failing revision still fails alone, as designed. Covered by a real-session test; the existing mocked test could not see it because a mock never expires.

Bug fix. No migration.

Fixed

• /curation/revisions no longer takes the site down on a large batch. With no batch chosen the page opens the busiest open sweep and asked for up to 500 revisions, loading every revision's and every parent's full extracted_data in one query to diff them. On the 437-revision sweep-2026-10 batch that exhausted the single uvicorn worker's memory: the page answered 502 Bad Gateway and so did every other page, /health included, until the container restarted. The page now shows 50 revisions at a time with previous/next links, and batch_revisions reads each page without its payloads and diffs them 20 at a time (DIFF_CHUNK), so peak memory follows the chunk rather than the page size — which also protects GET /api/v1/ingestion-revisions/batches/{batch}?limit=500. A paper patched in place now loads only the one earlier attempt it is compared with, not every attempt it ever had.

Vocabulary only. No migration.

Added

• Plastics: Dimethacrylate resin (other). A family anchor for cross-linked (di)methacrylate resin networks (dental composites, adhesives and coatings) with no row of their own; the resin's monomers go in substrate_composition. It takes the two cured dental resins of 10.1016/j.dental.2026.01.011 (Bis-GMA/TEGDMA and FPUA/TEGDMA, hydrolysed by a Pseudomonas cholesterol esterase), which were filed on the legacy Plastic placeholder (PlasticDB-AI #107).
• Compounds: Bis-GMA, TEGDMA, HEMA and perfluoropolyether alcohol, so that anchor's composition can be stated. The paper's one-off FPUA monomer is recorded by its precursors (IPDI, PFPE alcohol, HEMA).

Glossary corrections and two more confirmed settings. No migration.

Changed

• Admin settings: the Figshare API URL and "Run the ML notebooks" now need confirmation too, like the other four release switches. The API URL decides where the Figshare token is sent and where releases are published; the notebooks switch changes what goes into the next snapshot.
• CI: the build, tests and pg jobs get 30 minutes instead of 20. With a cold image build they were finishing at 19–20 minutes and timing out.
• Glossary: the GA / TC / NC entry now defines all three cut-offs and says the HMM screen applies GA by default (it still described the pre-1.39.0 behaviour). The Snapshot and Weekly release entries no longer say every snapshot gets a Figshare DOI.

Shareable job results, safer admin settings, and EC numbers on the pathway. This release carries a migration, 0026_reaction_step_ec_number, which adds one nullable column (reaction_steps.ec_number) and backfills nothing. One new table, job_share_links, ships through create_all.

Added

• Read-only share links for jobs (B13). A job's owner can create a link (optionally expiring in 1–365 days) and revoke it. Anyone with the link sees the results, provenance card and downloads without signing in; the owner's identity, other jobs, the input file and all write actions stay hidden. Tokens are stored hashed and shown once; revoked and unknown links give 404, expired ones 410; shared pages are noindex and no-store.
• My Jobs shows runtime, key parameters, input file and how long result files are kept (90 days, from the existing sweep). Pathway, tree and taxa-table jobs show a real result summary ("1 of 4 steps present", "276 tips") instead of "Results 0".
• EC numbers on the PET pathway. Steps carry an EC number, shown under the enzyme on the diagram and in the step list, linked to ExPASy ENZYME. scripts/backfill_pathway_ec.py (dry-run by default) fills them from the curated pathway file and the EC already on linked proteins, and reports disagreements instead of writing.

Changed

• Admin settings: irreversible switches need confirmation (A8). Figshare auto-publish, the Figshare article ID, the weekly dataset release and publishing the HMM library each have their own control with an in-page confirmation stating the consequence; the API refuses a change to them without confirm (422, nothing written). Secrets are write-only: shown as a mask with the last 4 characters, a re-submitted mask is never stored, and a secret can be explicitly cleared. The settings page no longer has inline handlers, so it gets the strict CSP.

Fixed

• Admin delete of a job with hits failed on PostgreSQL (foreign-key violation from job_results). The hit rows are removed first.

The HMM screen applies the profile library's calibrated cut-offs. No migration. Behaviour change for API clients: an hmm_screen job that sends only evalue now gets gathering cut-offs; send threshold_mode=evalue for the old behaviour.

Changed

• HMM screen uses gathering (GA) cut-offs by default (hmmsearch --cut_ga). Each profile's GA sits between its weakest true family member and the strongest near-miss from a confusable α/β-hydrolase family, so generic esterases and lipases that clear an E-value no longer count as hits. E-value only remains as an "exploratory" mode for distant homologues (threshold_mode=evalue). If any profile in the library lacks a GA line the job falls back to E-value and records why. Each hit shows its bit score next to the profile's GA, and the TSV download gains profile_ga and above_ga. In gathering mode job.evalue is empty.

Fixed

• Approving a revision could delete other papers' interactions. When a revision stopped using a protein variant its original ingestion had created, the supersede deleted that variant even though other papers' interactions still named it, and the database cascade removed those interactions with their experiments and observations. On production, approving revision 1229 (paper 204) removed the LCC-ICCG × PET interactions of ingestions 212, 359 and 360. A variant still named by any surviving interaction, kinetics or properties row is now kept. The three ingestions are repaired by re-approval.
• Deleting or withdrawing a paper could do the same. A variant belongs to the paper that first created it, so deleting that paper cascaded into every other paper's interactions using the variant (38 variants are shared like this on production). Before a paper is deleted, each variant other papers still use is handed over to the earliest of them.
• News item 5 restored. V2 had rewritten the announcement of the 2024 Environmental Microbiome paper into claims the paper does not make, dated it 2022 and dropped its DOI. It now carries the original text, quotes and full citation (doi:10.1186/s40793-024-00575-4).

Accessibility and findability. No migration.

Security

• Password reset accepted a client-chosen minimum length. The reset request carried a min_length field the server trusted, so a one-character password could be set. The field is gone and the server enforces 8 characters, the same rule as registration.

Added

• What's new (/whats-new, the 8 latest releases) and the full changelog (/changelog), rendered from CHANGELOG.md and linked from the footer and the landing page.
• Glossary (/documentation/glossary): 62 terms drawn from the existing documentation, marked as draft wording pending review.
• News archive (/news). The landing page shows the latest software and dataset releases instead of 2021–22 news; items under a year old still appear there.
• Structured data (JSON-LD) for search engines and FAIR indexes: Dataset on the landing and download pages (with the latest Figshare DOI), Bioschemas Protein and Taxon, ScholarlyArticle with DOI and PMID, ChemicalSubstance with verified ChEBI/PubChem only.
• robots.txt and sitemaps (/sitemap.xml, paged per entity type) listing public, non-tombstoned records with lastmod.
• PMIDs on paper pages (View on PubMed), in the papers list and in the papers API (pmid).
• External resources box on protein, organism, plastic and paper pages: PAZy (by accession), ESTHER (α/β-hydrolases only), BacDive (bacteria and archaea), and a cross-references line of stored identifiers as CURIEs.

Fixed

• Extraction drift scans failed on almost every ingestion (1.36.0). The new pairing score counted experiment keys, and a key carries additional_conditions, a JSON dict, so 575 of 584 payloads failed with unhashable type: 'dict' and the scan showed no drift. Keys are now frozen before counting; equality is unchanged.

Changed

• Headings: one h1 per page and no skipped levels across public, documentation, auth and tool pages.
• Text alternatives for the taxonomy-group charts and the 3D structure viewers.
• Visible labels on login, register and password forms, with the password rule shown up front; every form field has an accessible name.
• Page titles follow "Page - PlasticDB" everywhere.
• Card headers on public pages are plain; colour is kept only where it means a result.
• Chat renders headings, lists, code and links instead of showing ## literally (escape first; links limited to http(s) and site paths).

Strain details are shown and exported, and the public API is easier to call from other sites and tools. No migration. Breaking for API clients: /api/v1/jobs, /pathways, /releases and /snapshots now return {total, items} instead of a bare list.

Added

• Strain details on organism and paper pages (C9). Each strain's culture-collection IDs (linked to BacDive), isolation source, genome accession (linked to NCBI) and papers, plus a strain × plastic matrix on organism pages counting each strain's own whole-cell results. Expression hosts and knockouts are labelled and not counted.
• Strain fields in exports. microorganisms.csv/.json gain strain_designations, culture_collection_ids and genome_assembly_accessions; dataset.zip, dataset.json and release snapshots gain a strain_occurrences table (public, non-tombstoned organisms only). The changed export means the next weekly release gets a new DOI.
• scripts/report_duplicate_strains.py: read-only report of strains that look the same (shared deposit number or normalised designation, e.g. PAO1 and ATCC 15692) with their DOIs and ingestion ids, for corrections upstream.
• CORS for anonymous reads (E9). GET/HEAD on /api/v1/* from any origin gets Access-Control-Allow-Origin: *, never with credentials; preflights asking for Authorization or a write method are refused.
• Rate-limit headers. Limited routes return X-RateLimit-Limit, -Remaining and -Reset; a 429 also carries Retry-After. Limits are unchanged.
• Public API description at /api/v1/openapi.public.json with Swagger UI at /api/v1/docs: only public read endpoints, derived from the routes' own dependencies.
• Typed list items in the OpenAPI description for every paged list, and a typed /stats.

Changed

• /api/v1/jobs, /pathways, /releases and /snapshots use the standard {total, items} envelope. The site's own pages were updated; external scripts reading these lists need .items.

HMM screen and taxonomy tree jobs now show where their results came from, and upload tools submit once per click. No migration.

Added

• Provenance cards for HMM screen and taxonomy tree jobs. HMM screen jobs record and show the HMMER version, the command, the profile library (file, build time, profile count, fingerprint, corpus and toolchain), the E-value threshold and whether GA/TC/NC cut-offs were applied (they are not: the screen runs hmmsearch -E only). Tree jobs record and show the lineage source, the last completed NCBI lineage refresh, the tree method, how plastic columns are credited and whether the topology came from the cache. Older jobs show "not recorded" per row. Parameters shown on the card are no longer repeated under Job details.

Fixed

• Upload tools needed several clicks to submit, and each extra click posted another job (B14). The Submit button was reset in a finally block while the browser was already navigating to the job page. It now stays on "Opening job…" after a successful submit and comes back only when a submit fails. Affects Annotate list, gene and genome, Pathway analysis, HMM screen and Compare genomes.
• Organism search results showed the raw NCBI lineage string. They now show a short trail (domain › phylum › … › genus).
• Footer links were too small to tap (R02-10). Every footer link is at least 24px tall; Quick Links and Contact become 44px rows on touch screens and below 992px.
• Extraction drift report paired same-protein interactions by result. Two payload interactions sharing an accession could be matched crosswise, hiding real differences (paper 377). Pairs are now ranked by identity, actor match strength, experiment overlap, then result and notes. The remaining missing-experiment rows on 411 and 629 are genuine pre-1.35.0 shared experiments and now say so.
• Test isolation: the cached job store leaked between tests, so five HMM screen tests failed on machines without /app. Import order is pinned for alembic so ruff check agrees across checkouts.

V2 can now receive the fields PlasticDB-AI will extract, shows extracted text without editing it, ranks kinetics in common units, and carries its own chemistry reference data. This release carries a migration, 0025_declare_extraction_fields, which adds nullable columns only (protein_variants, enzyme_kinetics, enzyme_properties, observations, experiments) and two CHECK constraints; it backfills nothing. Two new tables, chemical_references and chemical_reference_monomers, ship through create_all.

Fixed

• Every paper exported the same PMID (1.34.0). The PMID lookup's evidence subquery was not correlated to the paper, so papers.csv/json and the BibTeX/RIS exports gave all 584 papers PMID 10103271. Each paper now gets the PMID of its own promoted ingestion, or none.
• Promotion lost properties when payload entries collapsed. Payload interactions with the same paper, plastic and actor merge into one live interaction; each entry's experiments, observations, kinetics and properties are now kept instead of overwriting one another (paper 377: LCC-G's half-life and Tm had replaced LCC-NG's). Existing rows are repaired only by a reopen-and-approve of ingestions 377, 204, 82, 219, 226, 254, 309, 411, 516, 607, 614, 615, 618, 629 and 1030.
• Re-approve now updates owned interactions. A revision updates notes, result and enzyme context on interactions its supersede chain owns; other rows keep their values and record a revision_conflict.
• The extraction drift scan no longer runs in an API pod. In 1.34.0 it ran inside the request and could exceed the API pod's 512 Mi limit. Rescan now queues run_extraction_drift_scan on the release worker; results are stored in extraction_drift_scans / extraction_drift_rows (create_all) and the page and API read the latest completed scan. The scan streams 50 ingestions at a time.
• Drift report accuracy. New collapsed and unstamped classes (not drift); ingestions replaced by an approved revision are skipped; pre-existing paper title/authors/journal count as normalised.

Added

• New extraction payload fields, end to end (payload → column → promotion and revision → drift report → API → pages → exports → /api/v1/vocabulary):
  • protein_variant: synonyms (searchable), numbering_scheme_code (mature | precursor | reference_homolog | pdb_structure | other), numbering_reference (required for reference_homolog), is_originating_paper, originating_doi. The free-text numbering_scheme is kept as extracted; a proposed code is computed on read (numbering_scheme_code_proposed, python -m scripts.report_numbering_schemes), never stored.
  • experiments[]: initial_mass, initial_mass_unit.
  • observations[]: time_point, time_point_unit (h | d | week), sd, n, control_value.
  • interactions[].kinetics and interactions[].properties (single objects): substrate, substrate_type (plastic | compound).
  • The curation form has controls for every new field, so approving never wipes them.
• Chemistry reference (V2-owned). chemical_references holds PubChem CID, ChEBI id, InChIKey, backbone, polymer family and monomers; a draft app/reference/chemistry/core_plastics.json covers the core plastics and their monomers (small molecules verified against PubChem/ChEBI; polymer identifiers null or marked unverified). python -m scripts.load_chemistry_reference is dry-run by default; nothing is loaded by the deploy. Plastic and compound pages and the API show the fields when present.
• PU subtypes. PU-ester, PU-ether and PU-carbonate under PU; the legacy Impranil row moves under PU-ester. Existing PU interactions stay on PU. New payloads naming a subtype ("polyester polyurethane", "polyether-based PU") resolve to the child. Reverses the 2026-08-07 curator refusal, at the maintainer's decision.
• Notes on ambiguous abbreviations. PES (poly(ethylene succinate)), PEF, PEC and Impranil carry a clarifying note on their plastic pages.
• Normalised kinetics. kcat_normalised (s⁻¹), km_normalised (mM) and kcat_km_normalised (M⁻¹·s⁻¹) beside the stored values in the API and variant export; the engineering comparison table sorts on them and marks units it cannot convert (mass-based Km).

Changed

• Extracted text is filtered on display, never rewritten. Public pages hide figure paths and curator-addressed notes through one rule set shared by app/services/text_display.py and app/static/js/text_display.js. The API and exports return stored text unchanged (DECISIONS.md §10). scripts/clean_public_text_artefacts.py --apply now refuses and points to the PlasticDB-AI correction route; its dry-run report remains.
• The legacy "Plastic" placeholder is left out of the plastics list, pickers, search, statistics and the taxonomy tree. Its evidence still shows on its own page (with a note), on organism, protein and paper pages, and in exports. Nothing was reassigned.

Groundwork for keeping V2 in step with the PlasticDB-AI extraction JSONs, plus incremental sync (E5) and richer exports (E10). This release carries a migration, 0024_record_updated_at, which adds an indexed updated_at to proteins, protein_variants, microorganisms, plastics and papers; the PreSync hook applies it. No existing value changes.

Fixed

• Approved revisions now land on variants and strains. A revision sent through /ingestions/{id}/revise and approved updates the payload-sourced fields of variants and strain occurrences its own paper created (and clears fields the revision drops). Rows created by another paper keep their values; each disagreement is listed as a revision_conflict in the approve response, the bulk revision report and the review page. First-time promotions are unchanged. Rows from the v1 import carry no ingestion stamp and are not corrected; genetic_background changes only when the payload states it.
• Glycolic acid formula is C2H4O3 (was seeded as C2H4O2, acetic acid's). The vocab seed corrects the live row only if it still holds the old value. All 97 seeded compound formulas were checked.

Added

• Extraction drift report. python -m scripts.report_extraction_drift (CSV/JSON; --paper, --table, --class) and the admin page /curation/extraction-drift (with GET /api/v1/curation/extraction-drift) compare every payload-sourced live field with the approved ingestion's stored JSON, classed as text_edited, value_changed, missing_live_row, extra_live_row or normalised. Read-only; the scan is batched and cached, with a Rescan button. apply()'s field copies are now named lists the report shares.
• GET /api/v1/vocabulary and /vocabulary/{name}: every controlled list the ingestion schema enforces (17, including measurement types with default units, environment types, strain types), plus plastics (kind, parent, aliases) and compounds, built from the validators' own sources, with per-list and overall versions and ETags.
• POST /api/v1/ingestions/validate (producer auth): runs the payload schema and vocabulary checks and returns errors, unrecognised_fields and unmatched_plastics without storing anything.
• Incremental sync (E5). ?updated_since= on the protein, variant, organism, plastic and paper lists; updated_at in responses; GET /api/v1/changes?since= lists updated, removed and merged records (staged records never appear); ETag/304 on JSON responses; gzip; ?view=summary on protein detail drops the evidence lists (IsPETase was 995 kB).
• Exports (E10). papers.csv/json gain title and pmid; /papers/export?format=bibtex|ris (same filters as the list) and /papers/{id}/export; the variants CSV gains numbering scheme, numbering offset, paper id/DOI/year and tm_celsius.

Hit tables keep every hit and filter on the page; search understands everyday names and typos (D1). No migration.

Changed

• Every hit is stored; subject coverage ≥ 70 % is the default view. Annotate Gene/Genome and Compare Genomes no longer drop hits before storage. The form's coverage/bitscore fields now set the job's starting view (result_metadata.view_defaults; scov 70 when left out, 0 = off). The job page shows "N of M hits shown · K hidden · Show all", keeps the threshold in the URL (?scov=&qcov=&bitscore=), and recomputes per-plastic counts, the best-hit view and the Compare Genomes matrix for the active threshold. 1.31.0 jobs' hit_thresholds are read as their default view; jobs from before 1.31.0 have no coverage and show every hit with a note.
• Downloads match the screen. /jobs/{id}/download and /jobs/{id}/comparison accept scov, qcov, bitscore and all, and write # default_view / # view_filter lines. The comparison TSV/CSV now start with # lines. The compare-genomes zip holds the filtered matrix plus comparison_matrix_all_hits.tsv. A "Download all hits" link applies no filter.
• GET /jobs/{id} stays unfiltered and now also returns view_defaults. /jobs/{id}/comparison gains format=json.
• Search (D1). Plastic names and abbreviations outrank fuzzy hits; words under 4 characters match whole ("PE" no longer finds Perez, "PS" no longer Pseudomonas); "polyether" is no longer stemmed into polyethylene. Raw "rank" scores are replaced by "exact match", "common name" or "similar spelling". Accession matches still rank first.

Added

• Everyday names for plastics (D1). styrofoam/EPS → PS, PET bottle → PET, plastic bag/polythene → LDPE and PE, PVC pipe → PVC, bioplastic → PLA and PHA, plexiglass → PMMA (app/services/search_synonyms.py).
• Typo tolerance (D1). Trigram similarity ≥ 0.5 on queries of 5+ characters against plastic names and synonyms, with a "Did you mean …?" suggestion; a reviewable list keeps real polymer words (polyether, polyamine) from being treated as typos.
• Plastic top card (D1). A query that names a plastic shows "Go to PS — Polystyrene" above the results; /search/grouped returns these and the suggestion.

Bulk review for producer corrections ("sweeps"). This release carries a migration, 0023_ingestion_revision_batch, which adds nullable revision_batch/revision_reason columns (and an index on the first) to paper_ingestions; the PreSync hook applies it. Existing rows read NULL.

Added

• POST /api/v1/ingestions/{id}/revise (producer). Delivers a corrected extraction for a paper already in V2, tagged with a revision_batch (e.g. sweep-2026-10) and a revision_reason.
  • On an approved paper it reopens and sets the new payload in one transaction, under reopen's row lock and through reopen's own code (app/services/ingestion_revision.py), so every reopen guard holds: one open revision per paper, approved papers only. Returns 201 revision_opened. An identical payload opens nothing (200 unchanged).
  • On a pending_review paper it replaces the payload in place (200 patched), unless a curator holds a live review claim.
  • A resent request is a retry: it updates the revision the same batch opened (200 revision_updated).
  • Callable by the account that delivered the paper (an auto row whose chain it submitted) or by a curator with the publication right.
• /curation/revisions and /api/v1/ingestion-revisions/batches[/{batch}[/approve|/reject]] (curators). Lists each batch with a server-computed diff per revision against the record it replaces: interactions added, removed and changed (matched by actor + plastic, not position), with result, strain type, observation count and values, notes and other fields. Approve or reject a selection, or the whole batch.
  • Each revision is approved by the existing approve code (replace mode for a revision of an approved paper), in its own transaction. A failure (e.g. an unresolvable organism) is reported on its row and the rest go ahead.
  • Retrying is safe: decided revisions report already_approved/already_rejected. Approve handles at most limit (default 20, max 50) per call and pages with after_id.
• app.services.extraction_diff, the pure diff function behind the page.

Changed

• reopen now calls the shared open_revision service. Behaviour is unchanged.

Follow-ups from verifying 1.31.0 on the live site. No migration.

Fixed

• Variant pages under /proteins/. /proteins/<variant accession> (e.g. PBRMK6) returned the 404 page; it now redirects (301) to the variant page, as the API already did.
• Accession search. Exact UniProt, GenBank (with or without the version suffix), PDB, PlasticDB accession and PlasticDB id now find the protein, case-insensitively and ranked first, in the /proteins list, /api/v1/search, grouped search and autocomplete. Visibility filtering is unchanged.
• Job pages show only the parameters a job type uses. Taxa Table jobs no longer list E-value or % identity; the taxonomy tree no longer lists "Wild-type only"; HMM screen no longer lists % identity; pathway jobs name the pathway.
• "Mode" shown twice on hit-table job pages; it, and the search thresholds, now live only in the provenance card when one exists.

Added

• Taxa Table provenance. A "Methods & provenance" card with data release, input file, matching method, synonym table and the column layout detected at run time; the taxa TSV gains the same # header lines (the column header row is kept; a leading #OTU ID loses its #).
• /jobs list (R07-7). Input-file column, real result counts and run times.

Tool output researchers can cite (B4, B5, B6), plus keyboard (G2) and mobile (G3) fixes. This release carries a migration, 0022_job_result_coverage, which adds nullable qcov/scov columns to job_results; the PreSync hook applies it. Hits stored before it show "–" for coverage.

Added

• Job provenance (B6). Every new job records the data release, reference DB build time and sequence count, DIAMOND version, mode, parameters and input filename in result_metadata.provenance. It appears in a "Methods & provenance" card on the job page and as # lines at the top of the annotate TSV. Older jobs show "not recorded".
• Hit-table coverage and thresholds (B5). Query and subject coverage columns; optional min_qcov, min_scov and min_bitscore on Annotate Gene/Genome and Compare Genomes (off by default); a best-hit-per-query view; live coverage/bitscore filters on the job page.
• Annotate Taxa Table output (B4). Every input column is kept (ASV/feature id included), followed by the match, linked NCBI taxid and PlasticDB URL, positive and negative plastics, paper count, up to 3 DOIs, and a genus-level summary with a caveat for genus-only matches.
• Genus synonyms (B4). Ideonella/Piscinibacter/Pseudideonella, Pseudomonas and its recent splits, and Rhodococcus/Rhodococcoides are searched together when a genus finds nothing under its own name; Synonym_Note says when this happened.

Changed

• Taxa table input (B4). QIIME 2, SILVA and GTDB ; lineage strings are parsed. An empty, header-only or unreadable table now fails with a message saying what was found, instead of completing with no rows.
• Annotate TSV (B5). Header row, plain accessions (no protein: prefix) and the extra columns; built on request, so older jobs get it too.
• Sortable tables (G2). Headers are buttons with aria-sort and a visible focus ring, through one shared script.
• /engineering (G2). Enzyme names are links, the pager is a labelled <nav>, the search box is labelled, and the page has one <h1>. Its inline handlers are gone.
• Mobile layout (G3). Search chips wrap below 992 px (navbar is sticky there); protein and other detail tables wrap at 390 px; the /microorganisms filters collapse below 768 px behind a "Filters (n)" button and open when filters are active; the chat box fits on a phone screen.

Fixed

• Hit order (B5). Hits are sorted by query (input order), e-value, bitscore and accession everywhere; Job.results had no ORDER BY.
• Accessible names (G2). Docs cards have no empty links; the chat widget, scroll-to-top button and profile image have names.

Trustworthy tool results and counts: Compare Genomes (B7), Pathway Analysis (B8), Taxonomy Tree (B9) and honest organism counts (C4). This release carries a migration, 0021_strain_genetic_background, which the PreSync hook applies. Every existing strain starts as unknown, so the migration itself changes no count.

Added

• Strain type (C4). strain_occurrences.genetic_background takes the values wild type, expression host, knockout, engineered strain or not classified.
  • Expression hosts, knockouts and engineered strains are never credited to their species. That covers badges, counts, "First Reported", the plastic page, stats, exports and snapshot, the Taxonomy Tree and Annotate Taxa Table.
  • Their results stay on the paper and protein pages, labelled (e.g. "expressed in E. coli BL21(DE3)"). interactions.csv gains a genetic_background column.
  • Curators set the type on the ingestion review form ("Strain type") or at the new /curation/evidence-checks.
• /curation/evidence-checks (and GET /api/v1/curation/evidence-checks). It shows strain-type proposals, and read-only lists of positives with no experiment, values copied across isolates, positives showing mass gain, and catch-all "bacterium"/"fungus" taxa where a genus is named.
• python -m scripts.propose_genetic_background. Dry run by default; --apply writes only the high-confidence class.
• Compare Genomes matrix (B7). A genome × plastic matrix counts the query sequences that hit an enzyme with its own positive result for that plastic. Hits with only negative or inconclusive evidence are listed separately.
  • Counts are normalised per 1,000 proteins (BLASTP) or per Mb (BLASTX).
  • The matrix is shown as a heatmap, a per-genome bar chart with a text alternative and PNG/SVG export, and a table.
  • It is also served by GET /api/v1/jobs/{id}/comparison?format=tsv|csv, the comparison field, and comparison_matrix.tsv in the zip.
• Pathway Analysis results (B8). A per-step table shows substrate → product, EC, best hit, identity, coverage, and present/absent with the reason. There is also a missing-steps list and an "N of M steps present" summary. The results TSV has a header and per-hit step columns, and the tool page lists references per step and flags single-organism steps.
• Taxonomy Tree downloads (B9). Newick (plasticdb_taxonomy_tree_<job>.nwk), standalone SVG with the plastic key, and PNG at 2x. Clades collapse and expand on click, and there is "Expand all" and a label-size slider.

Changed

• Evidence badges count distinct papers (C4). Per-plastic results gain positive_papers, inconclusive_papers, negative_papers and paper_count.
• Pathway Analysis references (B8). Each step is judged against the curated enzyme plus every PlasticDB protein with positive evidence and the same EC number (or a curator link), not P. sakaiensis alone.
• Pathway step presence (B8). A step counts as present only when a hit has at least 40 % identity, an e-value of at most 1e-10, at least 70 % coverage of the reference, and at least 50 % coverage of the query. The thresholds are adjustable and stated on the page.
• "Build Phylogenetic Tree" is now "Taxonomy Tree" (B9). Each page says it is an NCBI-taxonomy cladogram: ranks only, no sequence comparison, no branch lengths. URLs and the build_tree job type are unchanged.
• Taxonomy Tree leaves out placeholders (B9). Synthetic constructs and uncultured, unidentified, unclassified or metagenome names, plus bare "bacterium"/"fungus", are left out and named in the footer. The PET tree goes from 273 to 267 leaves. Annotate Taxa Table also treats a bare "fungus" as a placeholder.
• Job API hit proteins (B7). The protein object on job hits carries source_microorganism (public only), its own degradation_links and paper_count. The always-empty papers and paper fields are removed.
• d3 7.9.0 is vendored instead of loaded from the CDN.

Fixed

• "Tested whole only" (C4). evidence_scope=whole_organism on the organism list still showed via-enzyme rows in its badges and papers.
• Pathway Analysis false positives (B8). An E. coli K-12 proteome lit pathway steps from 42–64-residue partial alignments. The page also said "ten curated enzymes" while naming six.
• Base64 Compare Genomes endpoint (B7). POST /api/v1/bioinformatics/compare-genomes saved one file the worker never read. It now takes one fasta_b64 per genome.
• Taxonomy Tree viewer (B9). It drew no plastic dots for names containing brackets or punctuation, and the "Taxa" button never toggled the labels.

Small P1 fixes from the pre-release sweep: correct sequence-type handling and tool docs (B10, B11), release stamps on downloads (E6), stricter and more consistent API behaviour (E4), accession lookups (E7), API docs (E8), PlasticDB v1 ids (A7), and removal of the iTOL embed (D2, deployed from #539). No migration. After deploy, run python -m scripts.backfill_v1_legacy_ids (dry run, then --apply).

Added

• Accession lookups (E7). /variants/{ref}, /variants/{ref}/fasta (new), /proteins/{ref}/fasta, /proteins/{ref}/structure and /microorganisms/{ref} accept a PlasticDB accession as well as the numeric id or tax-id. Accession lookups ignore case. A variant accession on the protein routes answers 301 to /variants/{accession}.
• PlasticDB v1 ids (A7). The protein detail API has a v1_id field, set only when the id is verified against the v1 FASTA by accession or identical sequence. /proteins/<v1 id> (e.g. /proteins/00025) redirects to the enzyme v1 gave that number to. scripts/backfill_v1_legacy_ids.py (dry run by default; --apply, --clear-only) recomputes legacy_id.
• Release stamp on downloads (E6). Every bulk download names its data release in the file name (plasticdb_proteins_2026.39.12.csv, plasticdb_dataset_2026.39.12.zip) and in an X-PlasticDB-Data-Version header.
• API documentation (E8). Python (requests) and R (httr2) examples on /documentation/api and /profile/api-keys. The docs gain sections on error format, sorting, identifiers and unknown parameters. Tests check every documented example against the OpenAPI schema, and run the anonymous ones against a seeded PostgreSQL.
• Search settings stated (B11). The tool pages and documentation state DIAMOND 2.0.6 at default sensitivity, up to 25 matches per query, no coverage cutoff, the TSV columns, and HMMER 3.4.

Changed

• Unknown filter values are a 422 instead of a silent fallback (E4). This covers evidence_method, substrate_form, evidence_scope (on detail, /stats and /plastics: organism_or_enzyme or whole_organism), the plastic-detail sort, order and polarity, and NaN/inf kinetic thresholds.
• One error format (E4). Every 422 from a read endpoint is {"detail": [{type, loc, msg}]} and names the parameter. 429 bodies use {"detail": …}.
• Sorting (E4). NULLs sort last in both directions, and text sorts ignore case.
• 404 instead of 200 (E4). /snapshots/latest is a 404 until a snapshot exists, and /pathways/by-plastic/{id} is a 404 for an unknown plastic.
• Anonymous routes in OpenAPI (E8). Routes with optional auth, such as POST /tree/build, are marked as allowing anonymous calls.
• E-value defaults agree (B11). The tool pages default to the API's E-value, 1e-5, and every DIAMOND tool offers 1e-3 to 1e-50.
• "Legacy ID" means v1 only (A7). The label appears only for verified v1 enzymes, as "Legacy ID (PlasticDB v1)". Paper promotion no longer copies the V2 serial (plasticdb_id) into legacy_id; the serial itself is still issued and unchanged.

Removed

• The iTOL embed on /interactive-tree (D2). The bare URL redirects to Build Tree, and ?job=… still shows a built tree. The menu links, the iTOL line on /privacy and itol.embl.de in the CSP frame-src are gone.

Fixed

• Wrong sequence type (B10). Nucleotide sequences in BLASTP mode or HMM Screen, and protein sequences in BLASTX mode, are refused with a message naming the records ("Choose BLASTX, or translate/call genes first (e.g. Prodigal)"). The page checks before uploading, the API returns 422, and the worker re-checks the whole file. Such jobs used to "complete" with no hits.
• LIKE wildcards (E4). % and _ in search and filter text acted as SQL wildcards (lineage=% matched every organism).
• v1 ids (A7). Every protein showed its V2 serial as its "Legacy ID", so v1 citations landed on the wrong records. v1-format DIAMOND hits (00025|…) resolved through that serial too, and could dedupe against the wrong protein.
• Tool docs (B11). They no longer say "paste" or promise "summary statistics" on Annotate Genome.
• /download (E6). It no longer says both "live" and "weekly": downloads are live, and only new organisms and proteins wait for the weekly release.
• API docs (E8). The API docs and API-keys page had wrong filter semantics, stale counts, a placeholder host, contradictory rate limits and incorrect auth labels.

A fourth plastic kind, formulation, for one polymer plus non-polymer additives. Migration 0020 adds the label to plastic_kind_enum. The deploy's vocabulary seed then relabels the rows, after the migration has committed.

Added

• formulation as a value of Plastic.kind, for one polymer plus non-polymer additives. Migration 0020_plastic_kind_formulation runs ALTER TYPE plastic_kind_enum ADD VALUE IF NOT EXISTS 'formulation', inside the migration transaction as 0009 and 0017 do (allowed since PostgreSQL 12, because nothing uses the label in that transaction). The downgrade moves formulation rows back to blend and leaves the unused label on the type, because PostgreSQL cannot drop an enum value.
• The plastic page labels a formulation sub-type "(formulation)", as it labels a blend "(blend)". The label warns that evidence reached through "Include sub-types" is not evidence on the pure polymer.

Changed

• OXO-PE and pPVC are formulation, not blend (ruling 2026-10-01). They are PE plus a pro-oxidant and PVC plus a plasticiser, not blends of polymers. Both the seed terms and PLASTIC_KIND_CORRECTIONS (blend → formulation) are updated.
• P34HB is copolymer. It is a spelling of P(3HB-co-4HB), and was unclassified.
• PlasticKindCorrection.from_kind accepts None, meaning "only while the row is still NULL". Like the other corrections, it never overwrites a kind a curator has set.
• The vocabulary seed skips a kind the live plastic_kind_enum does not have yet, instead of failing the whole seed. The row keeps its kind until the next seed after the migration. This covers scripts/seed.py against an un-migrated database; init_schema always migrates first.
• scripts/check_migration_deploy_safety.py preflight reports the plastic_kind_enum labels and the kind of the three relabelled rows.

Two more compounds. No migration; they ship through the deploy's vocabulary seed.

Added

• Hydroxyapatite and Flax fiber as filler compounds.

Two vocabulary additions. No migration; they ship through the deploy's vocabulary seed.

Added

• ENR (epoxidised natural rubber) as a plastic, a sub-type of NR in the polydiene class.
• 1,4-Phenylene diisocyanate (PPDI) as a compound.

Signal peptides are predicted locally with DeepSig. The privacy and terms pages are final. Blend and PHA plastics are grouped under their parent polymers. No migration; the plastic hierarchy changes ship through the deploy's vocabulary seed.

Added

• DeepSig signal-peptide prediction in the worker image. It uses DeepSig 0.9 (GPL-3.0) in an isolated Python 3.11 with CPU-only TensorFlow 2.15.1. The model weights are baked in and checked by checksum, and the build runs a smoke test that IsPETase is cut at residue 27 and thioredoxin-1 has no signal peptide. Signal peptides are now predicted on PlasticDB's servers instead of by EBI Phobius, which remains a fallback only when DeepSig is missing. Each job makes one DeepSig run instead of one per sequence. Archaea jobs state that the Gram-positive model was used. The image grows by about 1.1 GB.
• scripts/erase_user_account.py erases an account on request. It is a dry run unless given --apply. The account is scrubbed in place: the name becomes "Former contributor" and the email a dead address, the credentials are cleared, and chats and jobs are deleted. Published records stay under CC BY 4.0, credited to "Former contributor". It refuses superusers.
• Blend sub-types are labelled "(blend)" on the plastic page, and the plastics API returns kind on each child plastic.

Changed

• /privacy and /terms are final. They state that the site is run by the PlasticDB team under New Zealand's Privacy Act 2020, with requests to contact.plasticdb@gmail.com, and that New Zealand law and courts govern the terms.
  • The AI assistant section says the question, up to 20 earlier messages and PlasticDB's own query results are sent to DeepSeek, a third-party service, and links DeepSeek's terms and privacy policy.
  • The server-log, signal-peptide, access/correction and deletion sections describe what the service actually does. Inactive accounts are not deleted.
• Single-base blends are sub-types of their base polymer: LDPE, LLDPE, PE, PBAT, PBS, PBSA, PCL, PHA, PHB, PLA, PS, PU, PVA and PVC Blend. ABS/SAN blend, Mater-Bi, Naturecycle and PA-EA stay top-level.
• PHB and the other legacy polyhydroxyalkanoates sit directly under PHA: PHV/P3HV, P3HP, P4HB, PHBV and its duplicate spellings, P(3HB-co-4HB)/P34HB, P(3HV-co-4HB), PHBH, PHBVH, PHO, PHN and PHPV. include_subtypes reaches PHB-Blend through PHB. Headline counts still count direct evidence only.
• The tools documentation describes signal peptides as DeepSig, not SignalP 5.0.

Fixed

• PHBV was classified as a blend. It is a copolymer, like its duplicate spellings. The fix comes from a new kind-correction step in the seed that changes a row only while it still has the known-wrong value, so a curator's edit is never overwritten.
• The Annotate Taxa Table docs claimed names are resolved against NCBI. They are matched against PlasticDB's own organism records.
• A failed DeepSig run no longer leaves scratch files in the worker's working directory.

LLDPE and O-PE become sub-types of PE. No migration.

Changed

• LLDPE and O-PE are sub-types of PE (second maintainer ruling, 2026-09-30). plastic=PE&include_subtypes=true now also includes their evidence, and PE's children lists them. As with LDPE and HDPE in 1.27.1, the change ships through the vocabulary seed at deploy and only fills an empty parent. Headline counts are unchanged. O-PE (oxidised polyethylene) and OXO-PE (the pro-oxidant formulation) stay two separate rows. The * Blend anchors (e.g. LLDPE Blend, PE Blend) are still undecided and keep no parent.

Hotfix: dataset.zip no longer runs the API pod out of memory, and the text-cleanup script no longer leaves broken sentences. Also: LDPE and HDPE become sub-types of PE, and an E-value of 0 displays correctly. No migration.

Fixed

• GET /api/v1/downloads/dataset.zip OOM-killed the API pods (exit 137 at the 512Mi limit), and one anonymous request was enough. The archive was built in memory, and the evidence-spine CSVs added in 1.27.0 made it too large. It is now streamed from the database in batches of 250 rows into a ZIP file on disk, and sent from that file with a Content-Length. Peak memory no longer grows with the data: on a test database with 30,000 observations the build added 29 MiB of RSS, against 280 MiB before (27 MiB against 251 MiB traced). The archive still contains README.md, LICENSE.txt and every entity and spine CSV, and staged and tombstoned records are still left out.
• scripts/clean_public_text_artefacts.py left sentences ending on a dangling word ("Read from.", "…annotated spectrum in."), about 132 of them in the production dry run. When a figure path was the object of a preposition, the preposition is now removed with it. If that leaves only a stub, the stub clause or sentence is dropped: Read from Fig. 1b (figures/…png). becomes Read from Fig. 1b., and Read from (figures/x.png). is removed entirely. The script is still dry-run unless given --apply.
• GET /api/v1/plastics returned children: [] for every plastic. It now lists each row's sub-types, and children on the plastic detail are sorted by name.
• An E-value of exactly 0 showed as "–" in tool job results (/jobs/{id}). It now shows "0", small values use scientific notation (e.g. 1.2e-45), and "–" means the value is missing.

Changed

• LDPE and HDPE are sub-types of PE (maintainer ruling 2026-09-30). plastic=PE&include_subtypes=true on organisms, proteins, papers, downloads and the tree now includes their evidence. The change ships through the vocabulary seed at deploy and only fills an empty parent. Headline counts are unchanged, because they count only direct evidence.
• dataset.zip is cached on the pod's disk. It is built at most once at a time per pod, behind a lock, and kept for DATASET_ZIP_CACHE_TTL_SECONDS (default 600). The cache key includes the release and each table's row count and max id, and the public-row counts for proteins and organisms. So new rows, a release, or a record being staged or tombstoned all trigger a rebuild straight away. Edits to existing rows show up within one TTL. Repeat requests are served from the file (X-PlasticDB-Cache: hit) without touching the entity tables. If there is less than DATASET_ZIP_MIN_FREE_MB (default 256) of free disk, or another build is still running after 5 minutes, the endpoint returns 503 with Retry-After. DATASET_ZIP_CACHE_DIR defaults to <tempdir>/plasticdb-dataset-zip. DATASET_ZIP_CACHE_TTL_SECONDS=0 turns the cache off: each request builds its own temporary file, which is deleted after it is sent.
• Inside the ZIP, README.md now comes after the CSVs, because it lists their final row counts.

Pre-release sweep fixes: tool correctness (B1–B3), exports and API (E1–E3, A4), licence (A1), HMM Screen access (B12), public-text markup (C10) and contrast (G1). No migration.

Added

• Data licence: CC BY 4.0 (A1). The licence and the citation are in the footer of every page. There are new /license (with a /licence redirect), /terms and /privacy pages; facts the maintainer has not confirmed are marked DRAFT. OpenAPI now carries info.license and termsOfService. API responses carry a Link: rel="license" header, and GET /api/v1/releases/current returns the licence fields.
• Submission consent (A1). Submit Data requires agreeing to publication under CC BY 4.0: POST /api/v1/submissions needs "license_accepted": true, and the consent is recorded with the submission.
• Evidence spine in downloads (E1). dataset.zip, dataset.json and release snapshots include interactions, experiments and observations tables that join to organisms, proteins, plastics and papers. They also carry README.md and LICENSE.txt and a metadata block with the licence, citation and a table guide. New columns: n_papers, paper_ids, paper_dois, evidence_by_plastic, source_protein_accessions and credited_microorganism_id.
• Evidence on tool hits (B1). Job hit results carry evidence / evidence_polarity, and the job page has an Evidence column. Annotate Taxa Table adds Evidence and Positive_plastics columns and a summary (B3).
• PlasticDB_registry.dmnd (every public record), used only for submission deduplication.
• include_subtypes on /microorganisms, /proteins, their exports and tree building. A page parameter on public list endpoints. sort_by=id everywhere, and title/journal for papers. A Papers entity and a sub-types option in the custom export form.
• Superuser-only GET /api/v1/admin/releases.
• variant_degradation_links on GET /api/v1/proteins/{accession}.
• Scripts, both dry-run unless given --apply:
  • scripts/fix_scaffold_and_synthetic_proteins.py: checks the six records behind B2.
  • scripts/clean_public_text_artefacts.py: removes figure paths and curator-only notes from public text, and reports truncated titles (C10).
• Automated WCAG AA contrast checks: tests/test_theme_contrast.py and tests/e2e/contrast.spec.js.

Changed

• Tool reference databases contain only records with a positive result of their own (B1). The wild-type database also excludes synthetic-construct sources. The rebuild fingerprint carries a policy version, so this release rebuilds the databases automatically.
• plastic= is an exact match (E2). On microorganism and protein lists and exports it matches name, abbreviation or full name exactly, ignoring case, so PE no longer matches PET or LDPE.
• HMM Screen is open to every signed-in user (B12). The hmm_screen_public setting is removed.
• Public /releases returns only version, date, counts and DOI (A4).
• A protein's own evidence is shown separately from its variants'. degradation_links and plastics on the protein detail cover only the protein's own evidence; its variants' evidence is listed separately.
• Organism export rows credit evidence the way the organism page does, and their per-plastic columns line up slot by slot.
• Annotate Taxa Table labels genus-level matches genus. Fuzzy matching stays within the query's genus and never matches placeholder names.
• Theme colours are defined once as tokens in plasticdb.css. Yellow fills carry dark text.
• Google Analytics on dashboard pages loads in production only.
• The snapshot licence changes from MIT to CC BY 4.0.

Fixed

• An E. coli K-12 negative control reported unmodified scaffold proteins and negative or inconclusive records as 100 % "PET" hits (B1).
• Scaffold proteins whose only result belongs to an engineered variant (E. coli OmpA/EfeO/FliD/CysP) read as PET-positive on their protein pages.
• Annotate Taxa Table crossed genera (Sphingomonas → Sinomonas), matched placeholder names, and presented negative-only organisms as degraders (B3).
• Paging skipped and repeated records; every sort now ends with a unique tiebreaker. An unknown sort_by or sort_order returns 422, and page is honoured (E3).
• Exports dropped the evidence, credited one paper per organism and misaligned the per-plastic lists (E1).
• The releases API exposed ML-bundle paths, withheld counts and fingerprints (A4).
• <i>, <sub> and <sup> in titles, citations and notes render as formatting through an allow-list, and several pages that inserted citations unescaped now escape them (C10). /openapi.json dropped the licence, contact and terms fields.
• Colour contrast meets WCAG AA across the site, including the register, forgot-password and reset-password pages, which never loaded the theme (G1).

Added

• Plastic Poly(methyl acrylate) (curator ruling 2026-09-30), needed by 10.1016/j.ijbiomac.2020.07.261. Vinyl homopolymer, keyed by name with no abbreviation on the Poly(ethyl acrylate) precedent, because PMA is also poly(malic acid).
• Compound Birch tar (curator ruling 2026-09-30), the 1–10 wt% additive of the PLA/PEG films in 10.1016/j.ijbiomac.2023.124226. No formula (a pyrolysis tar); is_additive=True.

Embargoes, deposits and visibility holds are removed; all curated data is public after the weekly release. Maintainer decision, recorded in DECISIONS.md §8. Breaking: the deposits API is gone. This release carries a migration, 0019_remove_embargo, which changes data; the PreSync hook applies it.

Removed

• Deposits and embargoes. The /api/v1/deposits API (create, /mine, /queue, /review/{reviewer_token}, get, /release, /notes, /withhold, patch), the Deposit model and its member tables, deposit_id on proteins and organisms, the daily release_due_deposits Celery task and beat entry, the /curation/deposits page, the curator-inbox "deposit" kind, the embargo / deposit card on Submit Data (and its JS), the deposit badges on the review pages, and the contributing guide's #embargo section (replaced by #release, "When your records become public").
• Operator visibility holds (the negative-results hold). app/services/visibility_holds.py, the hold columns and their session guards (VisibilityHoldMixin, the before-flush visibility deflection and the two deletion guards, HeldRecordDeletionError and its 409 handler), the /api/v1/admin/protein-visibility/* and /api/v1/admin/microorganism-visibility/* routes, and the admin panel's Visibility tab. Hold-awareness is gone from orphan cleanup, supersede (SupersedeRefused), dataset releases (the held-and-public invariant), protein merge, protein dedup, the duplicates report and the annotation protected-field list.
• private and embargoed as privacy states in the admin state filter, state badges and admin preview. private remains only as the parking value of a tombstoned/merged row; embargoed stays in the database enum as an unused label.
• The "Who can see held records" HMM disclosure on Submit Data.
• POST /api/v1/submissions no longer accepts a deposit block. The schema forbids unknown fields, so a client still sending one gets a 422.

Changed

• Migration 0019_remove_embargo (data): held, non-tombstoned proteins and organisms become public; held tombstones have their hold cleared and stay private; any non-tombstoned private or embargoed row becomes public; deposit_id is cleared (staged deposit rows stay staged for the next release). It then drops the hold columns, deposit_id, the deposit tables and the visibility_hold_reason_enum / deposit_visibility_enum types. accession_registry rows minted for deposits are kept (accessions are never reused). The downgrade restores the schema, not the data.
• Weekly release, hardened against a silent no-op. releasable_models() is an explicit (Microorganism, Protein) instead of being derived from the removed hold mixin. releasable_filter is visibility = 'staged' AND NOT is_tombstoned, and raises for a model it cannot gate instead of returning "publish nothing". The withheld report is now eligible / staged_but_tombstoned, and GET /api/v1/admin/releases/staging drops held_and_public.
• Protein merge: the survivor ranking is staged < public, and a tombstoned member can never be the survivor (a new tombstoned refusal covers an all-tombstoned group). The deposit_split refusal and deposit-link re-homing are gone.
• The accession merge/tombstone routes moved from deposits.py to app/api/v1/accessions.py. The URLs are unchanged: /api/v1/admin/accessions/merge and /tombstone.
• scripts/label_matrix_api.py enumerates proteins through GET /api/v1/admin/proteins, 200 per page, because the protein-visibility list it used is gone.
• The submission-envelope guard in apply() stays. Its stated purpose is now what it still protects: acceptance, contributor credit and structure folds. Submission approval stays superuser-only, with the rationale reworded.
• The HMM-screen rationale text (config, admin settings, tools page, hmm_access) no longer cites held rows. hmm_screen_public and its default are unchanged; opening the screen is a separate decision.
• Ingestion withdrawal no longer refuses a paper for belonging to a deposit.
• CLAUDE.md's required pg-tier test is now test_generate_snapshot_excludes_staged_and_tombstoned_records.

Pre-release sweep fixes F1–F10 (accounts, submission, curation tooling). This release carries a migration, 0018_ingestion_assignment (adds paper_ingestions.assigned_to_id / assigned_at); the PreSync hook applies it.

Added

• Signed-in pages keep their API token fresh (F1). New page route POST /session/token exchanges the session cookie for a short-lived bearer token (same-origin, header-gated, no-store). The API itself still refuses the cookie. static/js/auth_session.js fetches a fresh token when it is missing or stale, retries a 401 once, and shows a "Sign in again" banner when the session itself has expired, instead of chat, dashboard and admin pages failing silently.
• Curator Inbox (F5) at /curation/inbox: every pending queue in one list, with whole-backlog counts, age, sort, filters and assign-to-me. Assignment is a durable to-do mark; only superusers can assign others.
• Read-only curator reports (F5): duplicate proteins with the merge tool's verdict, unresolved organism names, and the curation activity (audit) log with old → new values.
• Synonyms through the admin API (F5). Protein and microorganism synonyms can be edited; edits are audited and searchable.
• Curator Guide (F6) at /documentation/curator-guide (also docs/curator-guide.md): review policy and the R-16…R-22 extraction rules. Judgement calls the maintainer has not ratified are marked DRAFT.
• Admin preview (F7) of staged, held, private and merged proteins and organisms at /admin/preview/{proteins|microorganisms}/{id}, and GET /api/v1/admin/proteins, which lists every protein in every state, paged and searchable.
• Ingestion review, one record at a time (F8). Cards collapse to a summary with Previous/Next; an opt-in source pane beside the form shows the paper text (with find) or the PDF.
• Flag queue links (F7). Flags on interactions, experiments, observations and variants link to their paper, or say the record no longer exists. A "Waiting" column shows age.
• Honest DOI lookup (F3). It now reports found / not found / unavailable / invalid, fills in the journal and title, and says when the paper is already in PlasticDB. There is a Paper title field.
• Submission checks (F3). Values are checked against physically plausible bounds (app/schemas/submission_bounds.py, enforced by the API and mirrored on the page), and a citation is required. Drafts are saved in the browser with an unsaved-changes prompt. The review step shows every value entered.
• Embargo and deposit guidance (F4). The visibility choices are explained, it is disclosed that accepted held sequences feed the HMM profile library, a "What happens after you submit" panel is added, and the contributing guide has an embargo section.
• Readable picker labels (F10) (term_labels, plastics[].full_name on /api/v1/vocabularies).
• /contributors directory (F9) and GET /api/v1/curation/contributors: name, ORCID, suggestions written, suggestions accepted and data submissions made. No email is shown.
• The login page explains itself (F9): why an area needs an account, and what signing in unlocks.

Changed

• /curation/stats counts the real backlog; the paper review list shows age and assignee; the sidebar leads with the Curator Inbox; /curator-guide opens the Curator Guide.
• Admin Visibility tab shows the real state (merged, held, staged, embargoed, private, public) instead of lumping them under "private".
• Ingestion review: upload, remove and re-run extraction are fenced in a "Manage source files" section, and Approve and Reject sit apart.
• Deposit visibility has no preselected default. The release date must be between tomorrow and 3 years out.
• The submit form hides fields irrelevant to the record type, and the Oxygen-requirement help text is corrected. DOIs are stored without the https://doi.org/ prefix.
• /forum tells non-members that it is for signed-in members only, instead of redirecting. Unverified users are no longer looped to /login.
• Contributor profiles explain how each counter is earned. The unused "Evidence Claims" counter is removed.

Fixed

• /curation/review never loaded (F2). It read the wrong localStorage key. It now shows a real error on failure, and opening a data submission works.
• A submission requesting an embargo with a release date failed with a 500.
• A non-existent DOI was reported as "Found."; the citation was sent as the paper title.
• Admin: Engineered tab names showed "undefined", creating a variant always failed, and edits saved nothing. The Proteins and Engineered tabs were capped at 500 rows and hid held and staged records.
• Contributor counters credited the curator who accepted a suggestion instead of its author, and counted reviews as data submissions. They are now derived from the source records, so no backfill is needed.
• The dashboard told signed-in users to "Login" while their jobs loaded.
• Open redirect through /login?next=.

Changed

• /pathways/{id} no longer redirects old plastic-keyed links. 1.24.0 sent /pathways/16 (PET's old address) to /plastics/16/pathways whenever no pathway 16 existed. Pathway and plastic ids share one number space, so that link would have silently started showing an unrelated pathway once a 16th pathway was curated. The id is now only ever a pathway id; anything else is a 404. Use /plastics/{id}/pathways for a plastic's pathways.

Fixed

• test_curation_form_roundtrip generated payloads the schema rejects. Its free-text pool ignored column length limits (e.g. temperature_unit ≤ 8 characters), so seed-dependent runs failed with a validation error. It only runs where Node is installed, which the tests CI job is not, so it was red locally only.

Pre-release sweep fixes D3–D8 (browse, search, statistics). Merged from #521–#526.

Added

• Filter papers by evidence. /papers and /api/v1/papers now filter by plastic (exact abbreviation or full name, where PE never matches PET, with optional include_subtypes), microorganism (tax ID, accession or name, including enzymes purified from it), protein (accession), evidence_method, result and has_quantitative. All of them apply to one reported result, so "PE + weight loss" means a weight-loss assay on PE. Filtered views are linkable, and unknown method or result values return 422.
• Papers on the plastic page. Each plastic page shows how many papers report evidence on it, with links to the filtered paper list. The API exposes the count as paper_count.
• Assay-condition filters on Microorganisms. Method and substrate form (which the API already accepted) are now labelled selects, and there are new sole carbon source (yes / no / not stated) and oxygen (aerobic / anaerobic / microaerophilic / not reported) filters. All conditions must hold in the same experiment, on evidence credited under the chosen evidence scope.
• Protein design filters. Experimental structure (PDB) yes/no, sequence yes/no, EC-number prefix (3.1.1 matches whole levels), and a Tm range. Each entity is matched on its own values, and variants inherit the parent's EC number. The proteins CSV/JSON/FASTA exports accept the same filters.
• Plastics can be sorted by evidence. The Microorganisms and Proteins count columns sort on click (largest first), and an Evidence filter hides or shows the plastics with no recorded degrader (has_evidence, also on the plastics exports).
• /plastics/{id}/pathways lists every degradation pathway for a plastic, with a selector and permalinks.
• Text alternative for pathway diagrams. A collapsible, ordered list of steps (substrate → enzyme (EC) → product) sits under the diagram on the pathway and plastic pages, and the drawn diagram is labelled for screen readers.
• Strain on collapsed evidence rows. Species pages show the strain designation without expanding the row.
• Easier navigation on long detail pages. Evidence conditions tables carry a screen-reader caption (plastic, outcome, strain/paper). Organism, protein and paper pages get an "On this page" section nav.

Changed

• "Top isolation environments" counts degrading microorganisms. It used to count every isolation record, whatever the test result, so one 1,000-strain fungal screen with mostly negative results made plant material the top environment. It now counts distinct public organisms with at least one positive result, like the tiles and the plastics chart; soil (158) now leads. Each number matches the microorganism list filtered to that environment.
• /api/v1/stats/charts environments carry code and a human label, plus a top-level current_year. label previously held the raw code.
• Genus pages mark organisms with no degradation evidence. Taxonomy stats gain evidenced_microbe_total and a per-member evidence_count. Members with no evidence are labelled "No degradation evidence recorded" and listed after those that have evidence.
• "Temperature class" is now "Growth temp. class" on the Microorganisms filter and the genus page, with a note that it comes from the recorded optimum growth temperature, not the assay temperature.
• The Proteins substrate-form filter lists every form in the vocabulary, not 5 of 9.

Fixed

• Pathway pages are addressed by pathway id. /pathways/{id} now shows that pathway; it used to treat the number as a plastic id, so /pathways/1 showed "No Pathway Found" with HTTP 200. Old plastic-keyed links 301 to the plastic's pathways, and unknown ids return a real 404. The breadcrumb and title name the plastic ("PET", not "Plastic #16"), and a plastic with no pathway shows a neutral message with no graph controls.
• Evidence badges count inconclusive results. Organism, protein and variant detail pages now show the ? tally (e.g. PET 30+ 3? 1−), as /documentation/reading-results describes. The detail API had dropped inconclusive_count, so inconclusive + negative was labelled plain "negative", and the organism API disagreed with the plastic page ("bacterium" LDPE 3/0/0 vs 3+ 20?).
• Paper pages say who each interaction is about: the organism (with strain), the protein with its accession, or the variant with its parent accession, instead of just "microorganism".
• Renamed organisms are findable by their familiar names. The Pseudideonella sakaiensis page says "Also known as / formerly Ideonella sakaiensis, Piscinibacter sakaiensis", including in its title and description; the API adds an authority-free also_known_as list. Annotate-list resolves Ideonella sakaiensis, Piscinibacter sakaiensis and bare Ideonella / Piscinibacter to taxon 1547922 (labelled synonym), and /taxonomy/genus/Ideonella (and its API twin) 302s to Pseudideonella instead of 404.
• Statistics charts say what they count. Readable environment names, a subtitle naming the unit on each chart, axis titles, "Plastics with the most degrading microorganisms" instead of "Most-studied plastics", the current year dashed and labelled "(to date)", and "1 Pathway" in the singular.
• The isolation-environment chart no longer counts hidden organisms (private, staged, embargoed, held or tombstoned).
• Paper export ignored the plastic filter. /downloads/papers.csv|json now accept the same evidence filters as /papers.
• Kinetic filters missed values written as min⁻¹ or s^-1, so FAST-PETase (43.42 min⁻¹) was absent from min_kcat=0.5. Unit spellings are now normalised (kcat to s⁻¹); rows with an unrecognised unit are excluded. A wild-type protein no longer matches on its variants' kinetics.
• Unknown temp_class, sole_carbon_source and aerobicity values are a 422 (temp_class=thermophilic silently returned every organism).
• Strict CSP on /papers. The last inline handlers on the page were removed.

Added

• Compound Bisphenol A (curator ruling 2026-09-28): the PU-C chain extender of 10.1002/jbm.a.34777, restored to its substrate_composition after decision D had moved it to notes as a one-off. A commodity monomer (bisphenol-A polycarbonate, epoxies) that will recur.

Changed

• PEA backfilled as poly(ethylene adipate) (full_name, polymer_class; NULL-fill only). Held out of 1.23.12 until the materials mis-filed on it — the Bayer Tir 1874 poly(ester amide) of ingestions 149 and 289 and the poly(diethylene glycol adipate) of 79 — were moved by revisions 1033–1035, approved in replace mode. The only approved records still naming PEA are ingestion 207's (Kasuya et al. 1999), which are poly(ethylene adipate).

Added

• **33 plastics terms, 1 legacy backfill and 2 compounds, all on one curator ruling (2026-09-27).** Each unblocks an extraction that already exists and fails vocabulary validation. Every row carries its reasoning and DOI(s) in app/services/materials_vocab.py.
• PHA copolymers — P(3HB-co-6HH), P(3HB-co-3H4MV), P(3HB-co-3HP), P(3HB-co-3HV-co-4HB), P(3HB-co-mcl-3HA), PHACOS and the homopolymer PHH (poly(3-hydroxyheptanoate), the missing C7 of PHHx/PHO/PHN/PHD), all parented to PHA. P(3HB-co-6HH) is not PHBH (3- vs 6-hydroxyhexanoate) and not PHB-b-PCL (random vs block); P(3HB-co-3H4MV) is not PHBV. Compounds 6-Hydroxyhexanoate and 3-Hydroxy-4-methylvalerate make their ratios writable.
• Polycarbonates — PTMC and PPC, and a new polycarbonate class.
• Other common plastics — PLCL, P(BS-co-ES), `poly(trimethylene adipate), PMMA, Poly(ethyl acrylate), PVP, P(S-co-DVB), SAN, ABS`, PPS (class polyarylene sulfide) and NBR (declared as SBR is).
• PCL/PLA–PEG block and graft copolymers, one canonical name each — PCL-b-PEG, PCL-b-PEG-b-PCL, (PCL)2-b-PEG-b-(PCL)2, PEG-b-(PCL)2, PCL-g-PEG, PLA-b-PEG-b-PLA, PCL-b-PLLA, PVP-b-PCL. All unparented, on the PHB-b-PCL rule.
• Blends and commercial materials — NR Blend (parent NR), PA-EA, ABS/SAN blend, and the trade-named mulch films Naturecycle and Organix (composition proprietary and unstated, the Mater-Bi rule).
• Anchors for covalent one-offs (rulings C and E). PCL Copolymer (parent PCL) is the covalent counterpart of PCL Blend: a graft, network, block or random copolymer of PCL, with the partner in substrate_composition. Recurring copolymers keep rows of their own. It covers 12 papers. The class anchors Aliphatic polyester (other), Aromatic polyester (other) and Poly(ester amide) hold one-off polyesters, with their monomers in the composition. Poly(ester amide) also receives the poly(ester amide)s now mis-filed under legacy PEA (ingestions 149 and 289), by revision after this release deploys.
• Bis-urea elastomers (ruling D) — PC-BU, PC(e)-BU and PCL-BU (10.1016/j.actbio.2019.05.037; PCL-BU is also the PCL-U4U of 10.1089/ten.TEA.2013.0517). Each is classed by its soft block. The (e) means ester, not ether. These are not polyurethanes: the paper says they contain no urethane group. They are not a subdivision of PU.
• PBLG (ruling G), a synthetic polypeptide in a new polypeptide class (a protease substrate, not a nylon), and judgement-call rows the curator is asked to confirm: PTMC-b-PBLG, PPDO, Cellulose butyrate (no CB abbreviation, because that is carbon black) and PCL diol (parent PCL).
• 45 compounds (rulings B-E). Fillers and plasticisers (Montmorillonite, Tricalcium phosphate, Calcium carbonate, Carbon black, Multi-walled carbon nanotube, Boron nitride, Triacetin, Cyclic polyphenylene sulfide). Biopolymer partners with is_additive NULL (Zein, Gelatin, Collagen, Agarose). Polyurethane/urea hard- and soft-segment monomers (HDI, IPDI, BDI, H12MDI, ethylenediamine, putrescine, Poly(tetramethylene oxide)). The partners of the PCL copolymers, and the monomers of the one-off polyesters and poly(ester amide)s, plus 5-Hydroxyhexanoate (the delta-hexalactone unit of poly(CL-co-HL)). Names match the remapped PlasticDB-AI extractions exactly.

Changed

• PTS merged into PBS (curator ruling H, 2026-09-27). PTS is poly(tetramethylene succinate), the Tokiwa lab's name for PBS (v1: "PTS (BIONOLLE-1020)", a PBS grade). New MERGED_PLASTICS / merge_plastics, run by the PreSync seeder after the upsert: every foreign key onto the old row (discovered from the model metadata) is re-pointed at the survivor, duplicate monomer/blend-component links are folded, the old row is deleted, and a per-table census must balance or the transaction rolls back (PlasticMergeInvariantError). No interaction, experiment, observation, kinetics row or pathway is ever deleted; a (paper, actor) pair with evidence on both rows keeps both interactions and is counted in the log line. Idempotent, and a no-op on a fresh database. Measured in production before writing it (read-only): row 67 carries 33 interactions (4 organisms positive, 18 negative), all from 10.1128/aem.64.12.5008-5011.1998 via approved ingestion 202; nothing else references it, and no actor is on both rows. The PreSync log now reports plastics merged.
• PBS Blend merged into PBS-Blend (curator ruling, 2026-09-27) by the same mechanism: one anchor spelt twice. The space-spelt row 53 was empty in production; row 54 carries the data, so nothing moves.
• A merged-away name still resolves on read-back. resolve_plastic and the composition lookup read a merged name as its survivor (merged_plastic_name), so re-promoting ingestion 202 lands on PBS instead of minting a needs_review PTS row. The live vocabulary no longer serves PTS, so a producer validating against it is refused (new REJECTED_TERMS entry) rather than aliased.
• PTC backfilled (full_name, polymer_class; NULL-fill only, the row carries no records): poly(tetramethylene carbonate), i.e. poly(butylene carbonate). v1 attaches it to Roseateles depolymerans 61A, whose description (Suyama, Hosoya & Tokiwa 1998, 10.1111/j.1574-6968.1998.tb12956.x) names that polymer. Not renamed to PTMC, which is the C3 homologue.
• PHC backfilled on the same evidence and with the same NULL-fill: poly(hexamethylene carbonate), polycarbonate (v1 "PHC (PCD-2000)"). Its only production records (ingestion 202) use it in exactly that sense. It is now the written form of a poly(hexamethylene carbonate) soft block, so no compound twin (PHMC) is added.
• Comments that guessed at the legacy PHC, PTS and PTC rows are corrected from the v1 data: PHC is poly(hexamethylene carbonate) ("PHC (PCD-2000)"), PTS is poly(tetramethylene succinate), i.e. PBS ("PTS (BIONOLLE-1020)"). None is a PHA or a polythioester.

Not in this release

• PEA backfill to poly(ethylene adipate): held until V2 records mis-filed under PEA are corrected. PEA must never be used for poly(ethyl acrylate) (new REJECTED_TERMS entry).
• PCL-b-PBLG has no row. It is filed on PCL Copolymer.
• BI-OPL: stays refused.

Fixed

• A task still running after an hour is no longer run a second time. task_acks_late=True acknowledges a task only when it finishes. Nothing set broker_transport_options.visibility_timeout, so the Redis transport's default of 3600 s applied, while task_time_limit allows 4 h 15 min. Any task past the hour was redelivered while still running and executed again as soon as the first copy finished. Measured in production on 2026-09-23: release 2026.39.9 ran 3,672 s, and its message (task d5ed60ea) came back as 2026.39.10 at 13:22:05, the same second 2026.39.9 finished. That run passed the hour too, and 2026.39.11 followed. One message was running as a self-perpetuating chain of forced releases, stopped by hand with a revoke. The same applied to analysis jobs on the users' queue: any job over an hour ran twice. BROKER_VISIBILITY_TIMEOUT is now 5 h, which exceeds the hard limit. Three tests pin it. Removing the setting fails one; reverting to 3600 fails two.

Fixed

• **Pins plasticdb-ml v0.5.3: calibration assigns each family's own Pfam against the complete catalytic panel.** Release 2026.39.9 was the first to calibrate end to end in production, and the library it built was wrong. Five of six families were NOT SEPARABLE, because notebook 04 assigned own_pfam against Phase 1's 11-model panel, which has no PETase. The release did not publish it: RELEASE_PUBLISH_LIBRARY_ENABLED was off for exactly this check. v0.5.3 assigns against all 29 models and reproduces the recorded assignment, 391 of 391 seeds.

Fixed

• **Pins plasticdb-ml v0.5.2: notebook 04 reads its calibration inputs from the bundle.** Release 2026.39.8 was the first to run all four notebooks to completion. It covered 751 live records, built 143 profile HMMs in notebook 03, and executed notebook 04. Notebook 04 then declined to calibrate: `staged inputs incomplete: (no directory staged)`. The bundle held every calibration input (optional_parts.calibration_inputs: true, missing_parts empty). PlasticDB-ML #19, which teaches notebook 04 to read them from there, had merged into its stacked base branch rather than main, so v0.5.0 and v0.5.1 shipped without it. v0.5.2 carries it.

Fixed

• **Pins plasticdb-ml v0.5.1: notebook 03 no longer assumes exactly four model heads.** Release 2026.39.6 cut its bundle from the live database, rebuilt the identity matrix, ran notebooks 01 and 02, and executed notebook 03 with the HMM profile block built for the first time in production (143 profile HMMs). Only one cell failed: KeyError: 'PBS'. The live corpus now has PBS as a fifth head, and the notebook's colour palette listed four. The frozen week-38 corpus had hidden this. v0.5.1 derives the palette from the heads the data produces and keeps the existing four colours.

Fixed

• Notebook 03 can find the build tools in the release pod. Release 2026.39.4 got further than any release before it. It cut its bundle from the live database, rebuilt the identity matrix over all 751 records (1,033 ids, about 6 minutes on 4 workers), and executed notebooks 01 and 02. Notebook 03 then refused with `BlockUnavailable: feature block 'hmm' was requested and could not be computed: missing tool(s): mafft=False hmmbuild=False hmmsearch=False`. That was on a pod where all three are installed and pinned, and where notebook 02 had just used them. PlasticDB-ML's rescore.probe_available() decides "am I in PlasticDBV2?" by importing app.services.bio_tools. The notebook child runs in its scratch directory with no PYTHONPATH, so app was never importable and the answer was always no. Two of this week's fixes each worked as written and contradicted each other. PlasticDB-ML v0.5.0 made the notebook probe for the tools instead of assuming they were absent, and made a wanted block that cannot be built stop the run. V2 already knew the child could not import its code (that is why stage_fetch cannot run in a notebook). The strict flag was right to refuse, because the premise it was handed was wrong. _child_env now puts the repository root on the child's PYTHONPATH. Measured on the pod in that exact environment, with no credentials: the probe imports, and _tools() resolves and verifies mafft, hmmbuild, hmmsearch and diamond. This adds importable code only. No credential moves, and ENV_WITHHELD still refuses one.

Three new tests. One checks the path points at a tree holding app/services/bio_tools.py. One runs a real interpreter with exactly the child's environment, from outside the repository, and imports the module. One checks the child still carries no credential. Removing the line fails two of them.

Fixed

• **The release's bundle export runs as a subprocess, because Celery forbids the process pool it needs.** The first live release, 2026.39.3, cut its bundle from the live database. That worked: 751 records, against the 597 in the frozen week-38 bundle, so 154 proteins curated since then reached the pipeline for the first time. The positional rebuild that should have covered them then failed with `AssertionError: daemonic processes are not allowed to have children. export_bundle called export_ml_bundle.main(argv)` in-process. Every Celery task runs in a prefork child, which is daemonic, and the rebuild (--identity-workers) is a multiprocessing pool. assert_bundle_covers_corpus did its job and refused the run 18 seconds in, before any alignment, naming the 154 ids and the recorded reason. No test could have caught it. Every test called the exporter from pytest or the CLI, and neither is daemonic. export_bundle now runs python -m scripts.export_ml_bundle through run_exporter. A process started with exec is a new interpreter, not a multiprocessing child, so its pool is allowed. The child inherits the worker's stdout and stderr, so the exporter's own messages still land in the release log.
• The database URL no longer travels in argv. In a subprocess, argv is readable from /proc/<pid>/cmdline, and the URL carries the password. It now goes into the child's environment as SYNC_DATABASE_URL, which is the exporter's own default.

Three new tests. One pins the fresh interpreter, the environment-borne URL and the repository-root cwd. One runs the real -m scripts.export_ml_bundle --help as a subprocess. One asserts the URL never appears in argv. Reverting to the in-process call fails three tests.

Carries #506 and #508 onto main. Both merged into their stacked base branches rather than main: gh pr edit --base failed silently on this repository (a Projects-classic GraphQL deprecation error) so neither retarget took, and v1.23.4 was tagged without them. No rework — the same commits.

Added

• The bundle carries the frozen Phase 3 inputs, so a release can calibrate. A weekly release runs PlasticDB-ML's notebooks 01–04, and notebook 04 is the only thing in either repository that produces hmm_library.PUBLISHED_ARTEFACTS — PlasticDB.calibrated.hmm and polymer_policy.json, the two files the deployed sequence search reads. It could not run in the release pod, because a calibration needs three inputs that are in none of the places a release already has: the negative-tier cache, a 35-model superfamily.hmm, and Phase 1's negatives.faa. Somebody had to copy them into the pod by hand and point $PLASTICDB_ML_CALIBRATION_INPUTS at them — the last manual step in a release that has no others, and one nobody can perform from outside the pod. So the release rebuilt the profile library every week while the deployed screen went on searching a library published by hand. Re-fetching at release time is not the alternative it looks like. calibrate.stage_fetch does `from scripts.export_seed_fasta import concat_hmms, ensure_pfam, and release_notebooks._child_env` builds the notebook environment from an allow-list with no PYTHONPATH, so the import raises in the child. It is also independently undesirable: *"a build that re-downloaded its negatives would silently recalibrate every threshold the day UniProt shipped a release."* They ship under calibration-inputs/, laid out as the relative paths plasticdb_ml.hmm.calibrate.CALIBRATION_INPUT_PARTS declares, so the subtree is a staged inputs directory: the ML side hands it to stage_inputs untranslated and $PLASTICDB_ML_CALIBRATION_INPUTS survives as an override of the same shape. No new flag — the cache, the panel and cache_manifest.json come from --phase3-dir (where stage_fetch writes them) and negatives.faa / nr90_cluster.tsv from --phase1-dir, both of which release_notebooks.cached_input_dirs already passes, so the weekly --from-database export picks them up with no release-stage change. The operator's one-off is a single calibrate … fetch into $ML_BUNDLE_WORK_DIR/phase3; docs/dataset-releases.md has it. Measured, not estimated. The real cache (SWISSPROT_POOL 20,000, the eight N2_ORGANISMS, a 6,069-record N3 pool) and the real 35-model panel, then a re-export of 2026.38 from the same sources: **24,523,428 bytes raw, 11,638,895 compressed, taking the bundle from 6,193,919 to 17,836,131 bytes** (corpus digest 72b0e37f… either side, so nothing but the new part moved). Nearly three times the artefact, and the module docstring carries the per-file table rather than burying it. Neither half is a lever: cache/swissprot.faa is 8.0 MB of the 11.6 and stage_negatives draws N1 and half of N0's 200,000 shuffles from that pool, so a smaller one moves the tier digests and with them calibration_fingerprint; and superfamily.hmm duplicates 29 of the 35 models already in pfam/, but shipping only the six absent accessions saves 0.7 MB of 11.6 and buys a second place where *the superfamily panel* is constructed, when the point of freezing it is that "same superfamily, different family" has to be a statement about the same set twice. Optional, on the line --phase3-dir and --phase2-dir are on: a bundle without them cannot calibrate and can still do everything notebooks 01–03 do. Requiring them would refuse every export that worked yesterday. The absence is a named missing_parts["calibration-inputs/"] entry with its consequence and the command that closes it, and a partial set is reported part by part — an operator who ran fetch into the wrong work directory has seven files and no eighth, and "absent" would send them to repeat the whole UniProt pass. test_the_exporter_looks_for_every_part_a_calibration_requires is the cross-repo ratchet: a required part added to CALIBRATION_INPUT_PARTS that this exporter did not learn about would keep producing bundles marked complete and refuse in the pod. It skips against a pinned plasticdb-ml that predates that declaration — requirements.txt installs a tag, deliberately, and CI proved the point by failing here first. That is not a hole: a required part can only be added by a version that has the constant, so the skip fires only where there is nothing yet to be out of step with. Verified end to end: a bundle re-exported from 2026.38's sources ran PlasticDB-ML's whole negatives → fit → tiers → report → publish chain offline in 194 s and produced both published artefacts, reproducing five of the six recorded profiles' GA/TC/NC exactly.

Fixed

• **The weekly release would have failed in notebook 03, fifteen minutes in, the first week it ran on a corpus that had moved.** 1.23.0 gave the release a fresh corpus every week and left identity.npz and features.json where they were: cached step-2 artefacts keyed positionally by protein id. Notebook 03 selects its identity rows with row_of[rec.id] — a raw dict lookup, not a join — so the first record the cached matrix does not cover is a KeyError after notebooks 01 and 02 have already succeeded. Measured on the shipped bundle-2026.38: rescore.build_records resolves 597 records, the matrix holds 879 ids, 0 uncovered — because the matrix was built over that same step-2 run. Any protein curated after it was uncovered by construction. 1.23.0 recorded the shortfall in the manifest; recording is not preventing. The features half was quieter and worse: notebook 03 reads it with feature_table.get(str(rec.id), {}), so an uncovered record reached the model as a row of zeros and nothing said so. That is the failure mode PlasticDB-ML's handover names — *"a join that degrades is worse than one that fails"* — and the one V2 #499 measured, where a patched-out key-space check wrote a complete JSON, exited 0, warned about nothing, and every number in it was zero. Tolerating the shortfall was therefore rejected: it would have made the loud half quiet to match the quiet half. Two changes, one closing the gap and one making it impossible to ship through:
  • **--refresh-positional (default auto) rebuilds both artefacts over the corpus the bundle actually carries**, using the same functions the step-2 driver runs — rescore.identity_matrix_parallel, falsify.extend_identity over the Matrix-B variant rows, and features.compose_features. Measured on the 2026.38 record set (597 records + 282 variant rows = 879 ids, 346,260 pairwise alignments): **250 s and 328 s** across two runs on twelve workers, of which the feature block is 12-13 s. It needs hmmsearch and the cached Pfam panel and **no other binary** — no MAFFT, no MMseqs2, no DIAMOND, no network. The per-pair cost is flat (7,140 pairs: 26.37 s on one worker, 8.59 on four, 5.41 on twelve), so a single-core pod pays about 21 minutes and a four-core pod about 7. Against the notebooks stage's existing ~15-21 minutes that is between a quarter more and twice as long, and it is the number to plan a release on; ML_BUNDLE_IDENTITY_WORKERS pins the worker count, because os.cpu_count() reads the host's cores inside a container and not the cgroup's quota. auto rather than always because the rebuild is exactly reproducible: re-run against the corpus the cached artefacts were built from, ids, pid and cov all come back array_equal with the shipped identity.npz and all 879 feature rows match at max |Δ| = 0.0. Skipping a rebuild that would change nothing therefore cannot change what a release measures. A rebuild that cannot run — no hmmsearch, an incomplete Pfam panel, no matrix-b/ — refuses rather than degrading (compose_features answers a missing hmmsearch by marking every sequence in-family and zeroing every Pfam column) and the export still writes, with the reason in the manifest's new positional_refresh block and the shortfall still named in missing_parts.
  • **release_notebooks.assert_bundle_covers_corpus refuses before the first cell.** Measured on bundle-2026.38: 0.36 s cold, 0.17 s warm, over the three members it reads out of the tarball — and it reports that bundle's 597 records against 879 matrix ids with 0 uncovered, reproducing the figure the defect was diagnosed from. Run straight after the bundle is resolved and before any notebook is executed. It measures rather than reads a manifest field, because bundle-2026.38 — the kind of bundle ML_BUNDLE_PATH pins on purpose — was cut before the exporter computed one, and a gate that trusted the field would have to decide whether to refuse on a question it could not read. Here alone an unanswerable check is fatal, and the asymmetry is deliberate: an unreadable identity.npz is not a bundle notebook 03 can run on either. The coverage it measured is recorded on the release row and in RUN.json, so the population the model ran on is named and counted rather than logged.
• POSITIONAL_COVERAGE_RECIPE named the wrong module. It said python -m plasticdb_ml.ml.features fetch/features, which writes a PET-only corpus and an identity matrix over proteins alone. The producer of work/step2-* is plasticdb_ml.ml.falsify, whose matrix carries the 282 Matrix-B variant rows as negated ids — 282 of the 879. An operator following the printed recipe would have rebuilt a narrower matrix that verify_identity_ids then refuses.

Known and named, not fixed

• mature.faa is still the cached run's, so a refreshed bundle's search FASTA names a subset of its own matrix's rows: a protein curated this week is in the model's rows and is not in the HMM screen's search set. It is left alone on purpose — nothing indexes it by row, and plasticdb_ml.ids.record_seq_id now spells its names protein:<id> where the cached run wrote p<id>, which is the spelling notebook 02 joins its hits back on, so a rebuild would be an identifier change wearing a freshness fix's clothes. The manifest's populations block no longer calls it "the sequences identity.npz is indexed over" when that has stopped being true.
• t4.json and t5.json are still the cached run's results, including its statement about its own corpus — so a bundle whose corpus has grown carries a t4 corpus block counting the population t4 was run on. Notebook 03 displays them as results rather than joining them to records by id, so this is a stale number on a page and not a wrong join. Named in positional_refresh for the same reason everything else here is: an absence a manifest names is a gap, and an absence nothing records is a wrong number with a provenance line on it.

Added

• Five plastics terms, each blocking a paper that is already extracted and cannot be delivered. 11 interactions and 70 observations across 3 papers are unblocked. All five came out of one overnight extraction batch, and in every case the extractor had already checked the near neighbours and refused to substitute — the reasoning is quoted in reviewer_guidance.data_gaps on each extraction.
• PPDL — poly(omega-pentadecalactone), the C15 macrolactone polyester. 10.3390/polym17192676 ranks it head to head against PLA, PLLA and PCL in one substrate-preference series, so folding it into any of them destroys the comparison that is the paper's result. The nearest-looking terms are different polymers: PPL is poly(beta-propiolactone) (C3), PMCL is poly(4-methylcaprolactone) (C7, branched), PES is poly(ethylene succinate), not a lactone at all.
• PE-2,18 and PE-12,12 — long-chain aliphatic polyesters, named in their own literature for the carbon counts of the diol and the diacid. 10.3389/fmicb.2025.1671731 assays one cutinase against both alongside PCL, so the three are a single comparison. The PE- prefix is not polyethylene: this vocabulary already spends PE on two senses (PE/PE Blend are polyethylene, PEA/PES/PET/PEF are poly(ethylene …) esters) and in PE-12,12 it is neither — the 12 is the diol, so PE there means polyester. The published names are kept because they are what both the literature and the extraction use; the full_name is what removes the ambiguity.
• PHB-b-PCL and PHB-b-PLA — synthesised diblock copolymers, not blends. The blocks are covalently joined in one chain, so no homopolymer was ever tested and filing either under PHB, PCL or PLA would assert an experiment 10.3390/polym17111558 did not do. Its PHB depolymerase shaves the PHB block away while the PCL crystallite edges splinter — a result that exists only because the two are one molecule. Both unparented, on the PLGA/CAP/SBR/PVAc rule: a diblock of two families has no single parent, and hanging them off PHB would pull them into every PHB query as though a PHB homopolymer had been assayed.

Not added

• BI-OPL was requested by a fourth paper (10.1371/journal.pone.0300503) and is refused, as it already was in REJECTED_TERMS: a commercial material earns a row only when its composition is proprietary and unstated. It is describable as PBAT-Blend plus substrate_composition. That paper needs its extraction corrected, not a vocabulary row.
• PE-18,18 appears in the same paper as the other two LCAPs but carries no interaction. This file adds a term when a paper proves the vocabulary could not express it, never in anticipation.
• PDLLA was suggested by the extractor and stays refused on the existing ruling: racemic poly(DL-lactide) is the default sense of PLA.

Added

• **scripts/experiments/reversal_censoring.py — how much of the 22.0% pair-reversal rate is a censoring artefact.** docs/phase5-censored-refit.md measures the ceiling on any condition-free within-study ranking head: 4,417 of 20,088 informative pairs reverse between two cells of one paper, so no condition-free score can get both right and the ceiling is about 0.78. The obvious follow-up had never been asked — a published 0 usually means "below this laboratory's detection limit", so a reversal could be two enzymes swapping places purely because one fell under the floor in one cell. It is not. Measured against the step-11 pull: **3,167 of the 4,417 flips (71.7%) are two measured magnitudes disagreeing**, and 1,250 (28.3%) need a censored comparison to exist at all. Restricting to comparisons of two real numbers makes the rate worse — 3,167 of 8,462 informative pairs is 37.4% — because censored comparisons contribute far more consistent pairs than flipped ones. The 0.78 ceiling is not an artefact of censoring. Also worth naming, because the question invites the conflation: a reversal is a within-paper event, one laboratory's two assay cells, so between-laboratory detection-limit variation cannot explain one by construction. The two are separate defects. The module decomposes rather than re-derives: the pairs come from censored_model.t3_pairs on a design built by the same functions from the same pull, and it reproduces 24,097 / 20,088 / 4,417 before splitting anything. The split is a restriction, so it can only remove flips and never create one — asserted at runtime, and pinned by a synthetic fixture built to make a flip out of censoring and nothing else. Recorded there and not fixed: t3_pairs reads a right-censored row ("at least the ceiling") as lower than any observed value, so 857 of 196,288 cell-level comparisons are scored against an inverted label. Correcting it would move a published figure, so it is the owner's call. And the Erickson powder-ceiling defect the pull still carries (83 observations, 20 design rows wrongly right-censored) moves the flip count by 2 — it matters for a magnitude fit, not for ranking.

Fixed

• A reading off the top of the scale is a detection. make_design built gate 1's binary target as row_detected = 1 if r.kind == "obs" else 0. row_kind is three-valued — 0 observed, 1 left-censored (*at most the detection floor*), 2 right-censored (at least the quantitation ceiling) — so this is the same two-way test applied to a three-way code that t3_pairs was carrying one layer up (C3), and it fails in the same direction: **a right-censored row, the strongest evidence of activity an instrument can return, went into a detection model as evidence of no activity.** | row_kind | published | corrected | |---|---:|---:| | 0 observed — a value inside the calibrated window | 1 | 1 | | 1 left — at most the detection floor | 0 | 0 | | 2 right — at least the quantitation ceiling | 0 | 1 | The left branch was measured before it was left alone, because trading one wrong sign for another is the obvious way to get this wrong: 2,944 of the 3,013 left-censored rows are a literal value <= 0 and the other 69 are a positive number below the paper's own declared detection limit, which that paper is declaring indistinguishable from zero. It also matches the probability the conditional arm actually scores, Phi((f − theta − floor) / sigma) = P(the true value is above the floor). Population on the published step-11 corpus: **95 of 6,253 rows, 1.52% of the target vector**, prevalence 0.5030 → 0.5182, all 95 from one paper of 79 (Erickson) across 49 of 902 cells. **Zero of the unit-clean variant's 510 rows**, which has no right-censored row at all — so not one unit-clean number moves, structurally. This was a re-measurement, not a sign flip. fit_censored_model never reads row_detected, so one set of fits scores both rules and the two arms differ by the label rule alone. Hyperparameter selection was re-run under both rules (360 nested refits) because its inner cross-validation selects on a likelihood over these labels — the choice is identical in all ten outer folds — and so were all three permutation nulls (600 draws, 3,000 further fits). Every published-rule figure reproduces the published artefact to three decimals. | | published | corrected | |---|---:|---:| | detection AUROC, paper folds | 0.603 ± 0.139 | 0.609 ± 0.141 | | detection AUROC, 60% cluster folds | 0.610 ± 0.052 | 0.620 ± 0.065 | | conditional AUROC (severity known) | 0.750 ± 0.051 | 0.754 ± 0.056 | | detection AUROC vs its cluster-level null, paper folds | 0.598 vs 0.524 ± 0.052, p = 0.080 | 0.604 vs 0.524 ± 0.053, p = 0.080 | | conditional AUROC vs its null, 60% cluster folds | 0.756 vs 0.735 ± 0.012, p = 0.033 | 0.760 vs 0.736 ± 0.012, p = 0.020 | **Ranking improves everywhere because the 95 mislabelled rows are the ones the sequence head already scored highest.** On fold 0's held-out scores the mean score percentile is 0.418 for left-censored rows, 0.548 for observed and 0.754 for right-censored: the model was being marked wrong for ranking them at the top. Only fold 0 owns them, so only fold 0's AUROC moves (0.608 → 0.640, folds 1-4 identical in the fourth decimal).

Changed — one published verdict is withdrawn

• **"Significantly better calibrated than a condition-blind logistic" does not survive the correction.** docs/phase5-censored-refit.md published that contrast at Δ = +0.020 nats, 95% CI [+0.003, +0.035], p = 0.0015. It is now +0.023, [−0.002, +0.041], p = 0.13. The point estimate moves the wrong way for the obvious story — the advantage grows and the interval widens, because each per-fold difference moves away from zero in whichever direction it already pointed, and a cluster bootstrap over 79 papers holding 3.5 effective studies cannot separate a larger mean from a larger spread. The exact sign-flip permutation over folds was p = 0.50 before and after, and it was in the published artefact all along. "Better than a constant" survives (p = 0.010, and p = 0.042 on ΔAUROC) and *"indistinguishable from the binary model"* survives. docs/corrections.md#c4 is the full account. C3's two owed T3 p-values are paid by the same re-run — T3's everything else at 0.581 reads p = 0.025 against a cluster-level null and p = 0.040 against a sequence-level one, and the 60%-cluster-fold row's observed moves 0.627 → 0.651 at p = 0.0066. **Under the published label rule the re-run reproduces all three published nulls to four decimals**, and `T3, dominant screen` does not move on any of them — exactly as C3 said it could not. Documents corrected in place, each with a dated pointer to C4: docs/phase5-censored-refit.md (gate 1's two summary tables, the per-fold table, both paired-arm tables, the prospective split and all three null tables), docs/discrimination-programme-results.md (the conditional arm against its null, and the T3 p-values), docs/corrections.md C1 (the 2026-09-07 refit's raw detection rates — C1's finding is unaffected, the film still detects far less often than the powder) and C2 (the conditional arm and its null — C2's finding is unaffected and slightly strengthened). Four new tests in tests/test_censored_model.py, none torch-gated, so all four run in the CI image: the three-way rule as a unit, a right-censored row detected through the real build_rows → make_design path, a left-censored row not detected through both routes into left, and the whole row_detected vector against the whole row_kind vector — which catches the call site rather than the rule. Each verified to fail against a mutation of the thing it tests: the defect restored, a mirror-image inversion, the call site reverted to the inline expression, and "everything is a detection".

Fixed

• An off-the-scale reading is not the smallest possible value. scripts/experiments/censored_model.py's t3_pairs decided which of two enzymes measured higher inside an assay cell, and it tested row_kind != 0 — "censored in either direction". row_kind is three-valued: 0 observed, 1 left-censored (at most the detection floor), 2 right-censored (*at least the quantitation ceiling*). Those are opposite facts, and the test treated both as lower than any observed value, **so the strongest result an instrument can return was ranked as the weakest**. Measured on the published step-11 corpus: **857 of 196,288 cell-level comparisons carried an inverted label** — every observed-versus-right-censored comparison in the corpus, no exceptions. The published pair-reversal count moves from 4,417 to 4,325 of 20,088 informative pairs, 22.0% to 21.5%, every one of the 92 removed reversals from outside the dominant screen (669 → 577; the screen's 3,748 does not move, because all 95 right-censored rows are Erickson's and the screen is Norton-Baker's). **The ceiling those numbers exist to state does not move at the precision it is quoted to** — 1 − 0.2199 = 0.780 against 1 − 0.2153 = 0.785, both "about 0.78" — so C2's "ship neither head" and every conclusion resting on the ceiling stand unchanged. The T3 accuracies are averages over the labels this fixes, so they were re-measured on one fit rather than adjusted by arithmetic (the fit does not read t3_pairs, so a single fit scores both rules and only the sign rule differs between the two columns). Experiment 2 moves 0.593 → 0.621 at the cell grain and 0.568 → 0.581 at the pair grain. **Experiment 1 does not move at all** — every count and both accuracies identical to every digit, because all 95 right-censored rows are Erickson's and the dominant screen is Norton-Baker's. The re-run reproduces the published fit: same per-fold hyperparameters, and experiment 2's old pair-grain accuracy to every digit, 0.5677147930893753. One remark in phase5-censored-refit.md is withdrawn: *"the two experiments agree more closely than they did (0.598 against 0.593 at the cell grain)"*. They do not — 0.598 against 0.621. The convergence was the defect. docs/corrections.md#c3 is the full account, including what is still owed: the two T3 p-values, because each permuted draw is scored against the same corrected labels, so the null moves with the observed and 200 nested refits are needed to restate them. Two further consequences of the same != 0 test are fixed with it. **Left against right was discarded as unorderable** (1,231 comparisons) although left ≤ floor ≤ ceiling ≤ right orders it — now ordered, guarded on the declared window actually satisfying floor ≤ ceiling, because a cell that cannot say must still say nothing. And the regression test that was supposed to cover this, test_t3_calls_two_proteins_tied_at_the_floor_uninformative, asserted nothing: its fixture's design is row_kind == [0, 0, 0, 1], one censored row, so the loop body never executed — and its condition was written as the same != 0 conflation. It now counts what it checked and runs on a fixture with 109 left-censored rows. Why the rest of the module was right and this was not: censored_loglik takes separate left and right masks, and make_design drops a right-censored row with no declared ceiling to be censored at. t3_pairs was the only place that collapsed the two, and it was invisible because a censored row's row_y is the placeholder 0.0 — an inverted ordering and a correct one are the same array.

Known defects

• make_design labels a right-censored row "not detected" (row_detected = 1 if r.kind == "obs" else 0) — the same conflation in gate 1's binary target, 95 of 6,253 rows. Reported in docs/corrections.md#c3 and not fixed: it moves gate 1's detection AUROC and every permutation null attached to it, which is a re-measurement rather than a sign correction, and the two should not be entangled in one change.

Changed

• **The weekly release cuts its own ML corpus instead of re-reading a frozen one.** The notebooks stage resolved its bundle from ML_BUNDLE_PATH, and the production ConfigMap set that to a literal /app/bioinformatics/ml-bundles/bundle-2026.38.tar.gz. So every release after 2026.39.2 would have re-analysed the week-38 corpus: newly curated proteins reached the public dataset and the DIAMOND index and never reached the profiles or the model, and each run reported completed. The failure signal was absent by construction. With ML_BUNDLE_PATH unset — now the normal case — the release exports a bundle from the live database at its own release label, into ML_BUNDLE_DIR (default <bioinformatics_dir>/ml-bundles), from the cached phase runs in ML_BUNDLE_WORK_DIR. Set, the path still wins and is recorded and reported as pinned: reproducing a past release is a real need, and a release that re-analyses a month-old corpus should not read like one that did not. Idempotent by label, so the notebook stage stays resumable: a release that failed in notebook 03 re-executes it against the corpus notebook 01 read, not against one that moved in between.

Added

• scripts/export_ml_bundle.py --from-database, the mode that stage runs, implementing the .. todo:: that module has carried. It re-derives corpus.json (plasticdb_ml.ml.falsify.fetch_corpus) and label_matrix_corpus.json (scripts.export_label_matrix.fetch_corpus) into a scratch directory that shadows --step2-dir for those two entries. The assembly, the manifest and the checksums do not fork. SELECT-only on a read-only session; dry-run by default, and the dry run fetches, because one that did not could not report what would be written. It refreshes the two corpora and nothing else, which the manifest now states rather than leaving to be inferred: the seed families and the phase-1 partition are a Phase 1 re-run by hand, and identity.npz / features.json stay cached. Notebook 03 indexes the identity matrix with row_of[rec.id] — a dict lookup, not a join — so the exporter measures that coverage with notebook 03's own build_records and records it under positional_coverage, naming any shortfall in missing_parts with the command that closes it. Measured on bundle-2026.38: 597 records, 879 ids, 0 uncovered. No row filter. PLAN.md's Risks note put the public-bundle filter in this mode; that was superseded on 2026-09-19 — reviewers may see the whole dataset — and the docstring now records the ruling instead of carrying the todo.

Fixed

• The notebooks stage could never write its output in production. Every executed notebook reaches its destination through os.replace, and os.replace cannot cross a filesystem boundary. /tmp is the container's overlay and <snapshots_dir> is an NFS mount, so the default TemporaryDirectory location sat on the wrong side of a mount point and every capture raised OSError: [Errno 18] Invalid cross-device link. Release 2026.39 found it: promote and reference both succeeded — 37 proteins and 9 microorganisms promoted, DIAMOND rebuilt for 559 proteins and 628 variants — notebook 01 executed, and the result was then thrown away. The scratch directory now sits beside the destination, so both ends of the rename are on one filesystem. shutil.move was rejected deliberately: it would hide the symptom by falling back to copy-then-delete, and a copy is not atomic, so a worker killed mid-write would leave a truncated notebook where a release record should be. The suite could not have caught this — tmp_path puts both ends in the same directory, so the rename never crossed anything. The regression test asserts the property that made production differ, not the behaviour the fixture already guarantees.

The identifier release. A PlasticDB accession is now what names a protein in every artefact this repository exports — the DIAMOND reference FASTA, the seed/HMM stack, the structure manifests, the ML bundle — in place of the four competing formats an audit found in use and the internal integer id they all degraded to. The pipeline core that consumes those artefacts moved out to PlasticDB-ML, and the weekly release gained a stage that executes its notebooks, because the notebooks are producers now rather than a report about the pipeline.

Minor rather than patch, on the precedent 1.19.0 set when it explained itself. Three things in that entry's sense: scripts/backfill_accessions.py gains a required argument and refuses an invocation that worked before; seed ids change value everywhere, so library_fingerprint and corpus_fingerprint move and every published profile library is stale; and the release gains a stage that can stop a release. The additive half — 1.21.7's read-schema fields, the exporter's new parts — would have been a patch on its own, and that is the line this repository draws.

This release carries a migration. 0017_release_notebooks, the first since 0016_experiment_window, which production is still stamped at. The deploy record is docs/release-1.22.0.md, and the tag carries this section and 1.21.7 below it — that section was a VERSION bump that no tag ever shipped, exactly as 1.21.2 rode out on v1.21.3.

Changed

• A seed is named by its PlasticDB accession (protein:PB24UZ, or protein:PlasticDB_<id> for a row with no accession minted yet). This was the last identifier in the seed/HMM stack still keyed by the internal integer id, and it is the join key for seeds.faa, seeds.tsv, families.json, no_profile.tsv, the tail arm's subjects, the provenance sidecar — and the structural arm.
• library_fingerprint and corpus_fingerprint move with it, deliberately. They digest the seed id as the library is keyed by it, so **every profile library published before this is stale and will rebuild.** That is the point: every artefact in such a library spells its seeds the old way. A re-minted or backfilled accession also moves the digest now, which is correct for the same reason.
• The DIAMOND reference FASTA reads protein:PB6JCU, and three things had to move with it. reference_ids.parse_subject_id's isdigit() gate was silently routing an accession to the legacy branch, where it was looked up against plasticdb_id and matched nothing — issue #330's failure mode a second time, and widened deliberately. resolve_subject now takes accession / pk / fallback and follows a merged accession to its survivor, which a bare primary key cannot do: a merged-away pk is simply gone, while a merged accession still resolves and a tombstone answers 410. And the accession is in reference_build.current_fingerprint, so a re-mint triggers a rebuild rather than serving a stale database.
• One definition of the identifier policy, not two. app/services/export.py delegates protein_seq_id / variant_seq_id to plasticdb_ml.ids instead of holding a second copy, and reference_ids.parse_subject_id routes through the same module. New accession.is_accession is checked against [A-Z0-9] rather than the narrower minting ALPHABET, because it is a classifier and its two errors are not symmetric.
• The pipeline core is PlasticDB-ML's now — about 16,000 lines of labels, sequence frames, the HMM stack and its drivers, features and the rescorer, with fifteen re-export shims here so every existing import keeps working. The shims bind the module object (sys.modules[__name__] = _moved) rather than star-importing: four moved modules declare __all__ and six callers here import names outside it, six of them private, so a star import would have dropped them silently. The eight driver shims keep a __main__ dispatch ahead of the alias, because under python -m the module's own name is __main__. What stayed is every fetch_corpus and all database access. requirements.txt pins plasticdb-ml at v0.4.0, which is the first tag to ship the notebooks as package data.
• A backfill write must name its entity. scripts/backfill_accessions.py read entirely as being about Protein.accession — *"run this before applying the NOT NULL constraint"* — while TARGETS silently also mutated Microorganism, whose accession is nullable=True by design. Measured on production on 2026-09-20: a bare --apply would have written **197 rows nobody asked for**, caught by a human reading the dry run, which is the wrong place for that check to live. --apply now requires at least one --entity, enforced in main() before a session exists, so a refused run cannot reach a database at all. A dry run still covers every entity and keeps its exit code, so anything gating on it is untouched. Microorganism was not dropped from the targets — backfilling it is sometimes exactly what is wanted, and was here.

Added

• The weekly release executes PlasticDB-ML's notebooks. They are not a report about the pipeline any more — notebook 02 builds the profile library and notebook 03 calls the gates — so running them is how the week's artefacts come into existence, and the executed notebook is the release's record of how. A new notebooks stage sits between reference and snapshot: after the DIAMOND rebuild because it is the long, externally-dependent one and nothing user-visible should wait behind it; before the snapshot because the executed notebooks ship inside the tarball, under notebooks/ with a RUN.json naming the bundle, the pinned tool versions and the plasticdb-ml revision. Resumable within the stage as well as across it — a notebook with a clean executed copy for this version is not run twice.
• A dedicated Celery queue for the release (RELEASE_QUEUE, with its own release-worker). app/workers/hmm_build.py refuses to auto-run this chain because it "must not start by surprise on a worker that is also running users' jobs"; that objection is answered by separating the workers, not by dropping the inversion. Its comment is extended rather than deleted, recording what changed and which of its three conditions each mechanism meets.
• A toolchain gate that refuses a version that drifted. bio_tools.assert_tool_usable tolerates a mismatch on purpose — a developer's host is not the image — but a release cannot: two toolchains produce two different profiles. Every pinned binary the build chain needs is probed before the first cell, and notebook 02's BUILD PATH: FALLBACK path is now unreachable from a release.
• requirements-notebooks.txt, installed in the worker stage only, asking plasticdb-ml for its [notebooks] extra by name at the same pinned tag. The rule it does not break is that a git-URL install must not quietly acquire a second HMMER; the release path stays hmmbuild against the pinned binary, and the gate above is what makes that structural.
• scripts/stage_structures.py — the missing producer for the directory of structure files scripts/structural_tier.py and scripts/experiments/structural_features.py read. It stages the object store on disk named by accession and records the plasticdb_id → accession crosswalk in manifest.json. Object store keys are not renamed; Protein.structure_file points at them, and renaming them is a data migration. Previously that directory was staged by hand as protein_<internal id>.pdb — a convention nothing in the repository produced.
• Key-space guards on the structural tier's four inputs (plasticdb_ml.ml.joins.reconcile_key_space). Three of the routes stage_yield compares are a set intersection against one universe of seed ids, and an intersection cannot tell "outside the restricted universe" from "spelled in another key space" — the latter produced a complete run reporting zero for every arm, which is also a publishable Phase 5 finding.
• scripts/label_matrix_api.py and fetch --source api — a second acquisition path for the label-matrix corpus, over the REST API instead of PostgreSQL, because fetch is a SELECT and production is not reachable from a developer machine. The SQL path is the default and is unmodified; it is the one to use wherever a database is reachable, because it reads the columns and owes nobody an inference. Everything the API path had to substitute is recorded in the corpus under provenance, and both self-checks run before anything is written: the derivation re-scored with ground truth withheld (1,417 checkable, 1,417 agreed, a disagreement aborts the run) and a possibility constraint that fired 9 times. What survives is stated rather than smoothed over — 333 of 1,898 protein-actor rows can only be placed inside a group sharing one polymer and one verdict, and **195 of 1,317 variant-actor rows** are dropped and counted rather than guessed. 1.21.7's actor ids are what make that residue removable; this release does not change the consumer.
• Four recorded gaps closed in the ML bundle (scripts/export_ml_bundle.py). The Pfam panel shipped 16 models against 36 declared, which made it systematically permissive in a direction invisible from the output — a hit a missing model would have contested was reported uncontested, and the worst absentee was PF12740 (PETase), in a plastics panel. All 20 fetch from InterPro today; --pfam-dir points the panel at its own directory and the manifest always carries a census, with an incomplete panel naming its absentees and the direction of the error. The 35 profiles were established to be falsify.ProfileCache's per-head T4 arms rather than family profiles — 35 of 35 stems recompute from the shipped FASTAs — and profiles/index.json now records the verified stem rule. MANIFEST.json.populations gives each artefact its population and what its count came from; an uncountable artefact records uncounted and never fails the export. Phase 2's and Phase 3's artefacts exist on no machine here and were not invented — they are named in missing_parts with why, consequence and how to produce them. BUNDLE_FORMAT_VERSION stays 1: every change is additive.

Fixed

• Three places where an accession we already held was dropped. All three confirmed real by stashing the fix and watching the new tests fail. bioinformatics.py hard-coded a variant's accession to None — the variant branch of _resolve_search_hit_entities never selected the column, so the API reported that a variant had no accession when it had one. This one was user-visible. proteins.py built its FASTA header and filename by hand, a character-for-character duplicate of export.protein_fasta_header(). structural_tier.py defined seed_name() and never called it, so both published FASTAs carried the raw filename stem instead of the typed identifier — precisely what that helper exists to prevent.
• The settings middleware was rewriting settings mid-test (#492), and the issue's leading hypothesis was wrong: there is no second Settings instance, and id() is identical at the fixture, the mutation and the route. _refresh_settings_middleware calls maybe_refresh_async() on every HTTP request; throttled to one reload per 30 s, it opens a connection to whatever PostgreSQL is listening on localhost and _apply_locked then rewrites every editable field on the process-wide singleton in place — the same object the fixture patched. **So the suite's results were silently depending on the contents of a developer's dev database.** A new autouse _no_live_settings_refresh in tests/conftest.py patches the name bound in app.main at import time; the regression test forces both throttles open, so it fails every time the guard is removed rather than half the time. No retry, sleep, marker, skip or reorder. Honest caveat: this is proved for the local failures and not for the CI instance, where localhost:5432 is unreachable and the fail-open path cannot mutate anything.

Note for operators

Three things in this release need a decision before they do anything, and one happens whether or not you decide. All four are in docs/release-1.22.0.md with what to expect.

• Every published profile library is stale. A consequence, not an action. The hourly DIAMOND reference rebuild fixes itself; the HMM profile library does not rebuild on its own and reports its staleness hourly until an operator or a notebooks-enabled release builds it.
• The notebooks stage ships disabled (release_notebooks_enabled=False) and the release-worker in plasticdb-gitops ships disabled too. They are turned on together; a worker with nothing routed to it is just a pod, and the stage without a worker is a queue nobody drains.
• **ml_bundle_path is empty and export_ml_bundle.py has no --from-database mode.** A release runs against a bundle an operator produced and pointed it at. Enabling the stage without one fails the release at notebooks.
• **A release promotes visibility='staged' to public and cannot be undone from the outside. Measured on production 2026-09-21: 37 proteins and 9 microorganisms**, plus 1 variant hanging off a staged parent.

Added

• The actor ids on FootprintInteractionRead — protein_id, protein_variant_id, strain_occurrence_id and the derived microorganism_id, served by GET /papers/{id}/records and GET /papers/{id}/footprint. The schema published actor_type and no actor id, so it could say *"this paper produced a negative protein-actor interaction on PET under these conditions"* and not which protein — while interactions.protein_id is the column the whole evidence spine is keyed on (docs/label-matrix.md: a label lives on a (protein, polymer) pair). This is exposure of columns that were already there, not new data.
• is_tombstoned on FootprintProteinRead and on the /admin/protein-visibility/proteins enumeration. It is half of plasticdb_ml.labels.drop_unsequenced()'s predicate and was obtainable only by resolving every accession one at a time and reading 301/410/200 back off the redirect — one HTTP request per protein for a boolean already loaded in the row.
• What it is for. scripts/label_matrix_api.py (PR #490) rebuilds Matrix A's corpus over HTTP because production is not reachable from a developer machine, and had to infer both. The inference is scored on every run (1,417 checkable, 1,417 agreed) but **333 of 1,898 protein-actor rows can only be placed inside a group sharing one polymer and one verdict** — a 97-enzyme panel written up in one sentence — and 195 of 1,317 variant-actor rows are dropped outright. With the ids published, that path becomes a transcription. This release does not change that consumer; it only makes the change possible.

Visibility — deliberately unchanged

• No caller sees anything it could not see before. An id is not exempt from the "something published what it should not have" class (app/services/dataset_releases.py records seven): protein_id on an interaction whose protein is withheld would confirm the record exists and pair it with a paper, a plastic and a result — which for the negative-results embargo is the withheld asset itself. It cannot happen here, because the interaction carrying the id is dropped first: public_actor_interaction_filter already excludes any interaction whose protein, whose variant's parent, or whose strain occurrence's microorganism is non-public or tombstoned. The curator /records route is ungated by design and already published those actors by name and accession.
• No second gate was written. The ids are reported and never read as a permission; public_actor_interaction_filter stays the only clause deciding what a public caller sees, because a second code path re-deriving a visibility rule is the shape every one of those seven leaks had. The reasoning is on FootprintInteractionRead itself, where the next reader will meet it.
• is_tombstoned is constant false on the public /footprint route and that is not a defect: public_visibility_filter excludes tombstoned rows, so a tombstoned protein is never in the public list to report itself. The admin enumeration is superuser-gated and already unfiltered by visibility.
• Three tests in tests/test_protein_visibility_holds.py pin it — a held protein's id never appears on the public footprint, the curator view still names all of them, and a tombstoned protein reaches neither the public protein list nor a public interaction. Two more in tests/test_papers.py pin the three actor slots, the derived microorganism_id, and that the other slots stay null.

No migration: every field is an existing column or a join over one.

Added

• Three monomer compounds, closing the sentence 1.21.5 left open. That release added P(MDO-co-CEVE) and P(MDO-co-TEGVE) and noted in the same comment that "its three constituents are not in compounds either". They are now: 2-Methylene-1,3-dioxepane (MDO), 2-Chloroethyl vinyl ether (CEVE) and Tetra(ethylene glycol) methyl vinyl ether (TEGVE), into the NEW_COMPOUNDS literal that seed_materials_vocabulary() upserts idempotently. 10.1021/acs.biomac.8b01500 is the last paper in the review queue: its polymer names have resolved since 1.21.5, and all ten of its substrate_composition components — five experiments, two each — still failed validation on these three names.
• Mapping them onto an existing compound was refused, on the same ground the PCL substitution was. The MDO molar fraction is the paper's finding — degradation tracks F_MDO across both the C1–C3 and T1–T2 series — and composition is the whole reason those two copolymers are single rows rather than five. Nothing in the list is these monomers: Tetraethylene glycol is the diol, not the methyl/vinyl diether built from it, and the vocabulary held no cyclic ketene acetal and no vinyl ether of any kind. Dropping substrate_composition instead would delete the variable the paper is about.
• is_additive=False on all three: MDO ring-opens during radical polymerisation to insert the backbone ester the Candida antarctica lipase cleaves, and the two vinyl ethers contribute the carbon-carbon units. None is dispersed in the matrix.
• **Every formula is verified against the paper's own stoichiometry rather than asserted** — C6H10O2, C4H7ClO and C11H22O5 reproduce all five stated initial molar fractions (0.1/0.2/0.3 for CEVE, 0.2/0.25 for TEGVE) to four decimal places. That arithmetic also settles tetra(ethylene glycol) against tri- independently of the prose: C9H18O4 would give 0.235 and 0.291 where the paper says 0.20 and 0.25.
• One test in tests/test_materials_vocab.py pins the three rows, their is_additive, the verified formulas, the separateness of the `Tetraethylene glycol diol, and that none of them is a plastics` row.

Added

• Seven plastics terms, each one blocking a paper already extracted: PBCn, poly(trans-hydroxynopol dicarboxylate), P(MDO-co-CEVE), P(MDO-co-TEGVE), PBATGA, poly(trans-1,4-isoprene) (abbreviation TPI) and PET Blend. 43 interactions across 8 papers are unblocked by them (24 + 12 + 2 + 1 + 2 + 2, in that order). They go into the NEW_PLASTICS literal, which seed_materials_vocabulary() upserts idempotently — the reviewed, git-tracked path — and not through POST /api/v1/admin/plastics, which records no audit of who created what.
• Two of the seven are family rows, and that is the ruling worth reading. PBCn covers the six BASF polyesters of 10.1021/acs.est.6b06060 and 10.1021/acs.est.7b01330, and poly(trans-hydroxynopol dicarboxylate) the four terpene polyesters of 10.1002/cbic.202400456. Six and four rows were refused because the member names do not exist: the Zumstein papers define their series once ("PBCn with n = number of carbon atoms in diacid") and name no member, and the ChemBioChem codes HN_FDCA/HN_DMS/HN_DMA/HN_DMSc name the dimethyl-ester reagent rather than the polymer. Minting poly(butylene suberate) for PBC8 would be our chemistry entered as the authors' — a confident term nobody can check against the paper. The member — the diacid carbon count, or the HN_ code — travels per record in specimen_description, the mcl-PHA and ionic-phthalic trade.
• Boundaries stated where a producer reads them rather than left to be rediscovered: PBCn's C6 member is PBA and stays there (and PBA is not re-parented onto the family row); PBATGA hangs off PBAT on the Apexa precedent, against the PBA one; poly(trans-1,4-isoprene) is a sibling of NR/IR and never a child, because collapsing the isomers puts one Lcp's cis result and its far slower trans result on one row. Five new REJECTED_TERMS entries carry the refusals, including why TPI is the abbreviation and not the name — ISO 1629 has no elastomer code for gutta-percha, so the siblings' naming convention cannot be followed here.
• Six tests in tests/test_materials_vocab.py pin the rulings, and PBAT joins the legacy set in test_declared_parents_exist so PBATGA's parent is checked at declaration time.

Documentation only. No runtime behaviour changes.

Added

• The 1.21.3 deploy record (docs/release-1.21.3.md): what shipped, the [SAFE] preflight, the deploy timeline, and every post-deploy check with its result. The screening-gate probe is the same one the 1.21.0 record used, run before and after, so the two releases are comparable line for line — the gate refuses a signed-in non-superuser with 403 and an anonymous caller with 401 on both job-creation paths, exactly as it did on 1.21.0.
• Recorded two things worth knowing next time. The PreSync hook is a no-op only in the second sense of the word: it applies no DDL, but it does re-run the reference-vocabulary upsert, which reports 0 added, 0 renamed once the rows exist. And a 422 from a malformed gate probe is indistinguishable at a glance from a 422 from an admitted caller — POST /api/v1/jobs is multipart and hmm-screen takes query parameters, so a JSON body never reaches the gate at all. The superuser control row exists to catch exactly that.

Housekeeping only. No runtime behaviour changes.

Fixed

• Lint is at zero. The fifteen repo-wide ruff errors that every agent has had to reason around ("pre-existing, in files I did not touch") are cleared. Thirteen were I001 import ordering, fixed with ruff check --fix scoped to the offending files; ruff format was deliberately not run across the tree, because about ten files on main are unformatted and formatting them would have buried the diff. Import order in a migration is not always cosmetic, so it was verified: all 16 migration modules import, and the migration suites are green against a real PostgreSQL 16 — which exercises env.py through an actual alembic upgrade head, the only way it can be exercised at all.
• Two errors that were not import ordering, both checked before being touched. app/api/v1/compounds.py had a dead order local — sorting goes through paginated_list, so the endpoint was never relying on it. The nested if in app/api/v1/ingestions.py is merged, with the comment explaining the inner test moved above the merged statement rather than stranded inside it.
• The local mock tier and CI now agree on the source tree. Twenty-seven stale agent worktrees were checked out inside the working directory under .claude/worktrees/, and tests/test_secret_redaction.py and tests/test_deploy_contract.py walk the tree and read every copy — so a clean branch failed locally and passed in CI, for 9.1 GB of reasons that had nothing to do with the code under test. All twenty-seven were clean, held by no process, and every branch preserved (25 removed with git worktree remove; two whose root-owned __pycache__ blocked deletion were moved out of the repository intact rather than half-deleted). Both tests now pass on the host.

Changed

• docs/discrimination-programme-results.md is current again. It described 2026-09-05, when "nothing in this programme wrote to the database"; production is now at 1.21.0 with the panel data ingested and the HMM screening tool gated. All six of its "open decisions" are settled and marked as such, the one outstanding write is named, and a known issues section records five things found and deliberately not acted on — the most consequential being that Hu & Howe's PET-nanoparticle assays are filed substrate_form = 'model_substrate', so the modelling is discarding genuine PET measurements as surrogate readings. A new where the evidence lives section records the archive path for the programme's working artefacts, which live under the git-ignored work/.

Fixed

• **Erickson 2022's 89 powder experiments no longer declare a quantitation ceiling they never had.** scripts/backfill_detection_limits applied a blanket 500 mg/L ceiling to all 202 of that paper's pre-existing rows, but the paper declares the ceiling for the film assay only — Supplementary Table D6 says "quantitative product release values are reported for amorphous PET powder and crystalline PET powder even above 500 mg/L" — and **30 of the 89 carried a stored value above the ceiling declared on them, to 10,403.99 mg/L**. Left in place it would right-censor real measurements out of every fit that reads the window. The 89 now match the 1,599 powder rows the panel insert already stores without a ceiling. The 93 film rows keep theirs, and the 20 rows of unrecorded morphology are left alone rather than guessed at.

Added

• scripts/experiments/provenance_and_conflicts.py — derive, census, plan, apply, restore, verify for the above, dry-run by default, with the rollback captured before the write and round-tripped on live rows. The apply and the restore share one body run with before/after swapped, so the undo is literally the write backwards.
• docs/phase5-provenance-and-conflicts.md — the adjudication of the fourteen sequence-provenance reviews, the three self-contradicting interactions, and the panel ingest's three loose ends.

Changed

• **The fourteen sequence-provenance reviews are adjudicated and change no label.** In all fourteen the stored chain is byte-identical to the NCBI record for the accession the paper states, and the supplement's chain translates exactly from the gene the paper synthesised — so the disagreement is the paper's own accession against its own construct, not a corpus ingestion error. Measured on an alignment rather than on k-mers the fourteen run **0.901–1.000 identical**, not the 0.33–0.86 previously recorded; ESM136 is not a discrepancy at all, making it thirteen. Every stored verdict already agrees with the workbook's own measurement, so none of them is a verdict question. No chain was rewritten and no row was split: both remedies need something outside the authorisation (a sequence_provenance vocabulary term, or a curator's identity decision on seven embargoed rows).
• **The three interactions "whose verdict contradicts their own observations" are correct as stored, and were not flipped.** PET46 releases 1,624 µM aromatics from semi-crystalline PET powder; TrmA and TrmH release TPA from PET nanoparticles at rates their paper reports as significant. The recorded premise that "the pNP-ester surrogate set the verdict" is falsified by the paper's own within-panel control: AroC retains 124 % pNPB activity and is stored negative, while its nanoparticle rate is not significant. All three are genuine conflicting condition cells, which app/services/label_matrix.py says to keep.
• **Enzyme 713's 30 deferred panel rows are resolved to the 168-residue interaction** on the construct geometry — the assayed 129-mer is M + 120 + LEHHHHHH, and that 120-mer is the 168-aa chain minus a 48-residue N-terminal leader, C-terminally flush, while in Thh_Est it is an internal fragment with 141 residues left over. The insert itself is a separate authorisation.

Added

• The record of the 1.21.0 release (docs/phase5-gate-release.md): the measured before/after on both job-creation paths, the merge order, the CI results, the deploy timeline, and the rollback procedure. Two things are worth reading even if the rest is routine — the exposure was **measured on the live 1.19.0 deployment** rather than inferred from the diff (a non-superuser was admitted to both paths), and the polymer-summary observation in §3 of the verification, which is not a regression but should be understood before hmm_screen_public is opened to the public.

Added

• The HMM profile library is built, calibrated and published. /tools/hmm-screen and the whole Phase 4 cascade shipped in 1.19.0, but <bioinformatics_dir>/PlasticDB.hmm had never existed in a deployment, so every screen failed at the hmmsearch call with an honest message saying nothing had been searched (#341). Phases 1–3 were run against the live corpus in the production worker pod and the calibrated library installed on the shared volume by the same code path the hourly check uses. The build record — family count, profiles built, families that ship no profile and why, and the pinned calibration source — is in docs/phase5-profile-library.md.
  • The calibration negatives are the pinned 2026-08-30 UniProt pull, reused verbatim rather than re-fetched. An unpinned pull silently recalibrates every threshold the day UniProt ships a release.
  • Nothing new is served. PUBLISHED_ARTEFACTS is still exactly the calibrated library and the polymer policy; the per-protein provenance sidecar, the only build artefact that names accessions and papers, stays on the build volume.
• An admin-controlled release switch for HMM screening (hmm_screen_public, Admin → Settings → "HMM profile screening"). **Default closed:** only superusers may create an hmm_screen job. Flipping it open admits every signed-in user. It is a row in the existing app_settings override table, so the change reaches every API worker and the Celery worker without a redeploy or a restart.
  • The switch exists because the profiles are derived from 291 records under an embargo that lifts when the owner's paper publishes. A profile is a statistical summary and names no record, but when it is safe to release them is his judgement, not a deployment constant.
  • Enforced at the API, not in the template. require_hmm_screen_access guards both job-creation paths — POST /api/v1/jobs with job_type=hmm_screen and POST /api/v1/bioinformatics/hmm-screen — before a byte of the upload is written. The page guard and the hidden sidebar link are cosmetics on top: a hidden page in front of a live public endpoint is not a gate, which this project has now learned twice.
  • Anonymous callers are refused in both states. Opening the switch widens the tool to signed-in users, never to the world: an ownerless job is readable by anyone holding its id.
  • tests/test_hmm_screen_gate.py pins all four states against both endpoints, plus the page, the sidebar, and the fact that the published set carries no provenance.

Fixed

• The published polymer vocabulary read the wrong shape, which the first real publish is what exposed. published_polymer_names() assumed polymer_policy.json was a flat {polymer: record} mapping; the build writes an envelope with a polymers list, so the reader returned the envelope's own keys — the tool page offered unit, rule and tiers as polymer options and the endpoint answered 422 to PET. The envelope is now read properly, the real artefact's shape is pinned in a test, and an unrecognised shape degrades to "vocabulary unknown" (falling back to the hardcoded list) rather than to a confident wrong answer.

Changed

• publish()'s atomicity claims are now asserted rather than documented: the fingerprint is proved to be written last, and an incomplete build directory is proved to leave no fingerprint behind to be trusted.

Added

• docs/corrections.md — a citable register of falsified claims. Two entries, each stating what was believed, what is now known, the evidence that overturned it, and which conclusions elsewhere are affected. Documents in docs/ now cite it by anchor (docs/corrections.md#c1) instead of repeating the correction, and docs/discrimination-programme-results.md links it from the top.
  • C1 — the accessibility axis had the sign backwards. The stringent negative rule counted a null on amorphous film as stronger evidence of a true negative, on the prior that amorphous chains are mobile so film must be the easy substrate. Film is the harder substrate. Matched-enzyme detection in two independent laboratories (0.171 vs 0.471 over 5,478 comparisons; 0.598 vs 0.804 over 826), a severity parameter fitted without ever being shown a condition column (film θ = +0.653, powder θ = −0.455), the confound isolated at matched particle size, and a refit on the enriched corpus (41,879 matched enzyme comparisons, 0.579 vs 0.755) all agree. Surface geometry dominates crystallinity in a whole-particle assay. Correcting the axis takes the stringent negative tier from 23 to 0, because all 23 qualified on the falsified clause and nothing else.
  • **C2 — "AUROC 0.759 when the assay cell is named" is mostly assay ordering.** A permuted-sequence null reaches 0.735 against the real model's 0.756. C2 explicitly does not apply to step 4's 0.740, measured inside a single fixed assay cell where between-cell severity cannot be the explanation, and which already carries its own cluster-level null (p = 0.025).

Changed

• **Every document that rested on the falsified accessibility axis now says so, in place.** Dated research records keep the original claim legible beside a marked correction; documents that state a current rule or instruction are corrected outright.
  • docs/phase0-pet-discrimination.md — a correction banner at the head, the falsified clause marked in the rule table, the tier census and the AUROC 0.864 / z = +3.49 stringent arm flagged as the output of a rule that no longer has a tier, and both wrong recommendations in "what would settle it" corrected: a second screen does not make the paper confound testable (it fails on cluster geometry), and the re-assay panel should not be run on film.
  • docs/discrimination-fix-plan.md — step 6's gate-2 worked example ("amorphous film at 70 °C should come out easy") corrected; step 7's instruction to file PET46 as *"a documented exception for one archaeal family"* withdrawn — it is the general case; the "91 % → 51 %, making Cpap constructible" claim superseded by the measured 91 % → 54 % on an empty tier; and §4's bare-AUROC success criteria flagged, since a pooled conditional arm reaches 0.735 without reading a residue.
  • docs/discrimination-diagnosis.md — H2's citation of the stringent arm and of Cpap, and the "more curation makes Cpap constructible" item.
  • docs/phase2-panel-extraction.md — worklist item B1 is worth 13 under the old rule and 0 under the corrected one, not 15.
  • docs/pet-negative-curation-worklist.md — the four remaining lines that still called PET46 a "documented exception", still graded amorphous film as accessible, or still prescribed a re-assay on film.
  • docs/label-matrix.md — cross-linked to C1 and updated with the 41,879-comparison confirmation.
• **The conditional-arm correction is applied with its scope stated, so it does not swallow a result it does not cover.** docs/phase3-censored-model.md's "worth keeping: the conditional head at 0.759" is withdrawn and its equation of 0.759 with step 4's 0.740 corrected; docs/phase5-censored-refit.md no longer describes the two as the same number reproduced by a third route; docs/phase2-benchmark-reproduction.md gains a section stating why its 0.740 is structurally out of scope (one assay cell, one severity, its own null already run); and docs/discrimination-programme-results.md withdraws "ship the condition-named head" in favour of shipping neither.

Fixed

• docs/phase5-censored-refit.md carried a stale upstream to-do saying the accessibility axis "still has the sign backwards" and that negative_stringency.py should be re-derived. Both were already done on 2026-09-05. Its pointer to the fix plan's success criteria said §6; they are in §4.
• **docs/discrimination-programme-results.md attributed the 0.740 benchmark to a 60 % identity holdout. It is measured at 70 %**; 60 % gives 0.694. Its within-study ranking figures (0.589 / 0.615, ceiling 0.79) are replaced by the refit's (0.572 / 0.568 against nulls, ceiling 0.78).

No code changed in this release. The accessibility axis was corrected in scripts/experiments/negative_stringency.py in 1.19.0 and is pinned by tests/test_negative_stringency.py; this release makes the written record agree with the shipped rule.

Added

• docs/phase5-censored-refit.md — the censored, condition-aware model refitted on the data step 5 wrote, and the three measurements step 6 recorded as not done: a cluster-level permutation null (600 refit draws), a paired test of one evaluation arm against another rather than two overlapping intervals, and a prospective split (train through 2024, test 2025 onward). Ship recommendation: do not ship either head, and the null is why — a model with its sequences shuffled reaches conditional AUROC 0.735 against the real model's 0.756, so almost all of step 6's headline number is the model ordering assays rather than enzymes.
• scripts/experiments/censored_model.py: four new capabilities. paired_arms tests the difference between two evaluation arms on the identical out-of-fold rows (per-row Δ log-likelihood under a paper cluster bootstrap; Δ AUROC computed inside each fold and never pooled; an exact sign-flip permutation over folds). permutation_null refits the whole cross-validated evaluation under whole-cluster feature permutations. prospective_split trains on papers through a cutoff year and tests on the rest. matrix_b_leak_check proves Matrix B never entered the fit, by sequence content rather than by id, and run raises rather than reports if it ever does. New subcommands null and digest, and run --only-doi fits a single paper out of the database.

Changed

• **The model reads the measurable window from the database instead of from a constant keyed by DOI.** SQL_OBSERVATIONS now selects detection_limit, detection_limit_unit, quantitation_ceiling, quantitation_ceiling_unit and buffer; attach_windows derives the floor regime per row. 278 cells now carry a declared floor where 4 did, 1,410 censored rows sit on a published threshold where none did, and the ceiling branch of the likelihood meets real data for the first time (95 right-censored rows, previously zero). buffer is part of the assay cell's protocol signature, which keeps 470 cells apart that would otherwise collapse.

Fixed

• **One identity matrix was being served to two variants with different protein sets.** Indexing a 400-long cluster-label list with a 118-protein design's row indices does not raise — it silently clusters the wrong proteins. identity.npz now carries its protein_ids and run refuses a mismatch; clusters takes --identity-out so each variant gets its own.
• A right-censored row was being dropped for having no floor. Floor groups are built from left-censoring only, and a right-censored row is dropped if and only if its cell declares no ceiling.
• **A bootstrap p-value could be exactly zero for a difference that is identically zero.** Both tails now use the (1 + count) / (1 + draws) convention.

Added

• **scripts/ingest_erickson_panel.py — the panel write step 3 planned and deliberately shipped without.** Four verbs, and only two of them write: a SELECT-only census (read-only session, transaction_read_only asserted, engine disposed in a finally, no visibility column and no accession named), an offline plan, an apply that inserts in one transaction and writes a rollback file naming every row it created, and a rollback that deletes exactly those rows after re-checking each one still belongs to the paper. It never touches interactions, proteins or protein_variants, asserts the 291 embargo holds inside the write transaction either side, and refuses any target interaction that is not PET.
• A rollback verb on scripts/backfill_detection_limits.py, sharing one body with apply run with before/after swapped — a rollback that is not literally the apply run backwards is not a rollback. Both verbs now assert the 291 holds before and after.
• scripts/rollbacks/ — the committed restore path for both writes: the id lists, the pre-write watermark, and a plain-SQL copy of each undo for a machine that cannot run the scripts.
• docs/phase5-panel-ingest.md — what was written, what re-derivation found that the plan did not, the verification, and four things found and left alone.

Changed

• Production data (two owner-approved writes). 2,329 experiments gained the measurable window their paper declares — the four 0016_experiment_window columns, empty since the migration landed. Erickson 2022's cell-level panel was inserted: 3,539 experiments and 3,704 observations, from 4,493 of the workbook's 4,680 rows, carrying per-cell temperature, buffer, pH, duration, substrate morphology and crystallinity, and the assay's 1 mg/L floor on every row. All 1,430 of that paper's zero observations now carry the threshold that produced them; none did before. No verdict moved, no interaction was created, and the effective study count for PET negatives is 3.9165 either side.

Fixed

• Three enzyme ids in Erickson's FASTA contain a space (LCC, LCC ICCG, LCC WCCG), and the usual "identifier up to the first space" rule folds all three onto one key with the last record winning — handing wild-type LCC the engineered WCCG chain rather than merely losing a sequence. The reader takes the whole header and treats a genuine duplicate as an error.
• A panel cell is keyed by its assay series. Table D6 reprints D3 cell for cell, but D5 and D7 are 168 h time courses whose 96 h sample is a different reaction from D3's 96 h endpoint — up to an order of magnitude apart. Without the series in the key, 74 distinct measurements merge away. D5 and D7 disagree on 63 cells and both values are kept, per the label contract.

Added

• docs/phase4-release.md — the record of the 1.19.0 release. What shipped and in what order, the amendment made to the curation worklist before #478 merged, CI per required check, the deploy timeline, and every post-deploy verification with its result: /health at 1.19.0, alembic_version at 0016_experiment_window, all five new experiments columns present, nullable and empty, 291 visibility holds unchanged, and row counts unchanged at 6,526 / 867 / 17,323 / 27,763. Also the rollback procedure and its two rules, and the one check that could not be completed — the signed-in journey, for which no local browser profile holds a v2.plasticdb.org session and no autofill exists to click.

The PET discrimination programme, landed. Nine steps asked whether PET-degrading and non-degrading enzymes are separable by sequence. The answer that had been recorded — that they are not, and that the difficulty is biological — turned out to rest in part on two measurement defects and on a substrate rule with its sign inverted. This release ships the corrected instrument, the documents, and the one schema change the corrected programme needs. It writes no data. Every curation and backfill the programme identified is staged as a dry run and still waiting on an owner decision.

Minor rather than patch: scripts/experiments/pet_discrimination.py gains a CLI flag (--allow-incomplete-blocks), a new exception in its public surface (BlockUnavailable), and a changed default — a feature block that cannot be computed now stops the run instead of degrading silently.

Added

• 0016_experiment_window — five nullable columns on experiments: buffer, detection_limit, detection_limit_unit, quantitation_ceiling, quantitation_ceiling_unit. They make an experiment's measurable window explicit, so a reported zero can say what it was below instead of being read as an absence. Applied by the ArgoCD PreSync hook on deploy; the columns land empty, because the 4,680-row insert that would populate them is not approved. Upgrade and downgrade both proven on a populated table.
• A condition-aware target. The same 212 proteins, the same features and the same models score 0.740 when the target names an assay condition and 0.534 when it is a cross-study protein-level bit. The target is worth 0.21 AUROC — more than any feature block in the programme.
• scripts/check_migration_deploy_safety.py — heads, preflight and prove-unknown-revision. Stops a schema stamp from taking the whole deploy pipeline down: the PreSync hook runs alembic upgrade head against the migration directory baked into the image, so a database stamped ahead of the image fails the hook and aborts the sync of every unrelated change.
• The programme's evidence, as documents, indexed by docs/discrimination-programme-results.md: the diagnosis, the fix plan, and one file per step (docs/phase1-* through docs/phase4-*), plus docs/phase4-premerge-resolution.md, which records hunk by hunk how this branch and #479 were merged without either side silently deleting the other's fix.

Fixed

• Two published nulls were measurement defects, not results. The profile-HMM block read below chance because training rows seeded the profile that then scored them, and because one out-of-fold vector was pooled across five models with different score scales. Corrected — leave-one-seed-out rescoring, per-fold AUROCs, and a cluster-level permutation null — the block reads 0.617 (fold sd 0.084, p = 0.32): uninformative and imprecisely measured, not anti-predictive. The old path is kept as a harness check and still reproduces the published figures exactly.
• A feature block that cannot be computed now stops the experiment. MAFFT's portable tarball invoked without MAFFT_BINARIES prints v0.000, exits 0, and fails every alignment; the block returned seven zero columns and a "ran" status, which scores AUROC exactly 0.500 — indistinguishable from a real null, in the one experiment whose entire claim was a null. It now raises BlockUnavailable. --allow-incomplete-blocks restores the old behaviour deliberately and keeps the dropped block's columns out of the composite rather than diluting it.
• The accessibility axis had the sign backwards. Amorphous film was treated as the accessible substrate. It is the harder one, in both independent panels (0.171 vs 0.471 detection over 5,478 matched comparisons; 0.598 vs 0.804 over 826) and in an assay-severity parameter fitted without ever seeing a condition column: surface geometry dominates crystallinity in a whole-particle assay. Correcting it **empties the stringent negative tier, 23 → 0** — all 23 members had qualified on the falsified clause alone. Every stringent-tier claim in the record needs restating before it is quoted again.
• Four vocabulary bugs in negative_stringency.py: a detection method the database can never hold (uhplc is not in METHOD_VOCAB), no temperature-unit normalisation, a NULL substrate_form clearing the polymer axis, and no years duration conversion. Each moves zero rows today; a rule member that can never match looks exactly like one that never fires, so check_vocabulary() now fails loudly at start-up if the rule names a method the schema cannot store.

Changed

• docs/pet-negative-curation-worklist.md is corrected in place. Its central claim — that curating five already-held papers makes the same-paper control Cpap constructible — is false. The 13-of-15 Cpap failure was reproduced exactly by rebuilding the fold geometry rather than fitting a model, and its cause is cluster geometry, not sample size: the donor paper's 96 negatives occupy two 25 %-identity clusters, GroupKFold cannot split a cluster, and a draw fits only when it catches the lone outlier. Curating other papers grows the draw without adding clusters. Also corrected: B1 delivers 13 not 15 (two are conflicting cells) and zero under the corrected axis, and B2's negatives are not unextracted but mis-signed. The corrections are marked in place beside the original text; the conditions tables, the ~40-paper rejected-lead registry and the honest assessment stand.
• The model recognises families, not activity. It ranks an improved variant above its parent 21 % of the time, and the failure is dose-dependent in substitution count (0.407 at one substitution, 0.045 at five or more), while prospective discrimination on the same corpus reaches 0.685. Both are true at once, and together they are the signature of homology recognition. T4 and T5 are wired and belong on every model card.
• Structural features are closed on measurement rather than on cost. They move discrimination by −0.068, upper bound +0.010; separately, none of the 19 candidate ORFs could be folded at all.
• The HMM screen became a feature and the whitelist stopped drifting (phase 7 integration): #318 closed, #341 in part, #459 fixed.

Not done, deliberately

None of the following was applied; each is staged, dry-run and diffed, and each is an owner decision:

• 13 verdict reclassifications forced by the source paper's own declared threshold — the only lever that moves the effective study count (3.50 → 3.92);
• the 4,680-row Erickson observation insert that would populate 0016's columns;
• 14 sequence-provenance reviews, 8 under a current PET negative;
• 3 interactions whose positive verdict contradicts their own stored observations (TrmA, TrmH, PET46), where a surrogate assay set the verdict.

No database write, no visibility change, and no Figshare publication is part of this release. The 291 protein visibility holds are unchanged.

Added

• **scripts/check_migration_deploy_safety.py — the check that stops a stamp from taking the deploy pipeline down with it.** The deployment applies schema changes from an ArgoCD PreSync hook Job (plasticdb-migrate) whose command is python -m app.db.init_schema and nothing else, and that module runs `alembic upgrade head` against the migration directory baked into the image. Advancing production's alembic_version to a revision the deployed image does not carry therefore makes the next hook fail — and a failed PreSync hook aborts the ArgoCD sync of every change in that release, not just the schema one. Proved against a throwaway PostgreSQL with the migration directory extracted from the deployed commit: alembic upgrade head raises CommandError: Can't locate revision identified by …, and so does the obvious repair, alembic stamp <older>, because stamp resolves the current revision first. Only alembic stamp --purge or a raw UPDATE alembic_version walks it back. Three verbs: heads (compare the repo's head with the head baked into the image the hook Job is configured with, read live from the cluster; exits non-zero when the repo is ahead), preflight (read-only census — revision, target columns, row counts, visibility holds, locks and idle-in-transaction backends; asserts transaction_read_only = on and disposes the engine in a finally), and prove-unknown-revision (reproduce the failure on a scratch database). Covered by tests/test_migration_deploy_safety.py.
• **docs/phase4-migration-landing.md — the landing decision for 0016_experiment_window, and the sequence that makes it safe.** The migration was not applied: production runs 1.16.0 from commit 7e34e4b, whose baked alembic head is 0015_protein_provenance, so stamping 0016 now would break the next deploy of anything. Nobody needs to run alembic by hand in the first place — the PreSync hook already does, so the merge is the apply. The document records the evidence, the two reproduced merge conflicts on the way in (VERSION/CHANGELOG.md on the second merge; 11 hunks in scripts/experiments/pet_discrimination.py's evaluate and fold_profile_hmm_features, which #479 hardens and this branch rewrote), the rollback command, and the rule that an image rolled back past 0016 needs the database downgraded first. Production is unchanged: alembic_version still 0015_protein_provenance, 0 of 5 columns on experiments, 6,526 interactions / 867 proteins / 17,323 experiments / 27,763 observations, and **291 visibility holds before and after**.

Changed

• **The thirteen Norton-Baker PET verdict corrections, applied — the first database write of the discrimination programme.** Thirteen enzymes were stored as PET negatives of 10.1021/acscatal.5c03460 while that paper's own supplementary workbook records them releasing aromatic product above the threshold the paper itself declares ("activity was defined as an aromatic absorbance reading at 260 nm of 0.05 above the blank in at least 1" condition). All thirteen were re-derived from the published workbook rather than inherited from step 3's list, and all thirteen survived: each has one to three cells at A260 0.17–0.28, against a workbook floor of 0.17. Identity was resolved on the sequence and never on the name — 3 exact, 10 after trimming the initiator methionine and the C-terminal His tag. One transaction, interactions.result on thirteen rows, negative → positive. The paper's PET negatives go 97 → 84, the corpus's 192 → 179, and the effective study count 3.4995 → 3.9165, matching the prediction to four decimal places. The 291 visibility holds were asserted inside the transaction either side and are unchanged; visibility was never written or filtered on. See docs/phase4-verdict-corrections.md.

Added

• scripts/experiments/verdict_corrections.py — the apply, and the way back. Five subcommands: derive (pure, re-reads the workbook), census (read-only, asserts transaction_read_only = on), plan (pure, emits the rollback), apply / restore (dry runs unless --apply, one transaction, every target re-read FOR UPDATE and refused unless it is still the exact row the plan described) and verify. The rollback is committed alongside it — verdict_corrections_rollback.json and verdict_corrections_restore.sql — redacted so it restores exactly without naming an embargoed accession: the workbook is published, so a sample_id beside a protein_id would be the leak. Proven by a full round trip on the live rows, not by inspection. Covered by tests/test_verdict_corrections.py.

Added

• docs/discrimination-programme-results.md — one page for the nine steps. The diagnosis document argues why, the fix plan says how, and this records what happened when all nine ran: every gate's outcome, the five findings that change what we believe, the six staged decisions waiting on the owner, and the merge order. Three published conclusions did not survive: the profile-HMM block was never anti-predictive, the accessibility axis has the sign backwards (which empties the stringent negative tier), and the paper-confound control fails on cluster geometry rather than sample size. Nothing in the programme wrote to the database; every write is dry-run and diffed.

Added

• /tools/hmm-screen — the HMM screen finally has a way in (#318). A tool page like the other five, gated behind an account, posting a multipart FASTA to POST /api/v1/jobs with a bearer token (the API rejects the session cookie on purpose, so a cookie-only fetch 403s on every job that has an owner). The polymer options are rendered server-side from the same vocabulary the endpoint validates against, so the form cannot offer an option the endpoint will reject. The landing page's "HMM Discovery" capability card — the one card #288 could not link, because there was no honest destination — now links to it, and both /documentation/tools and /documentation/api describe the feature, where before neither contained the string "hmm".
• **app/services/hmm_library.py + scripts/publish_hmm_library.py + an hourly check-hmm-library beat: the profile library is now publishable, and its absence is no longer silent (#341).** PlasticDB.hmm has never existed in a deployment — only the DIAMOND databases are on the shared volume — so every HMM-screen job failed inside hmmsearch on a path the user could not see. Publishing installs a completed Phase 3 build directory atomically (os.replace, fingerprint written last, an incomplete build refused), and the scheduled check republishes a missing or stale library within the hour, exactly as the pathway databases do. The operator path and the automatic path run the same code, so they cannot diverge. The publish script's database session is read-only and verified, and disposed in a finally.
• The library fingerprint is widened, which is the substantive half. The DIAMOND digest hashes (id, has_sp, cleavage, sequence) — enough for a database whose only input is sequences, and provably not enough for one whose profiles are grouped by family and labelled by polymer. An interaction flipping positive → negative changes the library while leaving every sequence byte-identical; the sequence-only digest would not move and the library would silently never rebuild, which is the exact shape of #330. Polymer associations and their verdicts, the Phase 1/2/3 recipe constants (TCS_MIN, MAX_TRIM_FRACTION, SPLIT_IDENTITY_LADDER, MIN_SEEDS_FOR_PROFILE, DEFAULT_FAMILY_IDENTITY, RANDOM_SEED) and the pinned toolchain versions all enter the digest. Everything is ordered; nothing iterates a set (#470).
• HMM_BUILD_DIR setting — a completed build directory for the hourly check to publish from. Empty by default, which reports staleness and publishes nothing.

Fixed

• **POST /jobs accepted a job type that could not work and rejected one that could (#459).** JobTypeEnum offered build_tree, whose handler never reads input_file_path — so a client following the enum uploaded a FASTA, got a 201, and the job built something unrelated to what it sent; that request belongs at POST /tree/build. It rejected hmm_screen with a 422, so the one path that can carry a real-sized file could not reach the HMM screen at all, which is what blocked #318. build_tree is out, hmm_screen is in, with target_polymers as a repeated form field validated before the upload is written. tests/test_job_type_registry.py now ties the enum, the worker's _PIPELINES registry and the job_type_enum column together so they cannot drift silently again.
• **An hmm_screen job submitted through POST /jobs would have been handed the DIAMOND engine.** dispatch resolves the backend from job.engine, which defaults to diamond; the endpoint now sets hmmer for that job type, as POST /bioinformatics/hmm-screen already did. pident is stored as NULL there too — hmmsearch reports no percent identity, so recording the form's default would put "30%" on a job that never applied it.
• An absent profile library no longer reads as an empty result. run_hmm_screen checks for the library before an engine is resolved or a job directory created, and fails with a sentence a user can act on. "Nothing was searched" and "nothing was found" are different answers.
• The job page rendered an HMM screen as though it were a DIAMOND search. jobs/status.html branches on job type in five places and knew nothing about hmm_screen, so the results table headed a profile as "Subject", showed a % Identity column reading 0.0% on every row because hmmsearch produces none, and claimed a sequence database had been searched. It now labels the column Profile, drops the identity column, and renders the per-polymer tally from result_metadata.

Changed

• VALID_POLYMER_NAMES is a fallback, not the contract. valid_polymer_names() reads the published polymer_policy.json — Phase 3.4's computed evidence-tier vocabulary — and falls back to the hardcoded frozenset only while nothing is published, because rejecting every polymer would be worse than a stale list.

Documentation

• docs/phase3-integration.md — what was integrated, where the job-type whitelist had drifted (the ORM column had not; the job status template and the API documentation had), the signed-in journey traced step by step, the part of #341 that is deliberately not automated and why, the embargo question left for the owner, and a twelve-step post-deploy verification checklist. Nothing here has been deployed and no HMM job has ever run end to end.

Changed

• **The structural feature question is reopened, measured, and closed again — on a different reason than the one on file.** (scripts/experiments/structural_features.py, tests/test_structural_features.py, docs/phase3-structural-features.md, docs/ml_engine_design.md §4.) ml_engine_design.md excluded *anything structural* on the strength of Phase 5's net −1 query. Phase 5 measured structure as a retrieval arm competing with DIAMOND, and said in its own words that *"this phase measured retrieval, not ranking, and those are different questions"* — so §4 cited Phase 5 as having answered a question Phase 5 explicitly left open. Step 8 answers it.
• **Gate: no. Δ AUROC = −0.068, 95% CI [−0.168, +0.010] over 71 test identity clusters, cluster-level permutation p = 0.16.** Pre-registered estimator, with source-organism growth temperature inside both arms, under Step 2's prospective split (train ≤2024, test 2025+; 64 train / 149 test, predicted models only). A better-powered 70 %-identity cluster-CV arm (n = 214, fold sizes 43/43/43/43/42) gives −0.026 ± 0.027 over folds, p = 0.64. **n is small and is not the reason for the answer:** the interval's upper bound is +0.010, so a gain above one AUROC point is excluded; detecting a true +0.05 would need 944 test proteins and +0.02 would need 5 895.
• **The growth-temperature control did not change the picture, because there was nothing to control.** Source-organism optimal growth temperature (TEMPURA, species then genus, 138 of 213 matched) is uncorrelated with the PET label: Spearman −0.015, 30.9 °C mean for positives against 30.1 °C for negatives. Dropping the control moves the structural difference from −0.068 to −0.073.
• A separate and sufficient reason: the block is not computable at inference. All 19 Phase-4 candidate ORFs failed to fold — the public ESMFold endpoint returns 413 above 400 residues (13 of 19) and 504 below it (6 of 19), while folding a 33-aa control in 1.5 s. §4's requirement that every feature be computable for a novel ORF is violated at the first ORF.
• The cost argument was also about a different object. The 1 500-CPU-year figure was for folding an nr-sized catalogue. 402 models already existed in the structure store, and extracting every geometric feature from all of them — 126 990 residues — took 15.6 wall-minutes on one core.
• **A predicted Tm does not inherit the ascertainment bias that excluded measured Tm.** Measured Tm scored 0.731 in Phase 0 and was refused because labs characterise the enzymes they believe are active. An ESM-2 + ridge model trained on FLIP's Meltome Atlas (held-out Spearman 0.530, RMSE 7.4 °C) runs the other way — AUROC 0.327 overall, and 0.359 / 0.358 / 0.364 in three disjoint slices — and is uncorrelated with growth temperature (ρ = −0.009). The rule that carries forward is about the label set a predictor was trained on, not about the word "predicted". It is still not shipped: it duplicates the thermo_ proxy (0.64 flipped against 0.628) and did not rescue the ablation.
• **Structure provenance is itself an ascertainment variable, and is now measured.** Among the split's proteins, deposited experimental structures are 89.5 % PET-positive against 61.0 % (AlphaFold) and 64.9 % (ESMFold). The ablation therefore admits predicted models only; once it does, having a model is unassociated with the label (63.6 % vs 62.4 %).

Added

• scripts/experiments/structural_features.py — the folding and feature-extraction pipeline, re-runnable, with the tool versions recorded: stages models, fold, features, ogt, tm, ablate, clustercv, power. Cleft detection is LIGSITE-style with the PSP threshold at 4 rather than the published 5, calibrated on two known active sites (IsPETase 6EQE and Fusarium cutinase 1CEX) before any label was scored, and the cleft measured is the one the catalytic serine sits in — on 6EQE the largest enclosed pocket is 24.4 Å from it. SASA, pocket geometry and triad detection are written out in-process; freesasa, fpocket, dssp and foldseek are on no host in this stack.
• tests/test_structural_features.py — 14 Tier-1 cases pinning the geometry against shapes whose answers are known, plus the pre-registration invariant that the two compared arms differ by the structural block alone.

Changed

• **The negative-stringency rule's accessibility axis had the sign backwards, and is corrected.** (scripts/experiments/negative_stringency.py, docs/label-matrix.md, tests/test_negative_stringency.py.) The axis read *"crystallinity ≤ 15 %, or an amorphous_film with no recorded crystallinity"*. The second clause applied a statement about crystallinity to a variable that is geometry. On matched enzymes — only the enzymes both cells actually ran — an amorphous film detects less often than a crystalline powder in two independent panels (0.171 vs 0.471 over 5,478 comparisons; 0.598 vs 0.804 over 826), and a severity parameter fitted without ever seeing a condition column agrees (θ +0.653 film vs −0.455 powder). Hold particle size constant and the crystallinity effect returns, small and in the expected direction. **A film is the hardest substrate measured, so a film null cannot be what certifies an assay was easy.** New GEOMETRY_LIMITED_FORMS; crystallinity remains the axis. PET46 — null on amorphous foil, 1,624 µM aromatic products from semi-crystalline powder — is therefore not filed as an exception: it is the general ordering, and it is the case that falsified the clause.
• **Measured cost of the correction on the production corpus: the N-stringent PET negative tier goes 23 → 0 and N-permissive 119 → 91.** All 23 stringent negatives reached the tier through the falsified clause and through nothing else. Exactly one negative experiment in the database clears the corrected axis — SfCut, a 4.8 %-crystallinity pellet at 70 °C — and its protein has no stored sequence. Phase 0's stringent-tier result (AUROC 0.864, z = +3.49) is the true output of the rule as it stood and can no longer be read as evidence that hard conditions separate.

Fixed

• **Four vocabulary bugs in the stringency rule, each costed separately and all four moving zero rows today** — which is the finding, because a rule member that can never match looks exactly like one that never fires. SENSITIVE_METHODS carried uhplc, which is not in evidence_vocab.METHOD_VOCAB, so experiments.method_id could never point at it; gc_ms and gc are in the vocabulary and were missing, so a GC-quantified null would have failed the detection axis silently. A new check_vocabulary() runs at start-up and fails loudly if the rule ever names a method or substrate form the database cannot hold.
• temperature_unit is a String(8) with no enum; the rule tested unit in ("C", ""), so celsius, °C, F or K would each have emptied the thermal axis. Production is uniformly C — the fix is the guard, not a repair.
• "real_pet": form not in SURROGATE_FORMS returned **True for a NULL substrate_form**: the one axis that cleared on missing data, contradicting the docstring directly above it.
• _HOURS_PER had no years entry, so the four production experiments recorded in years returned None and failed the duration axis.

Added

• Step 7's curation re-verification and dry-run payloads. (scripts/experiments/curation_worklist.py, tests/test_curation_worklist.py, written up in docs/phase3-curation-and-stringency.md.) Read-only; **there is deliberately no --apply flag.** 20 reviewable curation operations, embargoed accessions masked.
• **The gate answer: Cpap is still not constructible — 2 of 15 draws fit today, 5 of 15 after every worklist payload, and the tier is empty under the corrected rule.** Reproduced without fitting a model, by rebuilding the exact GroupKFold geometry evaluate uses. The cause is not sample size: the donor screen's 96 negatives occupy two identity clusters, 95 in one, and GroupKFold cannot split a cluster — so a draw fits only when it happens to include the single outlier. Curating other papers grows the draw, not the donor's cluster count. The paper confound remains untested.
• Three corrections to docs/pet-negative-curation-worklist.md (PR #478). B1 delivers 13, not 15 — two of Erickson's fifteen are already conflicting cells. B2's TrmA and TrmH are not unextracted: both PET-film nulls are already stored as observations under interactions whose result says positive, contradicting the paper and the observation text. And the worklist's headline — 91 % → 51 % single-study concentration making Cpap constructible — measures 91 % → 54 %, and moves the effective study count from 3.44 to 3.51 against a gate of ≥ 5.

Added

• **The censored, condition-aware model — the replacement target, fitted and gated.** (scripts/experiments/censored_model.py, tests/test_censored_model.py, written up in docs/phase3-censored-model.md.) Step 6 of docs/discrimination-fix-plan.md, and the largest step in the programme. a_ij = f(sequence_i) − theta_j + noise, observed inside each assay cell's measurable window and left- or right-censored outside it. f is sequence-only — it takes an ORF and returns a number, with no pH, no substrate form and no paper id, which is the answer to docs/ml_engine_design.md §4. theta_j is the severity of one (paper × protocol) cell under a hierarchical prior, **marginalised out in closed form at prediction time**. The censored likelihood is written directly in torch — log_ndtr, thirteen lines, no survival-analysis dependency — and checked against the closed forms.
• Gate 2 passes: the fitted severity recovers physics it was never told. Nothing about temperature, pH, substrate form or duration is a feature; they are grouping keys. Fitted on Erickson 2022's panel, theta orders the eight reaction times strictly monotonically with the zero-hour control as the single hardest cell in the study (Spearman −1.000), puts 60-70 °C easier than 30-40 °C on 12 of 13 paired contrasts, and puts higher pH easier. **The plan's own worked example fails, and the raw data says the example is wrong**: on the enzymes both cells actually ran, the amorphous film detects less often than the crystalline powder in both independent panels (0.171 vs 0.471; 0.598 vs 0.804). Isolate crystallinity at matched particle size, as Erickson's cryomilled powder does, and the effect is small and in the expected direction. Surface geometry dominates crystallinity in a whole-particle assay, and negative_stringency.py's accessibility axis has the sign backwards.
• Gate 1 fails, and the reason is stated rather than smoothed. Scored on the one event a protein-level bit can also predict — "would this assay have detected this enzyme?" — the censored model loses on held-out likelihood (−1.048 ± 0.326 against the binary model's −0.641 ± 0.275 over five paper folds) while winning on ranking (detection AUROC 0.573 ± 0.090 against 0.431 ± 0.171, i.e. the binary model is below chance). The likelihood arm is dominated by a laboratory's own detection rate, which this model deliberately refuses to predict. **When the assay cell is named, the censored model reaches AUROC 0.759 ± 0.153 at a 60% identity holdout** — the same region as step 4's condition-specific 0.740 on someone else's benchmark.
• T4 moves from 0.202 to 0.360 on the identical 114 PET triples, and the dose-response that was step 2's sharpest finding disappears: the old model scored 0.407 / 0.132 / 0.045 across one, two-to-four and five-or-more substitutions; the new target scores 0.366 / 0.385 / 0.286. Still below a coin flip and far below the ≥ 0.70 criterion — but the interval now contains 0.5, so "it fails T4" becomes "the test cannot tell".
• T3 reported as the two experiments it is. The plan's 23,072 within-study pairs reproduce exactly, and 96.3% of them are one paper. Split: the dominant screen alone is 22,155 pairs at 0.589 accuracy, everything else is 847 pairs over 36 papers at 0.615. Both grains are reported, because the same table holds 150,435 cell-level comparisons and only 23,002 distinct protein pairs. **20.5% of informative pairs reverse their ordering between two conditions of the same paper**, which puts a ceiling of about 0.79 on any condition-free ranking head — a property of the question that no gate in ml_engine_design.md currently states.
• **A named holdout stringency, chosen by a rule fixed before any AUROC existed:** 60% amino-acid identity, centroid linkage, 50% coverage gate. Step 1's pathology reproduces on the larger corpus — one 25%-identity cluster holds 83.8% of the 400 modelled proteins — so a 25-30% split does not exist here either, and every number in the write-up carries the threshold it was measured at.

Fixed

• A leak that gave the binary baseline its entire apparent advantage. Selecting every protein whose PET verdict came from a training paper hands the baseline the held-out protein's own label whenever a paper straddles the split — which under a sequence-cluster holdout is almost always. Restricting it to proteins that appear in the training rows moved its cluster-fold AUROC from 0.745 to 0.589.
• Three defects in the fit itself, each of which produced plausible output. torch.where evaluates both branches, so a sentinel floor on an uncensored row became log_ndtr(-inf) and its NaN cotangent poisoned every gradient — the first fit returned theta = nan for all 432 cells and reported success. A ceiling declared at log₁₀ = 0.0 is falsy, so ceiling or 1e9 silently threw it away. And the hierarchical variance is degenerate at its own joint mode (tau_within → 1e-9, every cell collapsing onto its paper's mean), so the two variance parameters are fixed hyperparameters selected by nested cross-validation rather than fitted. All three are regression tests.

Added

• The assay's measurable window is a column, not prose. (Migration 0016_experiment_window, scripts/backfill_detection_limits.py, written up in docs/phase2-censoring-schema.md.) Step 5 of docs/discrimination-fix-plan.md. experiments gains detection_limit / detection_limit_unit and quantitation_ceiling / quantitation_ceiling_unit. 73% of the PET-negative interactions carry an explicit value = 0 and 92% of those zeros say "below detection limit", "ND" or "no bar above the baseline" in free text — so the number and the meaning of the number lived in two different places, and every consumer read a zero without being able to tell "nothing happened" from "nothing this instrument could see". Both panels are interval-censored, not merely left-censored: Norton-Baker 2025 bottoms out at A260 ≥ 0.05 over blank, Erickson 2022 tops out at 500 mg/L, and a ceiling that flattens the best enzymes is as damaging as a floor that ties the worst. All four are nullable: "the paper did not say" has to stay distinguishable from "the paper said zero".
• experiments.buffer — the buffering agent the pH was held at. Erickson ran sodium phosphate pH 7.5 and HEPES pH 7.5 over every enzyme in the panel; stored without the agent, **545 of its condition pairs were indistinguishable**. It is part of _EXPERIMENT_CONTENT_FIELDS, the identity a promotion uses to decide whether an experiment already exists, because a condition column outside that tuple would let promotion merge the very pairs the column exists to keep apart. Curator-editable end to end: schema, curation form, curation_form.js, and the contributing docs (ExtractedData grows from 103 exportable leaves to 104).
• The backfill's dry run, and what it found. census / plan / apply, so only the verb named apply can write; the census is read-only, disposes its engine in a finally, and selects no visibility column and no accession. The plan covers **2,326 panel experiments — 100% of both papers — with 0 conflicts, and 86.7% of the zeros behind the PET negative class** now carry the threshold that produced them. The prose the plan hoped to mine turned out to be almost empty: 146 experiments declare a floor and state no number, and only 3 in the whole corpus state a floor with a number and a unit. The prose pass's first draft matched 12 rows and was wrong on 10 — LOD inside "cyclodextrin", a keyword paired with the next sentence's concentration, dissolved-oxygen "saturation", and "0.1 um" read as 0.1 µM — each now a regression test.

Added

• The published benchmark reproduced, both arms, on someone else's yardstick. (scripts/experiments/benchmark_nb2025.py, tests in tests/test_benchmark_nb2025.py, written up in docs/phase2-benchmark-reproduction.md.) Step 4 of docs/discrimination-fix-plan.md — the step the plan calls "the one that settles the argument". Norton-Baker 2025's balanced slice (crystalline PET powder / 40 °C / pH 5.5, **212 library candidates, 100 active against 112 zero) is run through our own feature blocks two ways: condition-specific**, held out by sequence cluster inside the slice, and cross-study, trained on our literature corpus with this paper's evidence removed — the direct analogue of PETML's D1-Scraped-513, which the same authors report at AUROC 0.470-0.474 on this condition. The contrast is the result. Same features, same two models, same 212 test proteins; the only difference is whether the training target names a condition. **Condition-specific: 0.740, fold sd 0.164, 95% CI [0.536, 0.944], p = 0.025 against a 200-draw cluster-level permutation null. Cross-study: 0.534, cluster bootstrap [0.449, 0.668], p = 0.81 against the same kind of null** — and two of its fifteen block × model pairs land on 0.442 and 0.450, the published number almost exactly, as does the strongest trivial baseline. AUROC is reported as a mean and spread over folds, never a pooled point estimate, and the nulls are cluster-level throughout (the sequence-level ones are reported beside them and give p = 0.005, which is the gap step 1 documented). The clustering is reported before any AUROC, as the plan requires. Norton-Baker's library is one homologous family — median pairwise identity 0.488, 99.9% of pairs at or above 25% identity — so at 25% and 30% identity the whole slice is a single cluster and **no held-out split exists at all**; at 40% and 50% one cluster holds 96.7% and 86.8% of it, which is step 1's fold-geometry pathology recurring. The identity threshold is therefore derived by a rule fixed in the source before the run (the most stringent threshold whose fold geometry is sound), which lands on 0.70, and the AUROC is reported across the whole 0.50-0.90 grid because it rises monotonically with how much homology the split lets through. Nothing was written to the database and no connection was opened at all — the benchmark's input is step 3's file and the corpus is step 2's cached read-only dump. visibility was neither read as a filter nor modified; the 291 holds are intact and no non-public accession appears in the write-up.

Fixed

• **A pre-registered estimator naming a composite block silently lost its permutation null.** (scripts/experiments/pet_discrimination.py.) evaluate's null-target selection tested p_block in blocks, and blocks holds only the atomic feature blocks — so pre-registering joint or joint_plus_esm produced a report whose primary estimator carried p_vs_cluster_null: null, which reads as "no association" rather than "never measured". Composite blocks now get their matrices, an unbuildable one gets an explicit not_run with a reason, and the full-pipeline null says why it cannot address a composite instead of returning nothing. evaluate also takes an optional primary_estimator so a different benchmark can pre-register its own pair without touching step 1's PRIMARY_ESTIMATOR.

Added

• **Both PET panel workbooks extracted quantitatively, and a dry-run ingestion plan that says the ingest does not do what step 3 assumed it would.** (scripts/experiments/panel_workbooks.py, scripts/experiments/panel_ingestion_plan.py, tests in tests/test_panel_workbooks.py and tests/test_panel_ingestion_plan.py, written up in docs/phase2-panel-extraction.md.) Step 3 of docs/discrimination-fix-plan.md. Nothing was written to the database — every session was opened postgresql_readonly=True and verified transaction_read_only = on, every engine disposed in a finally, and pg_stat_activity checked afterwards for leaks. visibility was neither filtered on nor modified; non-public accessions are masked NONPUBLIC_nnn throughout and the mask mapping was not stored. **The apply is documented as a single command and has not been run.** The gate, three ways. Effective study count for PET negatives, as inverse Simpson over per-paper protein counts: 3.4995 today (reproducing the diagnosis figure exactly), **3.4995 if the panels are ingested as a pure addition — unchanged to four decimal places — and 3.9165 if the verdict corrections the source data forces are applied with it**. Why addition cannot move it. All 65 of Erickson's screened enzymes resolve by sequence onto proteins already in the corpus (16 exact, 49 after trimming the expression construct's His tag and initiator methionine); Norton-Baker proposes two new rows, neither a negative. **Not one new PET-negative protein comes out of either panel.** And structurally it could not: Norton-Baker is already 97 of the 192 protein-paper negative pairs — 50.5% of the whole minority class — so more negatives from it make the class more concentrated. The only lever is subtraction, and there is one: **13 enzymes ingested as Norton-Baker PET negatives released product above the paper's own declared A260 ≥ 0.05 threshold**, taking that paper from 97 to 84. Norton-Baker is already ingested quantitatively — 2,124 experiments with per-condition temperature, pH, substrate form, crystallinity and a umol/mg value, 2,080 of them matching a workbook cell exactly. 3a is an enrichment of 78 cells, not a bulk insert. Erickson is where the information is: its 202 existing rows are best-condition summaries, and the apply would add 4,680 cell-level measurements (1,859 left-censored zeros, 310 right-censored at the 500 mg/L ceiling) across D3-D7. Sequence-level dedup, never name matching (the owner's standing instruction). Four tiers — exact, contained, near, new — and three rules that each prevented a wrong write: trimming the expression construct before comparing (without it, 32 candidates already in the corpus looked new), resolving a -nSP / -pelB construct to its base enzyme, and checking protein_variants before minting an engineered benchmark as a wild type. A number never becomes a verdict. Erickson's enzyme 410 has one cell at 1.19 mg/L out of 30 and the paper's own Supplementary Table 11 calls it "No Activity", so it stays negative with the measurement recorded beside it. Four enzymes screened at 2-5 µg rather than the study's standard 10 µg are reported as low_yield_artefact, not as hard negatives. **The balanced 216-enzyme slice is a standalone artefact with sequences attached** (work/step3-2026-09-05/nb2025_balanced_slice.tsv), which step 4 needs. Measured: 215 enzymes, 102 active against 113 zero — the plan's 216/102/114 counted the buffer-only blank well as an enzyme. Two parsing faults that would each have been silent. Erickson spells the quantitation ceiling 500+ in 163 cells and 500* in exactly one, so a +-only parser reports 1,949 D3 measurements where the sheet holds 1,950; and two time-course headers lost their degree sign (608C, 30*C), which reads literally as a 608 °C reaction. Both are covered by tests. Two curation queues the run surfaced. Fourteen Norton-Baker enzymes have a corpus row whose sequence is not the chain the paper's SI publishes under that sample_id although the NCBI accession agrees — eight of them under a current PET negative. And experiments has no buffer column, so Erickson's NaH2PO4 pH 7.5 and HEPES pH 7.5 collapse into one condition for 545 cell pairs unless the agent is carried in additional_conditions.

Added

• **The two falsification tests nobody had run — and the current model fails one of them, below chance.** (scripts/experiments/falsification.py, tests/test_falsification.py, written up in docs/phase1-falsification-tests.md.) Step 2 of docs/discrimination-fix-plan.md. Read-only with respect to the database: one SELECT-only dump on a session verified transaction_read_only = on. T4 — variant direction. Gate: FAILED. Matrix B's 332 matched (parent, variant, polymer) triples are the only controlled contrasts the corpus holds — same scaffold, a few substitutions, same assay, same figure. **203 were scorable; on the 120 whose measurement says the variant is the better depolymerase, the rescorer's forest ranks the variant above its parent in 0.212 of cases (95% CI over parent scaffolds [0.079, 0.396]**, 21 parents, 9 exact ties). The success criterion was ≥ 0.70. The masked multi-label MLP reads 0.383 [0.214, 0.587] — an interval that includes chance, so for that head the honest statement is "the test cannot tell", not "it passes". A constant-score control lands at exactly 0.500, which is the harness's own check that the paired test is wired correctly. The mechanism, which is the useful part. The trivial nearest-characterised-positive baseline fails in the same direction and harder (0.246, and 14% excluding its ties); the median variant-minus-parent score gap is negative; and the failure is dose-dependent — 0.407 for single mutants against 0.107 for variants carrying two or more substitutions. **The score is a decreasing function of distance from the characterised set, and engineering a protein is a walk away from that set**, so on a variant series the model ranks backwards by construction. Phase 0 inferred homology recognition from a leakage gradient; T4 measures it directly, with family membership held constant. Matrix B is a test set only and never entered a training fold: variant rows carry labels = {}, they sit after the record list every training index is drawn from, and two tests pin both facts. Two contaminations that would each have flattered the answer were removed first — **each parent is held out of its own contrast (the model is refitted 23 times) and removed from the profile arm it seeded**. T5 — the prospective split. Train on papers through 2024, test on 2025+. The plan's support table reproduces exactly, with no drift: PET positives 222 train / 181 test, negatives 41 / 151 — and it is at interaction grain, which the plan does not say; at (protein, polymer) cell grain it is 139/150 and 40/150. PET is the only head with a usable 2025 side. **The forest scores AUROC 0.685 [0.621, 0.776] against a trivial baseline at 0.521** — the largest honest gap the rescorer has produced. The caveat is quantified rather than gestured at: the 2025 side spans 15 papers but its effective study count is 1.68 and one paper supplies 76.8% of the test proteins, so the result is split on that paper (0.662 on it, 0.708 on the 63-protein remainder) and reported as one prospective trial. Read together, the tension is the finding: prospective discrimination without activity discrimination is what family recognition looks like. Phase 6.2's ENABLE_ML_ENGINE = False and PET's "polymer undetermined" verdict now rest on two independent measurements instead of one. Both tests are CLI entry points so they can go on every future model card, permanently, as the plan requires.
• A cluster bootstrap that resamples the right unit. An AUROC is a mean over pairs, and 272 test proteins make 18 471 of them; resampling pairs reports an effective sample size in the thousands and an interval three times too narrow ([0.656, 0.715] against the honest [0.621, 0.776]). cluster_bootstrap_auroc resamples identity clusters and recomputes the statistic. T4's interval resamples parent scaffolds, and reports the too-narrow Wilson interval beside it so the difference is visible.

Fixed

• rescorer.fold_profile_features would have crashed on its first real run. Step 1 changed run_hmmsearch to return (hits, error) so that "the profile matched nothing" and "hmmsearch did not run" stop being the same empty dict; this call site still unpacked one value and would have raised AttributeError: 'tuple' object has no attribute 'get' the moment a fold profile was built — which is every run with MAFFT and HMMER present, and none of the mock tier. The error is now handled, recorded in the block's status as degraded, and pinned by tests/test_rescorer.py::test_every_hmmsearch_call_site_unpacks_the_error.

Fixed

• The profile-HMM block was never anti-predictive; the instrument was broken. (scripts/experiments/pet_discrimination.py, written up in docs/phase1-instrument-clearing.md.) Phase 0 reported the profile HMM — the feature the whole pipeline is built on — at AUROC 0.467 and 0.439, below chance, and concluded that it "performs worst". Two independent defects in the measurement produced that number, and removing either one removes the below-chance result:
  • Train/test covariate shift in the headline feature. The fold's positive and negative profiles were built from the training rows and then used to score every row, so training rows were seeds of the profile they were scored against and test rows were strangers to it. The feature had two different distributions and the classifier fitted a boundary on the wrong one. Scoring is now leave-one-seed-out: each seed's row is deleted from the training alignment, the profile is re-estimated, and the seed is scored against a profile that does not contain it — 1,628 rebuilds per run, about four minutes.
  • One AUROC pooled across five differently-calibrated models. Folds here range from 15 to 345 test sequences and 0.40 to 0.94 prevalence, and concatenating their probabilities ranks incommensurable numbers against each other. This is the same defect that produced the spurious 0.460 in docs/discrimination-diagnosis.md §1.2. Every block now reports the **mean AUROC over folds with its fold standard deviation** and the per-fold values; the pooled figure is kept and demoted to auroc_pooled. Corrected, the block reads 0.617 with a fold sd of 0.084 over 5 folds (95% CI [0.513, 0.721]) and ranks fourth of eight rather than last. It is still not evidence of signal: a 200-draw cluster-level permutation null is centred on 0.499 with sd 0.112, giving p = 0.32. Phase 0's overall conclusion — nothing separates PET positives from negatives on this corpus — stands; the claim that profiles are worse than useless does not. docs/phase0-pet-discrimination.md needs five passages restated, listed in the new document.
• The permutation null was too tight and too small to carry a p-value. It permuted labels over sequences, which destroys the within-cluster label agreement the real data has, and it drew 20 samples — a grid that cannot express a p below 0.048. The null is now cluster-level (each cluster receives another cluster's label composition) at 200 draws by default, and is computed for the pre-registered pair as well as the historical one. On the profile-HMM block the two nulls disagree sixfold: p = 0.050 sequence-level against p = 0.32 cluster-level. --pipeline-permutations adds the null nobody had run — the one that rebuilds MAFFT, hmmbuild and the leave-one-seed-out rescoring from the permuted labels, because the profile columns are built from the labels. At 12 draws it is centred on 0.525 and reaches 0.721.
• The reported estimator is pre-registered rather than maximised. PRIMARY_ESTIMATOR fixes one block × model pair in the source. The historical maximum over every pair is kept for arm comparability but now carries n_keys_examined (16) and a Šidák-corrected p-value.
• The split is no longer described as 5-fold cross-validation when it is not. One 25%-identity cluster holds 84.8% of the labelled set, so the run is a single 62 → 345 split plus four folds of fifteen. A fold_geometry block reports the per-fold sizes, the 23× spread, an is_balanced_k_fold flag and a plain-English description, and the same sentence is logged as a warning.
• Four more silent failures in the same experiment. run_hmmsearch returned an empty dict for both "matched nothing" and "did not run", and callers zero-filled a feature column and reported success — it now returns (hits, error). A MAFFT or hmmbuild failure inside a fold no longer yields {"status": "ran"}; the block takes the worst status of its folds and a degraded block can no longer supply an arm's headline. A NaN AUROC is reported as failed with a reason instead of ran. A missing ESM embedding raises instead of becoming a zero vector at the origin of the representation space.

Added

• **A diagnosis of why PET discrimination fails, and a pipeline built on a different target.** (docs/discrimination-diagnosis.md.) Four measured nulls concluded that the difficulty is biological — that positives and negatives are genuinely not separable by sequence. Three new read-only measurements say the binding constraint is narrower and more fixable. The negative class is worth about 3.5 independent observations. 187 PET-negative proteins come from 22 papers with an inverse-Simpson effective count of 3.5. Every held-out-study estimate in the programme is roughly one draw from a distribution of three or four, which is exactly what Phase 6's fold-to-fold SD of 0.03–0.17 predicts. The binary label is a threshold applied to a number already stored. 163 of 224 PET-negative interactions (72.8 %) carry an explicit `observations.value = 0`, and 92.3 % of those zeros say "below detection limit", "ND" or "no bar above the baseline" in their text. They are left-censored observations, and each paper's limit is different. The programme has been binarising ~4 000 quantitative experiments down to 432 protein-level bits. The scale that would let studies be compared has no negative anchor. Only 23 of 411 PET proteins are measured by more than one paper and every one is positive-only, because nobody re-tests an enzyme that failed. 26 of 69 papers share no protein with any other. Measured inter-study disagreement is 17 %, which caps AUROC near 0.90 — so label noise costs about 0.10 and cannot explain a 0.30 gap. The document sets out six ranked hypotheses, six discriminating tests (three of which use only data already in the database and can each overturn a published conclusion), a five-stage pipeline built on a censored condition-aware target, and success criteria stated as prospective enrichment rather than retrospective AUROC. The remedy is already on disk and already validated elsewhere. Two supplementary workbooks in the local paper corpus hold ~267 enzymes and ~5,800 condition-indexed quantitative measurements with sequences attached, ~99 of them enzyme-level hard negatives verified as expressed and folded — including a **fully covered, balanced, single-condition slice of 216 enzymes, 102 active against 114 zero**. The authors of one of them report that models trained on a literature-scraped cross-study set score AUROC 0.47 on their assay, the same below-chance signature as our profile-HMM block, while **condition-specific models trained on the uniform assay reach about 0.8**. Nothing here retracts a null. It argues that no null so far was measured on a sample large enough, or against a target specific enough, to support the conclusion drawn from it.
• The execution plan that follows from it. (docs/discrimination-fix-plan.md.) Nine steps, each with a measurable gate and a stated consequence for failing it, written to be picked up cold in a new session: clear the instrument, run the two falsification tests nobody has run, ingest the two panel workbooks, **reproduce the field's published ~0.8 on a balanced 216-enzyme benchmark slice**, add detection-limit and quantitation-ceiling columns, fit the censored condition-aware target, finish the curation worklist, reopen the structural feature question, then integrate. Carries the operational constraints verbatim (read-only production, the 291 embargoed proteins, connection hygiene), the worker-pod recipe, the two steps that write to the database and must dry-run first, the measured limits that bound the design (only 26.7 % of stored values normalise; cross-lab reproducibility at matched conditions is 24–64×), and the two ingestion traps: 35 negative interactions carrying an off-target positive value, and ten detection methods that produce zero negatives because they are confirmatory.

Added

• **A curation worklist that makes the PET stringent-negative confound testable.** (docs/pet-negative-curation-worklist.md.) docs/phase0-pet-discrimination.md reports that the N-stringent PET negative tier is 21 of 23 negatives from one deep-sea study across three genera — a paper filter wearing a condition filter's clothes, with no constructible same-paper control. This document scopes what it would take to break that, read-only, against the production corpus and a 1 465-paper local store. **The finding is that the fix is almost entirely re-curation, not new literature.** Roughly 40 papers were read in full; the open literature contributed zero new stringent negatives and one unresolved candidate. Meanwhile four already-curated papers hold 18 stringent negatives that the rule cannot currently see:
  • 10.1038/s41467-022-35237-x (Erickson/NREL) — 15 negatives whose assay is a 6 pH × 5 temperature grid "spanning … temperatures from 30–70 °C", 96 h, amorphous Goodfellow film, UHPLC, zero product at every cell. They fail only the thermal axis because experiments.temperature is NULL: the grid has no scalar to record.
  • 10.1002/cbic.202500065 (Howe, Queen's) — TrmA (*Anaerolinea thermophila*, E8N6D9) and TrmH (Thermosporothrix hazakensis, A0A326UD21) were never extracted, though the paper names four PET-film negatives and PlasticDB holds two.
  • 10.1007/s00253-016-7992-8 (Ribitsch/Guebitz, BOKU) — PpEst (Pseudomonas pseudoalcaligenes), whose protocol runs at *"50, 65 or 80 °C … for 72 h"* on amorphous Goodfellow film; the extraction kept 50 °C.
  • 10.1021/acssuschemeng.6c00030 — a stringent negative (SfCut, Saccharopolyspora flava) already curated but outside the Phase 0 corpus because the protein has no sequence. Curating those moves the tier from 2 papers / 2 groups / 3 genera to 5 papers / 4 groups / 9 genera across 3 phyla, and drops the largest single study's share from 91 % to 51 % — which is what makes Cpap, the control the experiment could not build, constructible. The document also records why the literature cannot help much: **PET discovery screens are run below the glass transition.** The largest negative set published — 954 bacterial cutinases, only ten PET-active, titled *"PET hydrolysis is a rare property"* — is measured at 40 °C on >50 %-crystallinity powder by UV-Vis, and clears no axis at all. Every rejected paper is listed with its disqualifying axis so the work is not repeated. No database writes, no verdicts changed. This is a worklist, not a curation.

Fixed

• **A feature block that cannot be computed now stops the experiment instead of reporting a fabricated null.** (scripts/experiments/pet_discrimination.py, scripts/experiments/provenance_arms.py, scripts/experiments/negative_stringency.py, scripts/experiments/rescorer.py, tests/test_pet_discrimination.py.) MAFFT's portable tarball ships mafft.bat, a wrapper that exports MAFFT_BINARIES before calling mafftdir/bin/mafft. Invoke the inner script directly and it prints v0.000 != v7.526, exits 0, and then fails every alignment it is given. fold_profile_hmm_features treated that failure the way it treated every other: log a warning, leave the block's seven columns at 0.0, and return {"status": "ran"}. A block of zeros is not an absent row in a results table. It is scored, cross-validated and reported like any other feature, and it scores **AUROC exactly 0.500** — which is what a real null looks like, in the one experiment whose entire published claim is a null. The same silence covered a missing aligner, an hmmbuild failure, and a fold that could seed neither profile. Now: a requested block that cannot be computed raises BlockUnavailable. --allow-incomplete-blocks is required to degrade at all, and then the block is not_run, listed under the report's top-level incomplete_blocks, carried into every arm row of summary.json, and printed as a stderr banner — a missing block is easy to miss in a results table, so it is stated in every place a human looks rather than in JSON alone. The arm drivers refuse to start a 78-arm run with the aligner missing, instead of warning and spending two hours producing an incomparable arm set.
• **The pinned-toolchain guard accepted a MAFFT that could not align two sequences.** (app/services/bio_tools.py, scripts/check_bio_tools.py, tests/test_bio_tools.py, docs/bioinformatics-tools.md.) check_bio_tools.py asserted version_marker in output. The broken banner is v0.000 != v7.526 — the pinned marker is *inside the message reporting the fault* — so the substring matched and the check passed. There are three states, not two: absent, present at some version, and present and unable to work. Treating the third as the first is the actual defect; the first permits a caller to degrade, the third must not. BioTool gains version_disqualifiers (("v0.000",) for MAFFT), tested before the marker, and a misconfigured_hint naming the fix. The probe moves to app.services.bio_tools.probe_tool so the runtime can ask the same question of the binary it actually resolved: _tools() in every experiment harness now calls assert_tool_usable, and a --tool-dir pointing at the wrong MAFFT fails the run at once. A version mismatch still only warns — a host at another release is a host, not a defect. The image was never affected: its .deb hardcodes the helper directory and /usr/bin/mafft --version reports v7.526 with MAFFT_BINARIES unset. The affected runs were ad-hoc, on a host with an unpacked tarball, where a build-time-only guard never ran — which is why the runtime check exists as well as the fixed build-time one. Do not set MAFFT_BINARIES by hand: a packaged MAFFT requires it unset and breaks identically if it is set.
• **A profile-HMM fold that could seed only one of its two profiles said ran.** The per-fold status was also overwritten fold by fold, so one bad fold in five vanished behind the four that followed it. A fold now reports degraded naming the class and its seed count, and the block's status for the run is the worst fold's.

Changed

• docs/phase0-pet-discrimination.md gains a retroactive audit ("Was MAFFT working?") and corrects the environment-trap note, which claimed the harness marked the block not_run when it in fact marked it ran. Both published runs had a working MAFFT and no number is retracted: no arm scores the degenerate 0.500, the control distributions have real spread (hmm/forest sd 0.047 across 30 draws), every fold directory holds non-empty alignments and profiles, the run scripts invoke MAFFT through the mafft.bat wrapper, and the whole-corpus reference reproduces the 2026-08-26 baseline block for block. The audit does record a real degradation it found: 41 folds of the negative-stringency run built no negative profile because the arm had fewer than 8 training negatives — confined to Cpap/Cstr control draws the document already declines to read as results, and touching no published number.

Added

• **Negative stringency: the database can now say how hard a negative was pushed.** (docs/label-matrix.md → "Negative stringency", scripts/experiments/negative_stringency.py, tests/test_negative_stringency.py.) "Did not degrade PET" is a measurement made under conditions, and two hand-adjudicated cases show what that costs. Ahmed 2025 (doi:10.1002/mbo3.70144) records PET negatives by bulk UV absorbance with no stated limit of detection, and disclaims its own null in the text. Herrero Acero 2011 (doi:10.1021/ma200949p) records a PET negative for TfCut1 on 37 %-crystallinity film at 50 °C — an enzyme Roth 2014 and Wei 2020 report is PET-active. A negative is now scoped by five axes evaluated on a single experiment: real polymer (not a BHET/pNP surrogate), temperature at or above PET's hydrated glass transition (≥ 60 °C), an accessible substrate (≤ 15 % crystallinity or an amorphous film), chromatographic/MS detection, and ≥ 24 h. N-permissive is the polymer plus one accessibility axis; N-stringent is all five at once. Missing data never clears a bar. The rule was frozen from the polymer physics and the two adjudicated cases before any model was fitted. For PET: **160 negatives, of which 119 are permissive (12 papers) and 23 are stringent (2 papers)**. Positives are unaffected — an observed activity is evidence of ability whatever the conditions were. The corpus dump now carries the assay conditions (pet_discrimination.CORPUS_SQL_EXPERIMENTS), so the tiers are recomputed at every build and never stored on a row.

Changed

• **Phase 0 re-run on condition-scoped negatives: the PET null is not explained by what a negative means either.** (docs/phase0-pet-discrimination.md, new dated section; baselines kept verbatim.) Condition coverage was measured first, as the gate: on PET negatives, temperature is populated on 90.7 %, substrate form 99.4 %, method 100 % and crystallinity 59.6 % — enough to scope by, on both classes. (The four fields at 0 % — culture_medium, aerobicity, sole_carbon_source, substrate_mw — are whole-organism fields and every PET protein verdict is a cell-free assay.) 78 arms: full / permissive / stringent, plus 30 + 30 size-matched random control draws and 15 single-paper draws, all through pet_discrimination.evaluate unmodified. The full arm **reproduces the published whole-corpus reference exactly** (432 labelled, 272/160, best AUROC 0.641, every block to three decimals). For the first time in the programme both scoped arms clear their controls — permissive 0.709 against a control mean of 0.640 (sd 0.033, z = +2.10, 0/30 draws at or above); stringent 0.864 against 0.656 (sd 0.060, z = +3.49, 0/30). Neither survives its confound check. The permissive arm sits at the top of its control's leakage distribution (median train↔test identity 0.308 against a control range of 0.279–0.308, r = +0.31 between leakage and AUROC), and leakage-matched the effect falls to +0.028 AUROC, z = +0.92. The stringent arm is 21-of-23 one laboratory's deep-sea screen across three genera, and its negatives are **more distant from the positives than every one of 2 000 random size-matched subsets** (z = −3.21) — homology recognition, the artefact the 2026-08-26 run was built to expose. Its single-paper control could not be constructed: 13 of 15 same-paper draws have a training fold with no negative in it and cannot be fitted at all. Verdict unchanged: Layer C reports "polymer undetermined" for PET. What changed is the ranking of the open explanations — assay-condition noise is the only one of three that has produced any signal above its control, and the reason it cannot be credited is that this corpus cannot separate "measured hard" from "measured by one laboratory on one clade". Phases 1–3 were not re-run and the reason is recorded: Phase 1's seed set is the positives, Phase 2 consumes Phase 1, and Phase 3's N0–N3 tiers are Swiss-Prot and shuffles with corpus proteins explicitly excluded — not one measured negative among them.

Changed

• Phase 0 re-run on provenance arms: the PET null is not label noise. (scripts/experiments/provenance_arms.py, scripts/experiments/pet_discrimination.py, docs/phase0-pet-discrimination.md.) Phase 0 found nothing separating measured PET-positives from measured PET-negatives, and Phase 6.2's rescorer then failed to beat nearest-neighbour. Both were read as "this is biologically hard". v1.15.0's provenance audit offered a competing explanation: for **277 of 620 sequenced non-tombstoned proteins** the stored sequence is unverified against the organism the record credits it to, so a polymer verdict may be attached to a sequence nobody measured. That is an upper bound on possible label noise, not a measurement of it — most of those rows are probably fine. Tested by re-running the identical pipeline on four arms: A the 343 organism_consistent proteins; B thirty size-matched random draws (seeds 20260831–20260860); C the 277 unverified proteins; D ten clean draws size-matched to C. Same code, same split (centroid at 25% identity), same folds, same metric — only the corpus changes. **Arm A scores AUROC 0.663, the 70th percentile of its own random control (mean 0.644, sd 0.038); 9 of 30 draws score at or above it. Effect +0.019 AUROC, z = +0.51.** The sharper contrast is flatter still: the unverified arm scores 0.681 against a size-matched clean control mean of 0.679 — +0.002 AUROC, z = +0.08. Every arm sits at the same leakage (median train↔test identity 28%) and the same permutation floor. The whole-corpus reference run reproduces the 2026-08-26 verdict (best 0.641 against a 0.630 floor). One block moved and is recorded rather than buried: the profile HMM block is above all thirty control draws in the clean arm (0.660 vs a control mean of 0.548) and below the clean control in the unverified arm — consistent in direction, but one comparison of sixteen, with a bootstrap interval that includes chance and an MCC of 0.28. It licenses nothing. Layer C's "polymer undetermined" for PET stands. Curating the unverified rows is worth doing for correctness; it will not rescue PET discrimination.

Added

• --restrict-provenance, --restrict-ids and --report-out on pet_discrimination evaluate, so the same evaluation can be run over a subset of the corpus without a second implementation. restrict() slices the records and both identity matrices together — an excluded protein cannot bridge two clusters it is not part of. The corpus dump now carries sequence_provenance and accession_status.
• A staleness check on the ESM embedding cache. A cache written for an earlier, smaller corpus was previously a cache hit, and the proteins it did not cover were silently scored as zero vectors.

Added

• A protein duplicate-merge tool: the backfill half of sequence-level dedup. (app/services/protein_merge.py, scripts/merge_duplicate_proteins.py, new protein_merges audit table.) v1.14.0 closed the door going forward — a paper naming an enzyme whose residues are already stored folds onto that row at promotion. It deliberately did not touch what was already there: **29 groups of byte-identical mature chains covering 64 of 638 sequenced proteins, and 63 of those 64 rows carry interactions, experiments and observations no other member of their group carries**. Consolidating them is a union of evidence spines, which is this. Detection is not re-implemented: protein_dedup.duplicate_groups_sync — the same function scripts/report_duplicate_proteins.py reports from — finds the groups, and this plans and applies. One detector, two consumers. Union the evidence, never pick a survivor and drop the rest. Every row pointing at a retired record is re-homed onto the survivor: interactions (and with them the experiments and observations that hang off them by foreign key), variants, kinetics, properties, structures, enrichments, deposit membership, job hits, submission matches, reaction steps. All twelve tables are counted before and after and the group aborts if a single row went missing — EvidenceLostError, an assertion rather than a log line. A test reflects every foreign key to proteins.id out of the metadata and compares it against the hand-written referrer list, so a table added later fails the build rather than quietly not being re-homed. Tombstone, never delete. merged_into_accession + is_tombstoned, so a PB… from a published paper still resolves — 301 to the survivor, through the resolve_accession state machine that already existed for exactly this. The registry entry moves to merged. Because it is an UPDATE, the two #463 deletion guards are never met, let alone worked around; the pg tier proves both halves by merging a group containing a held row and then watching the very same guard refuse an actual delete of it. The survivor rule is a safety property, not a taste. Most restrictive visibility first (private < embargoed < staged < public), then richest evidence, then most complete accessions, then lowest id — the same tie-break SequenceIndex.match uses, so ingestion and the backfill agree about which row is canonical. Ordering restrictiveness first means the survivor is the resolved visibility, so the tool never writes visibility on it at all. That matters: a held row is parked at private, and the before_flush parking guard turns any write to a held row's visibility — including assigning private to a row that is already private — into an overwrite of visibility_hold_prior. The hold would survive; its reversibility would not. No visibility_hold_* column is written anywhere in the module. What it refuses, with the reason printed. Verdict conflicts (positive/negative on a shared polymer) and soft disagreements, both classified from the live spine rather than from an audit file; organism splits, which protein_dedup already calls a taxonomy question; deposit splits; visibility collapse — the most restrictive value demoting a member, a public row leaving the public corpus or a staged one leaving the release queue, off by default and reachable with --allow-visibility-collapse; and G09/G21 on a hard list keyed by accession, because their adjudications are specific and are not "merge". Repo backfill conventions. Dry run by default and `SET TRANSACTION READ ONLY while it is one; --apply` to write; one group per transaction through commit_with_deadlock_retry (the backfill_signal_peptides lesson: a long transaction deadlocked against the scheduled reindex_search_vectors bulk UPDATE of proteins and lost a production run at record 140 of 577); SIGINT/SIGTERM stop between groups; resumability is a property of the query — tombstoned rows drop out of the detector, so re-running finds nothing, which is also the idempotence test.
• protein_merges — the audit row #463 wished it had ("no tombstone, no merged_into_accession, no audit row — the loss had to be reconstructed from row counts"). One row per retired record: which accession became which, in which sequence frame, and how many rows moved in each table. A new table, so it ships through the ORM models and python -m app.db.init_schema; no Alembic revision, because a migration must never op.create_table() something the models also declare.

Added

• **The database now records what each stored sequence is, and whether the accession it hangs on is still alive upstream.** Two new proteins columns (accession_status, sequence_provenance), a classifier (app/services/provenance.py), a worker that keeps them current (app/workers/provenance.py), and a one-time catch-up (scripts/backfill_sequence_provenance.py). Migration 0015. Two hand adjudications of contradictory PET verdicts both dissolved into provenance rather than biology. PBUU2H (WP_108898452.1) carries a PET-positive for Bacillus thermoamylovorans JQ3, but the accession came from the paper's own supplementary table under the header Nr_ID — an NCBI nr best hit. The authors deposited SRA reads and never JQ3's protein, so the stored sequence is an annotation transfer onto a different isolate, and that accession is also suppressed upstream. PB5BTT is named "Tfcut2" and carries E5BBQ2, which is TfCut1 and is now deleted outright; the real TfCut2 is E5BBQ3, merged into Q6A0I4. Nothing in the schema could tell an authoritative deposit from a homology guess, and everything downstream inherits the difference — the profile-HMM pipeline builds on these sequences and the label matrix attaches polymer verdicts to them. What is measured, and what is not. Full provenance is not reconstructible: nothing records whether a curator read an accession off a methods section or inferred it, and no query invents that. What is computable is consistency — what upstream says about the handle, and whether upstream's organism agrees with the row's. Both vocabularies therefore keep an explicit unverified, it is the default for all 867 existing rows, and a failed lookup leaves it rather than fabricating a verdict. accession_status: unverified · absent · active · secondary (merged upstream) · deleted (deleted or suppressed) · unresolved. sequence_provenance: unverified · no_sequence · unaccessioned · organism_consistent · species_projection (a RefSeq WP_ — the non-redundant namespace, so it belongs to no single isolate whatever the species says) · strain_mismatch · organism_mismatch · uncomparable. Two columns, not one, because they are independent facts and each named case proves it from a different side: a live, primary accession can still name a different organism than its row, a dead one may have named exactly the right one, and WP_108898452.1 is defective on both axes at once. A single column would have to pick which half to report. Measured against production, read-only (867 live proteins): organism_consistent 357, no_sequence 229, unaccessioned 121, species_projection 100, uncomparable 34, organism_mismatch 26, strain_mismatch 0; and active 454, absent 350, deleted 61, secondary 2.
• **Forward-looking capture, following the #469 pattern rather than inventing a third.** Both approval routes dispatch classify_protein_provenance for the proteins the promotion created, after the commit — never inside the request, because a lookup is seconds per record and an approval can mint 46 at once. A daily sweep-protein-provenance beat (05:50 UTC) is the net behind it. It is deliberately not a rung of the weekly enrichment ladder: that batch walks every protein in id order under a four-hour limit, so the newest rows are the last it reaches — the same reason a NULL has_signal_peptide survived months of weekly runs.
• app.services.external_api.eutils_get_sync — the sync twin of eutils_get, so Celery and the backfill scripts pace, identify and retry against NCBI through the same one place the async callers do, instead of adding a second bare client.get. Lookups are batched at 100 accessions per request against both UniProt's /uniprotkb/accessions and NCBI esummary, and deduplicated across a batch.

Fixed

• eutils refusals are no longer under-counted: esummary emits one "error" key per refused id in the same JSON object and every parser keeps only the last, so refusals are now read from the raw body. And an "Invalid uid" is no longer taken as "no such protein" — 31 of this corpus's accessions are MAG/WGS proteins that esummary and esearch both refuse while efetch serves them in full.

Changed

• **A protein's identity is now its accession, then its sequence, then — within one paper only — its name. Global name matching is gone.** (app/services/ingestion_promotion.py, new app/services/protein_dedup.py.) An audit of production found **29 groups of byte-identical mature chains covering 64 of 638 sequenced proteins (~10% of the corpus)** — the same enzyme stored two, three or four times, with its papers and evidence split between the copies. Accession matching was not broken, it was one-dimensional: UniProt legitimately holds several accessions for one chain (one group is four rows, one 261-aa chain, three UniProt accessions), so **15 of the 29 groups share no accession value in either column** and no widening of the accession lookup could ever have joined them. Promotion now compares the canonical sequence, reusing the normalisation app/services/uniparc.py already defines (whitespace stripped, upper-cased, trailing * dropped, None for anything that is not a protein sequence) and keying an in-memory index on its CRC64. A digest is never the verdict: the full canonical strings are compared before a match is returned. A second paper's interactions, experiments and observations now land on the protein that already holds those residues. What it catches and what it does not. It compares the stored precursor, which is all it has — SignalP runs after promotion commits — so it reaches 12 of the 15 accession-distinct groups. The other three (audit groups G09, G15, G22) are identical only after the signal-peptide cut and are invisible to it by construction. scripts/report_duplicate_proteins.py --frame both finds those, read-only, over rows SignalP has since annotated.
• **The global name fallback is removed, and the measurement is the argument for what replaced it.** It matched Protein.name == name with limit(1) against the whole table with no regard for the organism: PHB depolymerase appears in 8 approved production ingestions naming 8 different species, and all 8 resolved to whichever row came back first. But 229 of 867 production proteins carry neither an accession nor a sequence, so the name was genuinely their only handle, and dropping it outright is not free. Measured over the 1,722 protein actor slots in the stored payloads: | | slots | |---|---:| | accession-bearing (never reached the fallback) | 974 | | no accession, a sequence supplied (now covered by the sequence check) | 199 | | no accession and no sequence | 549 | Of the 510 bare slots in approved payloads, only 291 are distinct (ingestion, name) pairs — 219 are one paper naming one enzyme again. Dropping the fallback with nothing in its place would have split a single study's enzyme into several rows, which is worse than the cross-paper duplication being fixed. So two narrow replacements keep exactly that:
  • a payload-local actor cache, so one paper's repeated name is one actor; and
  • a paper-scoped name check (interactions.paper_id), so a re-approval finds its own earlier work — the only handle a bare actor has, and the reason re-promotion stays idempotent for the 291 pairs. Neither can reach another paper's row. **The cross-paper cost is 23 bare names used by more than one approved ingestion, 52 arrivals that previously folded onto an existing row and will now mint their own.** Many of those folds were wrong (the PHB depolymerase case above), so this is not purely a loss — but it is a real change in behaviour and it is stated rather than buried.
• A sequence match with an organism mismatch is refused, not resolved. Production holds one 355-aa nylon hydrolase stored under both Arthrobacter sp. and Paenarthrobacter ureafaciens. Which name is right is a curator's judgement, and settling it on approval would make the answer depend on which paper was reviewed first. Promotion keeps both rows and records a DedupNote(organism_mismatch) on the PromotionResult, also logged at WARNING. The comparison is on tax_id, so two spellings of one NCBI node are not a mismatch.
• **Metadata on a reused row: the stored record wins, and nothing is thrown away.** A null uniprot_accession/genbank_id is filled from the incoming record; a different value is recorded under cross_references.alternate_uniprot_accessions rather than substituted (structures and enrichment records are keyed on the column). sequence is never touched — ruling R-1 makes it the deposited record verbatim, and a match means the two already agree.

Added

• proteins.synonyms (Alembic 0014_protein_synonyms), modelled on microorganisms.synonyms down to the FTS weight 'B' and the to_jsonb(row)->>'synonyms' trigger idiom. Sequence dedup made it necessary rather than nice: one 261-aa Thermobifida fusca chain is published as TfH, TfCut2, TFH and TfC, and folding four spellings onto one row while keeping one of them would delete three terms a reader of those papers will type. Promotion appends and never renames the row. It is deliberately not an identity key — nothing resolves to a protein through it, so a wrong synonym costs a spurious search hit and never a merged record.
• scripts/report_duplicate_proteins.py — read-only detector for duplicate groups in either frame. There is no --apply: it has no write path, opens the session READ ONLY, and disposes its engine in a finally.

Fixed

• Nothing about visibility. Stated because it is the constraint: promotion writes neither visibility nor any visibility_hold_* column on a reused row, in any branch. Six of the 29 groups straddle the embargo, so a public paper's record folding onto one of the 291 held proteins is now an ordinary event — tests/test_pg_protein_dedup.py pins that the held row stays held, that its hold columns are byte-identical afterwards, and that query.public_visibility_filter still excludes it.

Not done

• The existing 29 groups are not merged. Detecting them is in scope; consolidating them is not — 63 of the 64 rows carry evidence no other member of their group carries, so a merge has to union two evidence spines, re-home plasticdb_ids that are citable, and decide six embargo straddles. That needs its own tool and its own review. This change is forward-looking only.

Changed

• The release snapshot streams instead of materialising the database (#271). generate_snapshot held the whole corpus in memory roughly three times over — every ORM entity, a second copy as flat row dicts, and a third as one json.dumps string — and protein and variant sequence columns were resident in all three. That was tolerable while the snapshot ran monthly; since v1.9.0 it runs weekly and unattended inside weekly_dataset_release, in a worker pod that also runs DIAMOND alignments, where an OOM takes the release with it. Every entity is now read through a server-side cursor in batches of STREAM_BATCH_SIZE (250, the same bound /api/v1/downloads uses) and written straight into its file. One pass per entity feeds the CSV, the snapshot.json fragment and — for proteins — the FASTA, so the tarball still costs the same six SELECTs. dataset_fingerprint streams the same way. On a corpus the shape of production (867 proteins, 1308 microorganisms, 6526 interactions, 17323 experiments, 477 papers), peak Python allocation drops 56.5 MB → 15.7 MB and process RSS above baseline 117 MB → 37 MB. The point is the slope, not the number: at four times that corpus the buffered implementation peaks at 214.0 MB (RSS +450 MB) while the streaming one peaks at 17.2 MB (RSS +41 MB) — 3.8× versus 1.1×. dataset_fingerprint moves the same way, 56.5 MB → 16.4 MB. The artefact is byte-identical. Every member of the tarball — all six CSVs, sequences.fasta, snapshot.json, README.md — matches the buffered implementation's output byte for byte on the same input, and dataset_fingerprint returns the same digest, so this release does not mint a DOI for a dataset nobody edited. snapshot.json is reassembled to the exact shape json.dumps(..., indent=2) produced, empty-list rendering included. The visibility filters are untouched. public_visibility_filter, variant_parent_public_filter and public_actor_interaction_filter are applied by the same statements, now collected in one _ENTITY_SPECS table that the CSVs, snapshot.json and the fingerprint all read from, so there is one place a row set can be defined. New pg tests pin this down by row set rather than by count: test_streamed_snapshot_is_byte_identical_to_the_buffered_one runs a transcription of the old implementation alongside the new one and compares bytes, and test_streamed_snapshot_excludes_staged_embargoed_and_private_together seeds embargoed, private, staged and tombstoned records in one corpus — the case where a predicate lost between batches leaks some rows rather than all. Note for anyone streaming ORM rows elsewhere. Result.unique() is not used: over a streamed psycopg2 result it silently drops rows — the ORM's uniquing key for a mapped entity is id(obj), and a streamed result lets CPython reuse the addresses of collected batches, so later rows collide with dead ones. Measured here it returned 724–1975 of 3468 proteins, differing run to run; the download routes' asyncpg path did not reproduce it, so it is a hazard to avoid rather than a defect to fix there. Uniquing is preserved by comparing against the previous primary key, which every statement orders by, at O(1) memory, and test_every_streamed_statement_eager_loads_with_selectin fails the build if an eager load ever becomes a joinedload, which is the one thing that could make a parent row arrive twice. Nor is the batch size passed as a yield_per execution option, though it is the same mechanism. Doing so records it on the top-level QueryContext, and orm_pre_session_exec merges that context's execution options into every post-load query selectinload issues — so the sub-load inherits yield_per and then trips over the .unique() it calls on itself, `Can't use the ORM yield_per feature in conjunction with unique()`. That leak is only reachable when a do_orm_execute listener is registered on Session, because only then does Session._execute_internal run orm_pre_session_exec twice and feed the first pass's merged options into the second (SQLAlchemy #13301, fixed after the 2.0.35 pinned here). v1.13.0 added exactly such a listener — _refuse_bulk_deletion_of_held_records, the guard that stops a bulk DELETE destroying an embargoed row — and it took every generate_snapshot test in the pg tier down the moment the two met. Narrowing the listener does not help: it already returns immediately for anything that is not a delete, and the registration alone is what changes the execution path, so the only way to fix it there would be to delete the guard. stream_results plus Result.yield_per batches identically — measured side by side on one corpus the two shapes peak at the same figure to the decimal, at both scales — and never reaches a sub-load. TestStreamingSurvivesTheDeletionGuard pins it down in the mock tier, where a branch carrying only one of the two changes can still see it.

Fixed

• A newly promoted protein no longer lands with a NULL signal peptide. The 2026-08-29 approval minted 61 proteins with usable sequences and every one of them had has_signal_peptide = NULL. That is not cosmetic: the profile-HMM pipeline clusters on the mature chain when a signal peptide is known, so those 61 were clustered as precursors against 577 incumbents clustered as mature chains — a representation mismatch nothing reported, which invalidated a Phase 1 run. scripts/backfill_signal_peptides.py closed the gap by hand (391 True / 247 False / 0 NULL); this closes it for good. New app/workers/signal_peptides.py, reached two ways:
  • On promotion. The curator approve route and the admin accept route each dispatch predict_signal_peptides for the proteins that promotion created, after the commit — the worker is a separate process and cannot see a flushed-but-uncommitted row. Not inline: the default predictor is EBI's hosted Phobius at ~12s per sequence, so a 61-protein approval would have been a twelve-minute HTTP request.
  • On a schedule. New daily sweep-signal-peptides beat task (05:35 UTC) predicts for a bounded batch (SIGNALP_SWEEP_LIMIT, default 40) of rows that still have none, newest first — the opposite of the weekly enrich-all-proteins batch, which walks every protein in id order under a four-hour soft time limit and therefore reaches a freshly minted row last, if at all. That is part of why the gap persisted. Three rules the implementation holds, each with tests:
  • Both columns, or neither. signal_peptide_cleavage_site is what sequence_frames trims on and what the reference FASTA is built from, so the pair is written in one place. A positive call whose cleavage site could not be parsed is booked as a legible permanent failure rather than stored as True with a NULL site.
  • Failure is never written as False. A failed prediction leaves both columns NULL and records why on the protein's ProteinEnrichment (source='signalp') row, which the admin enrichment coverage table already shows. Transient failures (the two EBI 500s the backfill hit) are retried by the task and then by the sweep; a row that reaches the retry cap or is terminally unpredictable is named in a Slack line, so a protein nothing will retry is visible rather than merely absent.
  • **One protein, one short transaction, and no transaction open across a network call.** The batch form of this deadlocked in production against the scheduled reindex_search_vectors bulk UPDATE and killed a run at record 140 of 577; writes go through commit_with_deadlock_retry. Filling a staged record does not invalidate the public DIAMOND databases: reference_build.current_fingerprint hashes only visibility='public' rows and promotion creates staged ones, so approval triggers no rebuild. Pinned in the pg tier, with a control proving the fingerprint does move for a published row.

Fixed

• **A record under a visibility hold can no longer be deleted — by the supersede path or by anything else (#463).** On 2026-08-29 a bulk approval of the pending review queue moved six ingestions to superseded, and five proteins from a deliberate 296-row embargo (bulk_negative_results, batch 3ace0ee3-…) went with them. The batch is 291. They were not tombstoned, not released, carry no merged_into_accession, and left no audit row: the loss had to be reconstructed from row counts and id ranges. The mechanism was indirect, which is why no review caught it. ingestion_supersede.retire deletes the evidence a retired revision stamped (delete_prior_ingestion_rows), then runs the global orphan sweep (delete_orphaned_organisms_and_proteins), whose entire test for "this row is rubbish" is referenced by nothing. For an embargoed protein whose interactions have just been retired, that test is satisfied and wrong. Nothing anywhere asked whether the row was held. Three layers now, each independently load-bearing:
  • app/services/orphan_cleanup.py never sweeps a held row. Held orphans are left standing and reported back under held_proteins / held_microorganisms, so the superuser paper delete and the ingestion withdrawal keep working and simply leave the embargoed record in place. A ProteinVariant whose parent is held is spared with it — a variant carries no hold of its own and is gated on its parent's visibility, so deleting it would destroy part of the held record while leaving the hold looking honoured.
  • ingestion_supersede.retire refuses the whole correction (SupersedeRefused → 409) when the sweep had to spare something. It aborts rather than reporting: the retire runs inside the approve route's single transaction and commits nothing, so the raise rolls the promotion, the deletes, the re-stamps and the status move back together and the paper returns to the queue. Proceeding would have spared the protein row while keeping the deletion of every interaction, experiment and observation behind it — an embargoed husk with nothing left to publish when the embargo lifts, which is a quieter loss rather than a smaller one. The 409 names the accessions so the curator can release the hold or re-assert the record.
  • A guard at the flush boundary, beside the existing parking guard in app/models/models.py and keyed on VisibilityHoldMixin the same way, so a third holdable entity is covered the day it declares the mixin. Two listeners, because the two deletion idioms take different routes: before_flush for session.delete(obj), do_orm_execute for db.execute(delete(Model)…) — which never enters the unit of work and is the statement that actually destroyed the five rows. HeldRecordDeletionError answers 409 through a new handler in app/main.py, so any door nobody has found yet refuses loudly instead of deleting quietly. There is no opt-out flag: the sanctioned way through is to release the hold first, and an exemption keyword would be another door with a nicer name. Also closes the laundering variant noted in the issue — a superseding revision could re-create equivalent content as a fresh staged row carrying no hold, which the next weekly release would publish. It did not happen in this incident (verified: no staged row shares a sequence or accession with any held protein), but nothing prevented it; the record can no longer be destroyed for a replacement to take its place.

Changed

• The supersede now records what it destroyed. retire returns an identifier for every protein and organism the orphan sweep deleted, and record_paper_superseded stores them in the paper_superseded contribution event under rows_removed.deleted_entities. A deleted row leaves nothing behind — that is what deletion is — and the absence of any such trail is why #463 had to be reconstructed from counts. ContributionEvent.metadata_ is JSON, so this is additive: no new table, no migration. **One entry per deleted row, always: the accession where there is one, otherwise #<id>.** Microorganism.accession is nullable (only Protein.accession is NOT NULL), so an accession-keyed trail would have been complete only as long as the data happened to be tidy — the same shape of assumption that produced the bug above. The numeric id is also the better forensic handle: id ranges are what pinned down the original incident once the accessions were already gone. The 409 refusal uses the same fallback, so a curator blocked by an accession-less record is given something they can act on rather than an empty string.

Fixed

• **--cut_nc was unsafe on M0001: the library shipped a -1000000.00 noise cutoff for a value nothing measured.** M0001, the nylon-amidase merge, has no definable NC — not one of its 5 324 N3 hard negatives scores above the reporting floor, so there is no highest-scoring non-member to read a cutoff from. THRESHOLDS.md printed — for it; the .hmm printed NC -1000000.00 -1000000.00;, because gathering_lines clamped an undefined cutoff to a floor. HMMER does not treat that as "undefined". It reads it as a noise cutoff every sequence on earth clears, so hmmsearch --cut_nc against the shipped library returned hits --cut_ga rejected, silently, and the ambiguity was visible only inside the model file: | M0001 vs | --cut_ga | --cut_nc (before) | --cut_nc (after) | |---|---:|---:|---| | N1 background | 0 | 3 | exit 1 | | N2 ecological | 0 | 6 | exit 1 | | N3 hard | 2 | 684 | exit 1 | | the 150 curated negatives | 0 | 2 | exit 1 | 684 is exactly the PF01425 relatives N3 excludes by construction — the tier inversion docs/phase3-calibration.md already documents, arriving through a threshold instead of through a report. A cutoff that was not measured is now written as nothing at all, for all three columns rather than NC alone. Measured against the pinned HMMER 3.4 before choosing it, for hmmsearch and hmmscan alike: with the line absent, hmmpress succeeds, --cut_ga and --cut_tc are unaffected, and --cut_nc exits non-zero and returns nothing — on the worker pod with the message NC bit thresholds unavailable on model M0001, though the exit code is the contract and the sentence is not: the 3.4+dfsg-2 build in the CI image prints a bare Error: with the detail missing. The cost is that HMMER aborts at the offending model, so --cut_nc over the library now fails for all six profiles rather than for M0001 alone — which is the right trade against a search that answers a question nobody could have answered. hmmfetch still gets you any single profile. --cut_ga and --cut_tc are unchanged for every profile against every tier: 70 model × query-set cells (six profiles + the whole library, against the seeds, the 150 curated negatives, N1, N2 and N3) re-measured before and after, zero moved. Five of the six calibrated profiles come out byte-identical; only hmm/M0001.hmm moves. insert_gathering_thresholds now checks its own output and refuses to return a model carrying a cutoff that was not measured, so this cannot be reintroduced by a formatter edit. make hmm-smoke re-measures HMMER's real behaviour in CI, because omitting the line is only safe while the toolchain keeps refusing rather than defaulting. Republished at /app/bioinformatics/phase3-2026-08-30/ (PlasticDB.calibrated.hmm, its .h3i/.h3m/.h3p press files, hmm/M0001.hmm, calibration.json, THRESHOLDS.md) with SHA256SUMS regenerated and verified; every other file in the artefact is untouched.

Changed

• **Phase 3 of the profile-HMM pipeline re-run against Phase 2's six profiles — and the answer is no.** Consumes /app/bioinformatics/phase2-2026-08-30/; artefacts published to /app/bioinformatics/phase3-2026-08-30/ (40 MB, including the UniProt cache so the negatives cannot drift again). Full write-up in docs/phase3-calibration.md, which keeps every baseline number alongside the new one. **The 638-protein corpus did not improve discrimination. Of the four profiles that existed at the 2026-08-27 baseline, two are unchanged to the last decimal place and the other two got worse.** | the same biological family | baseline GA / TC / NC | now | NC→TC gap | |---|---|---|---| | the big polyesterase family | F0001 137.00 / 220.60 / 53.40 | 111.30 / 163.20 / 59.40 | 167.20 → 103.80 (−38%) | | the second polyesterase family | F0002 92.80 / 161.30 / 24.40 | unchanged | 136.90 → 136.90 | | the PHB depolymerases | F0003 174.90 / 270.40 / 79.50 | 173.60 / 264.80 / 82.40 | 190.90 → 182.40 | | the nylon amidases | M0001 389.10 / 389.60 / — | unchanged | still untestable | | Lip_C (PF24708) | — | F0004 299.80 / 553.50 / 46.00 | 507.50 (new) | | the carboxylesterases (PF00135) | — (rejected at 3 seeds) | M0002 192.90 / 328.60 / 58.90 | 269.70 (new) | All six profiles recover 100% of their own members at GA. The whole gain is the two new profiles; nothing existing improved on any metric. This is the third phase in a row to return a null on the grown corpus, and the first where the direction is negative rather than flat.
• F0001's 63-bit loss is the partition, not the new proteins. Its TC is set by PBE66Y APET-5 at 163.2 bits — a protein that was in the 577-corpus already, in a family that shipped no profile. Of the 13 members F0001 gained, six came from baseline F0008, one from baseline F0006 and six are new arrivals, and the three weakest members of today's family are all in the first two groups. The re-clustering widened the family boundary, which is exactly what costs a membership threshold its margin. Nothing was tuned to recover it.
• **The per-profile N3 exclusion is measured, and it is the number Phase 4 needs.** "Same superfamily, different family" removes a family's own Pfam relatives from its hard tier — correctly — so a 0.00% FPR against N3 is a statement about N3 and not about what a screen returns. Own-Pfam relatives removed / how many clear the GA fitted without them: F0001 2/2, F0002 65/56, F0003 18/0, F0004 20/20, M0001 684/2, M0002 163/130. F0003 is the only profile whose GA sits above every excluded relative.
• **M0002 is the first profile in the project with a non-zero false-positive rate**: N1 0.12% — twelve mammalian carboxylesterases, cholinesterases and neuroligins, the best at 370.9 bits, above M0002's own TC of 328.60. All are PF00135, all were removed from N3 by construction, and N1 is unfiltered. It is also the first profile whose cross-validation disagreed with itself (MCC 0.975 ± 0.035 over 3 folds, against 1.000 ± 0.000 everywhere else). --cut_ga on M0002 is a correct membership call and needs a report-level guard before it becomes an activity claim.
• **§3.4 recomputed on the grown label matrix: Tier 1 = 4 (unchanged), Tier 2 = 11, Tier 3 = 38, and no polymer earns a calibrated confidence.** PET grew from 264/151 to 284/164 pairs and stays where it was — tier cleared, discrimination gate failed. Nylon 6 still misses Tier 1 by exactly one negative pair (24 positive, 9 negative — the baseline's two numbers); 61 new proteins contributed none.
• F0004 was calibrated against N0–N3 and never against its PET labels. Its N3 correctly excludes PF24708 Lip_C and the merges resolve to their constituent families, both by the existing code. Its MCC is flagged as a resubstitution score: all 16 members fall into one identity group at 40%, so no folds exist and its leakage is not measurable. Reported as such rather than as a clean 1.000.

Fixed

• (reported, not fixed — no code changed) **--cut_nc is unsafe on an untestable profile.** M0001's undefined NC is written into the model as the -1000000.00 sentinel and HMMER reads it as a real threshold: hmmsearch --cut_nc M0001.hmm over the 3 487-sequence N2 tier returns **6 hits where --cut_ga returns 0**. THRESHOLDS.md prints — so the human artefact is correct; the .hmm a tool reads is not. Phase 4 must use --cut_ga only until the NC line is omitted for such profiles.
• (curation, reported only — nothing written) **29 groups of byte-identical mature chains cover 64 of the 638 protein rows**, 7 of them disagreeing about at least one polymer verdict and **2 carrying an outright positive/negative conflict on PET**: Phase 2's PBUU2H Lip4_120 / PBTXZS Ct_Est, and a pair Phase 2 did not see — PB8XKW TfCut1 / PB5BTT Tfcut2, two rows named for enzymes the literature treats as different proteins holding the same 261-residue mature chain with opposite PET verdicts. That one reads as a data-entry fault. None of it contaminates a threshold: TC is a minimum over identical-scoring chains, _corpus_exclusions keeps every corpus sequence out of every negative tier (606 sequences, 231 accessions), each conflicting pair collapses onto one 90%-dedup representative so none straddles a fold boundary, and none of the eight accessions is among the 155 curated all-negative proteins.

Notes

• Determinism checked rather than asserted: the whole phase was run twice into separate directories from the same cache. Identical tier digests, identical fingerprint, identical per-profile thresholds, and a byte-identical PlasticDB.calibrated.hmm (sha256:ea8bb78acf35fed4…).
• -Z/--domZ are not passed, in this run or the baseline, and should not be: every threshold here is a bit score, and bit scores do not depend on database size. -Z rescales E-values only and would change no number in the document.
• Read-only throughout. One database connection, opened by export_label_matrix fetch on a readonly=True psycopg2 session and closed in a finally. Verified afterwards: 291 private proteins, all visibility_hold_reason = 'bulk_negative_results' — unchanged — 466 public, 110 staged, 638 sequenced wild types, 0 sessions idle in transaction.

Changed

• **Phase 2 of the profile-HMM pipeline re-run on the corrected 638-protein partition — six profiles, not four.** Consumes /app/bioinformatics/phase1-2026-08-30/ (the mature-chain artefacts, not the 08-29 precursors); artefacts published to /app/bioinformatics/phase2-2026-08-30/. Full write-up in docs/phase2-profile-construction.md, which keeps every baseline number alongside the new one. 200 of 350 deduplicated seeds are inside a profile — 57.1%, against the 2026-08-27 baseline's four profiles over 173 of 310 (55.8%): | profile | seeds | TCS | M | eff_nseq | trim | vs baseline | |---|---:|---:|---:|---:|---:|---| | F0001 | 137 | 90 | 258 | 1.57 | 0.0% | +7 seeds, same model length | | F0002 | 18 | 80 | 178 | 1.36 | 0.0% | unchanged in every number | | M0001 (Amidase merge) | 15 | 82 | 434 | 1.80 | 1.6% | unchanged in every number | | F0003 | 11 | 83 | 217 | 1.25 | 0.0% | +1 seed, one column shorter | | F0004 (PF24708 Lip_C) | 10 | 93 | 382 | 0.73 | 0.0% | new | | M0002 (PF00135 COesterase merge) | 9 | 85 | 492 | 1.39 | 0.0% | new — the baseline rejected this group at 3 seeds | Every profile recovers 100% of its own seeds with itself best-scoring, no in-family seed is missed (317 of 317), and **106 curated proteins are still reachable only through the non-profile phmmer/DIAMOND route** — 105 in the baseline, unmoved by 61 new proteins. Half the corpus is still the tail. The honest summary: **an 11% larger corpus improved no existing profile and added two new ones**, and more of the coverage gain comes from a merge that crossed a threshold than from anything the corpus discovered.
• **The §2.3 TCS gate fired on nothing again — and F0004, the family it was most expected to catch, scored the highest of all six (93 against a gate of 50).** F0004 is the first gate-clearing family carrying its own measured negatives: nine PET-negative members against two PET-positive ones. It is an excellent homology profile — zero columns trimmed, only two cross-family hits — and it supplies no discriminative signal whatsoever:
  • 0 of 408 alignment columns separate the nine negatives from the two positives (200 random 9/2 splits of the same sequences: median 0, 95th percentile 2, max 8);
  • pairwise identity negative-to-negative and negative-to-positive have the same median, 47.4%;
  • nine profiles built from eight negatives each score the PET-positive members at the top of the held-out-negative range, not below it;
  • one PET-positive member, PBUU2H Lip4_120, is a **byte-identical 372-residue mature chain** to the PET-negative PBTXZS Ct_Est — two curations of the same enzyme, from two papers, disagreeing about PET. The 90% dedup therefore collapses it onto the negative side, leaving a seed-level contrast of eight negatives against one positive. Phase 3's N3 tier and Phase 6's rescorer must not be pointed at this family. What it can honestly support is §3.1's specificity report, not a cutoff. (Two further identical pairs — PBKNV6/PB4VN7 and PBMKXJ/PBFAGC — carry conflicting PET verdicts too. Recorded for the owner; nothing was written.)
• --strip-signal-peptide re-tested end to end and kept as the default. Phase 1's A/B said the mature-chain rule was costing a profile-eligible family (5 gate passes without the strip, 4 with). Carried all the way through Phase 2, the two arms give the same six profiles and the same coverage — 200/350 (57.1%) stripped, 201/351 (57.3%) not — because §2.0's trim-fragmentation repair puts the nylon amidases back together as fifteen seeds, three more than the untrimmed arm ever had. Four of the six profiles are identical between the arms; the two that differ both favour the mature chain (+2 TCS each, less end-trimming). Phase 1 was counting gate passes one stage too early.
• **The trim-fragmentation repair is now worth 6.8 points of seed coverage and two of the six profiles** (50.3% → 57.1%), against 4.8 points and one profile on the baseline corpus. It found four candidate groups and admitted two; both rejections are recorded with the number that failed them and their constituent families restored to the tail.
• Determinism re-verified rather than assumed. The whole build was run a second time into a separate directory: all six profile digests, all six TCS scores and the concatenated library are identical — sha256:3ac9e874e7b35d802a30d6bd304e530661d8aa97a96bae0fcd2f081092953478.
• docs/hmm-pipeline-plan.md §2 and §2.0 carry the re-run's status block and the updated repair figure; the baseline's block is kept unedited above it.

No code changed. This release is measurement and documentation: the same scripts/build_family_profiles.py and app/services/hmm_profiles.py the baseline ran, byte-identical, against a larger corpus. **Nothing was written to the database** — Phase 2 reads no database at all. Verified after the run: 291 private proteins, all visibility_hold_reason = 'bulk_negative_results', unchanged; 466 public / 110 staged; 0 sessions left idle in transaction.

Changed

• **Phase 1 re-run a second time, on the corrected corpus, and the 2026-08-29 partition turns out to have been right.** The 1.11.2 re-run flagged a defect in its own inputs: all 61 newly approved proteins carried has_signal_peptide = NULL, so they were clustered as precursors while the 577 incumbents were clustered as mature chains, and it could not say how much of its +14 families was that artefact. The SignalP backfill has since run against production — §1.1 acceptance goes 577/638 = 90.4% → 638/638 = 100%, 391 true / 247 false / 0 NULL, 0 true-with-no-cleavage-site — and Phase 1 was repeated with the same code, tools, panel and parameters. The two corpus dumps differ in exactly 61 rows and only in their signal-peptide columns: one variable. The corrected partition is the uncorrected partition. Compared as a set of member blocks the two are identical — 108 families, same membership, zero splits, zero merges, zero proteins changing family; the 90%-dedup grouping matches too. Seeds 483, post-dedup 350, gate passes 4, seed coverage 50.3%, largest family 137 (39.1%), 78 singletons (72%). **Of the +14 families the larger corpus produced, 0 were the precursor artefact and 14 were real**, and the fourth gate pass is real. Everything 1.11.2 rested on survives: F0004 still clears n ≥ 8 at 10 seeds / 16 members on PF24708 Lip_C, still with nine PBAT/PBSA-positive, PET-negative members — still the only gate-clearing family with a large internal negative set. F0012 (RoxA/RoxB rubber oxygenases, 5 seeds) and F0017 (Lcp, 4 seeds) both still exist at the same size and still miss the gate.
• **Catalytic-domain detection re-measured on properly trimmed mature chains: no movement whatsoever.** 584 / 638 (91.5%), the same 54 undetected proteins protein for protein, and not one of the 584 changed its best-scoring panel profile. The arrivals still detect at 47/61 (77.0%) against the incumbents' unchanged 537/577 (93.1%), so the 1.6-point fall from the baseline is a panel-coverage gap in the new material and not a trimming artefact. The accessory FASTA is byte-identical to the 08-29 run's.
• Why the artefact was worth nothing, measured. The signal-peptide cut and the domain-envelope cut are both N-terminal and the envelope is nearly always deeper: for 552 of 638 chains it already begins after the cleavage site, so stripping first is a no-op. 40 proteins gained a signal peptide but only **17 of 483 seed sequences changed**, 11 of them by 1–6 residues at an envelope boundary. The six RoxA/RoxB chains lost 26–34 residues each — the panel finds no domain in them, so the strip is their only trim — and F0012 did not move. A control export leaving all 391 signal peptides on does distort the partition (351 dedup seeds, 107 families, 5 gate passes, 54.7% coverage), so the mature-chain rule still earns its place; 61 untrimmed rows was simply too few to bend a 638-protein clustering.
• Determinism control repeated: four consecutive cluster runs on identical inputs give a byte-identical families.json, md5 65970c68aad56c1ac3d16da846e1f2ee, 108 families and 4 gate passes every time. The canonicalise_fasta fix from v0.95.3 still holds on the corrected corpus.
• Phase 2 artefacts published to /app/bioinformatics/phase1-2026-08-30/ and work/phase1/ (git-ignored; they contain embargoed sequences and are never a deliverable — the release filter is Phase 8). **Phase 2 must consume the 08-30 directory, not 08-29**: the partition is the same but 17 seed sequences are now correctly trimmed, and a profile is built from mature chains.
• docs/phase1-seed-families.md, docs/phase1-catalytic-panel.md and docs/hmm-pipeline-plan.md updated so the corrected numbers are operative; the 2026-08-29 sections are marked superseded and kept verbatim, as is the 2026-08-26 baseline. No earlier measurement is edited or deleted.
• Read-only throughout: SELECT statements only, through a session the server held readonly. No visibility or visibility_hold_* column was read for filtering or written. Verified afterwards — 291 private proteins all still bulk_negative_results, and 0 sessions left idle in transaction.

Changed

• **Phase 1 of the profile-HMM pipeline re-run against the grown corpus (577 → 638 sequenced wild types), and the result is close to a null.** The owner's approval round added 61 proteins, all visibility = 'staged'; the 220 embargoed and 357 public rows are untouched. No code changed — same hmm_seeds.py / export_seed_fasta.py / build_seed_families.py (md5-checked against the deployed image), same mmseqs and HMMER 3.4, same 29-profile catalytic panel, same parameters. The only variable is the corpus. Seeds 427 → 483, N3 hard negatives 150 → 155, post-dedup seeds 310 → 350, families 94 → 108, families clearing the Phase 2 n ≥ 8 gate 3 → 4, and the fraction of the deduplicated corpus inside gate-clearing families 51.0% → 50.3% — i.e. down. Singletons stay at ~73% of the family table and one family still holds ~40% of the corpus. **An 11% larger corpus bought one additional profile-eligible family and moved nothing else.** What did change is worth having: the new family is F0004 (baseline F0009, 5 seeds, joined by 11 new proteins → 10 seeds, PF24708 Lip_C), and it is the first gate-clearing family carrying a large internal negative set — nine members are PBAT/PBSA-positive and PET-negative, which is exactly the within-family contrast Phase 3's N3 tier and Phase 6's rescorer need. Natural rubber also enters the seed corpus for the first time, as two families (RoxA/RoxB rubber oxygenases, 5 seeds; Lcp latex-clearing proteins, 4 seeds), neither of which reaches the gate. The incumbent partition was not re-shuffled: 2 of 94 baseline families have members in more than one family now, and one merged. A Phase 2 profile built on the baseline partition is still built from seeds that are still together.
• Catalytic-panel coverage re-measured: 537/577 (93.1%) → 584/638 (91.5%). The panel is unchanged at 29 profiles and PF12740 PETase — the baseline's largest omission — is still in it and still the best-scoring profile for 368 of the 584 proteins that find a domain. The drop is entirely cohort-driven: the 577 incumbents score identically to the baseline (93.1%) and the 61 new arrivals score 47/61 (77.0%). Panel usage shifted where the corpus did — Lip_C (PF24708) 5 → 16 and Abhydrolase_5 (PF12695) 0 → 7, the latter having been the best hit for nothing at all until this round.

Fixed

• Nothing in code. One defect found and reported, not fixed here: all 61 new proteins carry has_signal_peptide = NULL, so the SignalP enrichment has not run on the approval batch. signal_peptide_stripped is therefore still 351 — not one of the 61 could be trimmed to a mature chain — and Phase 1.1's acceptance bar now reads 577/638 = 90.4%, one approval round from failing its own > 90% threshold. scripts/backfill_signalp.py should run on the batch and Phase 1 be repeated before Phase 2 consumes this partition.

Notes

• Read-only throughout: the corpus was read once through a session the server held readonly, SELECT statements only. No visibility or visibility_hold_* column was read for filtering or written at all; pg_stat_activity shows no idle-in-transaction session left behind. Every artefact lives under the git-ignored work/ (build host) and /app/bioinformatics/phase1-2026-08-29/ (worker pod) — they hold embargoed sequences, are never deliverables, and the release filter remains Phase 8.
• Documentation: docs/phase1-seed-families.md and docs/phase1-catalytic-panel.md each carry a dated re-run section. **The 2026-08-26 baseline numbers are kept verbatim** — the comparison is the point.

Fixed

• Approving an ingestion that mints a new protein returned HTTP 500. Production carried two hold-reason enum types at once: microorganisms.visibility_hold_reason was on visibility_hold_reason_enum (what the ORM declares, and therefore what asyncpg casts every INSERT parameter to) while proteins.visibility_hold_reason was still on protein_visibility_hold_reason_enum, the name 0010 gave it. Every INSERT INTO proteins raised asyncpg.exceptions.DatatypeMismatchError: column "visibility_hold_reason" is of type protein_visibility_hold_reason_enum but expression is of type visibility_hold_reason_enum out of ingestion_promotion._get_or_create_protein's db.flush(). The discriminator is whether the approval creates a protein row, and nothing else. Measured against the live queue on 2026-08-29: of 92 ingestions in pending_review, 46 would mint a protein and 46 would not, and the 19 ingestions a curator had actually tried and had 500 on were all in the first set — while all 36 that returned 200 promoted only organisms or reused proteins. A SELECT needs no cast, so every read path stayed healthy and the split was invisible from the outside. The same fault silently broke the operator visibility hold on proteins, which UPDATEs that column. Migration 0013_reconcile_hold_enum moves every column still on the old type across to the shared one and drops it. The two types held **identical labels**, so USING col::text::newtype is exact — the 296 held proteins keep their reason, prior visibility, batch and audit trail, and no value changes.
• The deployment order that produced it, so it cannot recur. 0011 renamed the type and guarded the rename with "does the new name already exist?". Three starting states were considered and the one that happened was not among them: the migrate job runs create_schema() before alembic, and metadata.create_all(checkfirst=True) creates every missing enum type in the metadata even when it creates no table — an Enum with no explicit metadata= registers a before_create listener on table.metadata. So the new type was conjured out of the ORM declaration before 0011 ran, 0011 saw it and returned, and proteins was left behind. tests/test_pg_migrations.py was green throughout because its _strip_holds drops the enum before replaying 0010 → 0011 — the one starting state in which the rename does fire. It now also replays the deployment's order, and ends on the literal INSERT ... NULL::visibility_hold_reason_enum the outage was made of. 0011 is left exactly as it shipped: it has already run, and rewriting an applied migration would not change what is in the database.

Added

• **Two measurement_type terms for quantities the vocabulary could not hold: depolymerization_extent and degradation_half_life.** The vocabulary had 21 terms and not one of them was a dimensionless extent or a duration, so both went to the nearest-sounding term and corrupted it. Measured across 549 extractions: depolymerization_rate carries **1055 observations in 100 papers under 100 distinct unit strings, and only 839 of them are rates**. 140 are dimensionless extents (%, % of PAM, % of wild type, % of WT, % of theoretical, % of control, fold) in ~20 papers; 29 are durations in h or d — a number whose dimension is the reciprocal of the term it is filed under. Any cross-paper aggregation on that field is already wrong, and nothing downstream can see it: a float with a unit string is a float with a unit string.
  • depolymerization_extent (enzymatic, default %) — how much of the polymer came apart, as opposed to how fast. Method-agnostic on purpose: NaOH consumed in a pH-stat titration, monomer quantified by HPLC and converted to conversion, or a gravimetric conversion all answer one question. The pH-stat is a real measurement of how much polymer was broken down — the base neutralises released carboxylic acids, so it is proportional to ester bonds cleaved — and it had no honest home. It is not weight_loss, which means mass lost from the specimen on a balance: 10.1002/cssc.202501789 reports "% PET depolymerization calculated from total NaOH addition" and "% direct PET weight loss after recovery, washing, drying and weighing" for the same condition, 42 observations under one measurement_type, so a reader cannot tell the titration number from the balance number (45 corpus-wide, with 10.1021/ma9005318).
  • degradation_half_life (enzymatic, default h; days are equally legitimate) — the time for half the polymer substrate to be degraded. 10.1016/j.envint.2018.12.029 reports 16 explicit nanoparticle degradation half-lives (t½, 3–97 h, wild type against variant) as depolymerization_rate in h, and 10.1038/s41529-023-00342-9 reports first-order removal constants in day⁻¹ with the half-life it derives stranded in prose. It is deliberately not the enzyme's thermal half-life, which already lives in ExtractedProperties.half_life with half_life_temperature: one is how long the protein survives at a temperature, the other is how long the plastic survives the protein. That is the one pair in this vocabulary a single word could silently collapse, so both term descriptions say it, the documentation page carries a warning callout, and a test pins it. Both sit in the enzymatic group beside depolymerization_rate rather than under the assay that produced them. The group is a dropdown bucket, and the defect being fixed is a curator reaching for depolymerization_rate when they mean an extent or a duration — three terms in three different buckets would preserve exactly that. No migration. The vocabulary lives in MEASUREMENT_TYPE_VOCAB and is served from it; the measurement_types table is a lookup that _resolve_measurement_type_id get-or-creates on promotion, so an existing database acquires the rows the first time either term is promoted (or on the next make seed), and no live table is altered.
• METRIC_POLARITY learns both terms, and one of them inverts. depolymerization_extent is higher-is-better; degradation_half_life is lower-is-better — an enzyme that halves the polymer in less time is the better enzyme — which is the opposite of half_life, the protein's thermal half-life, three lines away in the same namespace. Km is the other entry in that registry whose direction cannot be assumed, and it is there for the same reason.

Fixed

• weight_loss's term description now says what it is not. A conversion computed from pH-stat titration or from released monomer is depolymerization_extent "even when the paper prints the two side by side" — which is the case that produced the 42 indistinguishable observations above. Same for depolymerization_rate, which had no description at all while carrying three different quantities under 100 unit strings.

Notes

• Two gaps found by the same sweep were deliberately not decided and are recorded as deferrals in REJECTED_TERMS with their counts, for a curator ruling rather than a recount: specific_activity (22 observations in 8 papers filed as depolymerization_rate in U/mg, U/mg protein, nkat/mg, kU/mg — a rate per unit enzyme, not per unit time, so the existing term is not lying about what the number is, only about what it may be compared with), and time_to_conversion (10 observations of "reached 90% PET conversion in 27.4 h" — a duration to a threshold the paper chose, which is not a half-life and must not be absorbed by one).

Changed

• The REST API page is reorganised around what a researcher came to do. /documentation/api opened with Conventions and then a nine-row Authentication table — an integrator's reading order, for an audience that is not an integrator. A scientist arrives wanting a sequence, an organism, a filtered export or a job, and met the auth model first and a <protein_id> placeholder second. The order is now enzymes → engineered variants → microorganisms → plastics, papers and pathways → search → bulk downloads → dataset releases → jobs → then authentication. Auth is not deleted, because a job genuinely needs a credential; it is scoped honestly to the three things that need one and put where you reach for it. The cookie/Bearer split is stated rather than implied: the API refuses the browser session cookie on purpose, because a cookie the browser attaches by itself is a CSRF hole, so being logged in to the website authenticates nothing an API client does. Every example is a runnable command against v2.plasticdb.org with real accessions from the live database — IsPETase PBG5RN (numeric id 21), Pseudideonella sakaiensis tax id 1547922, PET as plastic 16 — and the whole page was verified against production before being written down.
• Weekly dataset releases are documented where users will look for them. GET /api/v1/releases/current returns the version, the DOI and a ready-made citation string; the page now says to record version in a methods section and quote citation verbatim, and explains why a quiet week's release carries an earlier release's DOI. /releases, /releases/{version} and the three /snapshots routes go with it.

Fixed

• Documentation that described endpoints we removed. README.md still carried the "Verified subset (WS07)" block: GET /api/v1/downloads/verified-set, a release= parameter and an X-PlasticDB-DOI response header, all removed in #248, plus app/services/verified.py, which no longer exists. The endpoint 404s in production. It was also the only place the README described the release/DOI surface, so a reader asking how to cite a version was sent to a dead endpoint — while the correct ### Dataset releases section sat directly beneath it. (#270)
• Five wrong claims on the API page, each found by reading the routers rather than the previous copy: /search was said to search variants (it covers microorganisms, proteins, plastics and papers — there is no variant scope); GET /jobs/{id} was marked auth-required (it is OptionalUser, so an anonymous job is readable by anyone holding the id); POST /submissions/lookup-sequence and /validate-sequence were marked auth-required (both handlers take no user dependency); "list endpoints cap at 500" is wrong for /search/grouped (100) and /releases//snapshots (200), and /papers defaults to 20 rather than 100; "entity detail routes are not rate-limited" is wrong for /variants/{id} and /engineering/proteins/{accession}, both 200/hour.

Added

• The public surface that existed but had never been written down. The release and snapshot routes above; GET /jobs/{id}/download; /search/grouped; /microorganisms/names; /microorganisms/accession/{accession}, which is the only address an organism with no NCBI tax id has; /microorganisms/taxonomy/{rank}/{name}; /proteins/accession/{accession}; /variants/{id}/structure; the four /pathways routes; and all seven /bioinformatics/* job endpoints, which had never appeared on the page despite being the JSON path built for scripted access. The page now also links the generated schema at /docs, /redoc and /openapi.json — it previously referred to "the OpenAPI schema below" without ever linking one.
• A copy button on every example. Bound with addEventListener and delegated from a single container, on a nonce-carrying script with no on*= attribute anywhere, so the page keeps the strict CSP and the inline-handler ratchet does not move. The button carries an aria-label and its own :focus-visible outline at specificity (0,2,0), which clears sb-admin-2's button:focus{outline:0}.

Added

• Weekly dataset releases — the database now has a version you can cite. Curation used to publish on approval: the rows a curator's approval created took the visibility column default, public, and were in the DIAMOND reference databases and the next Figshare snapshot within the hour. That is a continuously moving corpus. A user who ran an alignment on Tuesday and again on Friday had no way to say what they had searched either time, and no way to get Tuesday back. The model is now inverted. Curation stages; a release publishes. Approval writes visibility='staged' — a fourth rung on the entity visibility enum, and therefore invisible to every read path in the codebase on the day it was added, because they are all spelled visibility == 'public'. One scheduled task per week, Thursdays 06:20 UTC, turns the staged set public, rebuilds the reference databases from it, and cuts an archived snapshot. **The release version is YYYY.WW and is deliberately not the application version.** 1.9.0 says which code is deployed; 2026.35 says which data you aligned against. It is monotonic, sorts as plain text, and is the same string as the snapshot's release_label — so the version a user quotes and the archive they cite cannot drift apart. It appears in the footer of every page, in GET /api/v1/releases/current (with a ready-made citation and the DOI), and inside the archive itself. A DOI is minted only when the data actually changed. A new dataset_fingerprint hashes exactly the rows a snapshot would emit, through the same visibility-filtered builders, so a quiet week completes with no snapshot and no second DOI for an identical dataset. This replaces the monthly snapshot beat, which minted one on the 1st regardless. The archive itself is still built by the existing generate_snapshot_task — a second copy of that pipeline would be one more thing that can publish. **The release is built to be structurally incapable of publishing a withheld record**, which is the requirement the whole design answers: it runs unattended, its entire purpose is making records visible, and 296 proteins are currently held ahead of unpublished lab work. There is one query for what may be published — query.releasable_filter, placed immediately below public_visibility_filter so the gate on what the public sees and the gate on what a release makes public are read together — and four independent things have to fail before a held record reaches a DOI. A hold does not annotate a record, it moves it, so a held record is not staged and no query the release runs returns one; publish_record re-checks every condition on the loaded object, so a superuser's id list is a narrowing and never a widening; the before_flush guard stays the last line rather than the only one; and an invariant counting records that are both held and public runs before promotion, inside the promotion transaction, and once more immediately before the snapshot. The release never touches a deposit's records — those stay with deposit_visibility.records_to_publish and the superuser doors #417/#428 narrowed to, because a seventh leak of this class is likeliest to be born as a well-meaning second door. Failure and re-runs. Promotion is one transaction, so an interrupted release has published nothing. Each of the three stages is idempotent and the release row records which one it reached, so a re-run resumes rather than starting over: a failed DIAMOND rebuild leaves records correctly public and is repaired by the hourly rebuild beat anyway; a failed snapshot reuses this version's row rather than cutting a second. The unique version column is the lock — two beats in one week compute the same string and one finds the other's row. Operator controls, both of which can only ever show less or the same. POST /api/v1/admin/releases cuts an out-of-cycle release now, with its own citable version, publishing exactly what the scheduled run would have. POST /api/v1/admin/releases/publish-now publishes named staged records for a correction that cannot wait a week, through the same filter and guard. A new Dataset releases setting holds the weekly release back entirely. Nothing in the release path lifts a visibility hold — not the schedule, not a force, not a superuser — and a record that was staged when it was held returns to staged when the hold is lifted, rejoining the next release on the operator's timing. Every run posts to Slack with the half that matters: what it declined to publish and why ("withheld — proteins: 296 under an operator hold"), which is the weekly proof, without anyone going to look, that a withholding is still in force. New: docs/dataset-releases.md (cadence, version scheme, citation, and the operator's view), a Dataset releases panel on the Downloads page, migration 0012 adding the staged label to entity_visibility_enum, and tests/test_pg_dataset_releases.py — which has to be the PostgreSQL tier, because every rule it proves is a WHERE clause and a filter that silently stopped filtering would leave the mock suite entirely green.

Changed

• Records created by ingestion promotion are staged rather than public, so a newly curated record appears on the next Thursday rather than immediately. The latency is accepted deliberately: a stable, nameable corpus is worth more than same-day visibility. Contributor submissions are unaffected in outcome — their rows take the deposit's visibility a moment later, as before.
• The monthly snapshot beat is retired; monthly_snapshot_trigger remains as a manual way to cut a dated YYYY-MM archive.

Added

• A curator surface for reviewing and releasing deposits (#429). #417/#428 made publishing a deposit a superuser act and closed three doors into a privilege-escalation bypass. What it left behind was a right with nowhere to exercise it: there was no listing of deposits anywhere in the product, so the only signal that one was waiting was a nightly Slack line, and the only way to act on one was to already know its accession. A permission that exists only as a 403 is not a workflow. /curation/deposits is the queue. It shows every deposit, most urgent first, with its records, its depositor, how long it has waited, and — the part no column could previously answer — *whether anything is going to happen to it on its own*. Two embargoed deposits with the same past release date have opposite futures: the one carrying a submission_id was sanctioned by a superuser at acceptance and the nightly beat publishes it, while the one without is waiting on a person, indefinitely. That second row is Awaiting curator release, and it is the queue's headline. Each row expands in place to the records it contains, flagged with the two things that make a release do less than it looks like it will: members the deposit merely lists (reused from the corpus, not its to publish — #430) and members under an operator visibility hold (the flush guard parks the write, so a release leaves them exactly where they are). The row states up front how many records a release would actually publish. The curator/superuser split is drawn deliberately and does not widen #428. A curator sees the whole queue — reading unpublished material is review, and restricting the list to superusers would mean the only person who can see the queue is the only person who can clear it. A curator's action is a recommendation: POST /deposits/{acc}/notes records checked / recommend_release / needs_changes / recommend_reject and a note as a ContributionEvent, and its payload has no visibility field at all. Release stays with the superuser and goes through the route that already existed — this change adds no second publish path, since a seventh leak of the "something publishes what it should not" class is likeliest to be born as a well-meaning new door. POST /deposits/{acc}/withhold is the reject half, superuser-only for the mirror reason: it clears release_date, cancelling a publication the depositor arranged and the beat would have carried out. It refuses a public deposit (un-publishing is a retraction, with exports and possibly a Figshare DOI already carrying the records), it only ever lowers, and it skips held rows via the new deposit_visibility.records_to_withhold — writing to a held record's visibility would overwrite the visibility_hold_prior that its eventual release restores, which is how the 296 deliberately embargoed proteins would have come back private instead of public.
• **GET /deposits/mine says what state a deposit is in, not just its visibility (#429).** A depositor reading embargoed on a deposit whose release date passed last month could not tell whether the database had forgotten them or was still deciding. state / state_label now say "Awaiting curator release", which is a queue position rather than an embargo that never ends.

Changed

• The deposit maturity predicates (due_deposits_stmt, unsanctioned_due_deposits_stmt) moved from app/workers/tasks.py to the new app/services/deposit_queue.py, so the curator queue and the nightly beat classify a deposit by one predicate rather than two copies of it. tasks.py imports them; there is no behaviour change and no second definition.

Fixed

• **The Organism Type control on annotate-gene / annotate-genome does something (#342).** It was submitted, stored on the job row and displayed — and no worker ever read it: grep -rn organism_type app/workers/ returned nothing, so "Gram-positive" and "No signal peptide" produced byte-identical jobs. Until v0.58.0 the About This Tool panel had additionally claimed SignalP was applied to the selection, which made the control worse than absent: it implied the analysis had accounted for something it had not. The predictor that was missing when the issue was filed shipped in v1.4.0, so this is wiring. Picking an organism now runs a **signal-peptide prediction over the query sequences that hit the database** — the question the tool is actually asked, since a plastic-degrading hydrolase has to be secreted to reach the polymer. signal_peptides.predict() takes an organism and translates the form's vocabulary into each predictor's own argument: DeepSig's -k kingdom (euk / gramp / gramn, with archaea mapped to gramp — DeepSig has no archaeal model and archaea are monoderms) and SignalP 6's --organism (eukarya or other, its two models). **Phobius, the default, has no organism argument at all**: there the selection decides whether to predict rather than which model to use, and the job page says exactly that rather than over-claiming. The result is visible where the job is read: the organism type on the status card, a per-query Signal peptide column in the results table (Yes, with the cleavage site, or No), and a summary above the branch split so the "no matches" and file-backed branches report it too. The step can never fail a job. An unconfigured predictor, a BLASTX job (whose queries are nucleotide, and the predictors take protein) or a per-sequence error each settle as a stated reason in the results — the DIAMOND hits are what the user came for. A whole proteome is capped at its best-scoring queries (SIGNALP_JOB_MAX_SEQUENCES, default 50) because the default predictor is a rate-limited hosted service taking one sequence per submission, and a truncated run says so rather than reading as a complete one.

Fixed

• **Accepting a submission can no longer publish a pre-existing private record (#430) — the fourth door into the embargo bypass #417 closed three of.** submission_promotion._cover_reused attaches a Protein or Microorganism that was already in the database to the deposit the submission asked for. Two cases were already guarded (already public, already owned by another deposit); the third assigned the deposit's visibility to a row that was **private and owned by nobody** — so a submitter asking for visibility='public' published a record that had never been curated for publication, silently, as a side effect of somebody else's acceptance. 296 proteins are currently under a deliberate operator embargo. A reused row is now never made more visible than it already is: it is recorded as a deposit member (so the deposit still describes the whole finding, and a reviewer holding the token still sees it) and its own visibility and deposit_id are left exactly where its owner put them. private → embargoed counts as an escalation too — an embargoed deposit publishes what it owns the day its release date passes, so adopting the row would have deferred the leak rather than closed it. Every refusal logs, matching the two branches that already did.
• **A deposit release now publishes the records it owns, not the ones it lists.** The same bug with a timer on it, found while fixing the first half: both release doors — POST /deposits/{accession}/release and the daily release_due_deposits beat — iterated deposit membership, which includes every reused row. So the record _cover_reused was not allowed to relabel got published at release time instead, and a member still embargoed under another deposit went with it. Both doors now call one function, deposit_visibility.records_to_publish, which filters on deposit_id.
• **A community field suggestion can no longer carry visibility, deposit_id or an accession.** Found by auditing the same pattern elsewhere. POST /annotations is open to any authenticated account and takes a free-text field_name/new_value; accepting one is a curator right, one flag below the superuser right that assert_may_publish reserves publication to. The applier ended in a bare setattr(entity, field_name, value), so {"field_name": "visibility", "new_value": "public"} on an embargoed protein published it, and deposit_id re-homed a row out of its embargo container. A short governed-field denylist now refuses both, and a field_name that is not a column of the target model is refused rather than written as a stray Python attribute that never reaches the row.
• PATCH /annotations/{id}/review returned 500 for every review. The blanket db.refresh() after the write expired the eagerly-loaded user that the response builder reads, and an async session cannot lazy-load it (MissingGreenlet). The endpoint had never been driven from a route-level test; the refresh is now scoped to the columns it can meaningfully update.

Changed

• app/services/deposit_visibility.py is the one place the rule lives: the private < embargoed < public ladder, escalates() for cover time, and records_to_publish() for release time. DECISIONS.md §8 states both halves.

Added

• **Phase 6.1 — docs/ml_engine_design.md, the design document app/workers/ml_engine.py had referenced since it was written and which did not exist.** Training data, head gating from the computed evidence tiers, the feature set and the three things deliberately absent from it, the gates a head must clear, the weight storage layout and its atomic swap, the model card that doubles as the cache key, and the inference contract. It settles a shape problem nobody had written down: **MlEngine was stubbed as a SequenceSearchEngine, and a rescorer is not a retrieval backend.** A SearchHit carries pident, evalue and bitscore — three numbers with agreed meanings — and has no field that could hold "probably degrades PCL" without becoming the composite score Phase 4 and §6.4 both ban. The engine therefore keeps the signature, delegates retrieval to DIAMOND unchanged, and attaches a per-polymer annotation block; it never invents, reorders or suppresses a hit, and Phase 4's CLASS_ORDER remains the only quantity that crosses the arm boundary.
• **Phase 6.2 — the masked multi-label rescorer. Trained on all 903 labelled (protein, polymer) pairs, measured with its own leakage reported, and recommended for deferral.** Result document: docs/phase6-ml-rescorer.md. Code: scripts/experiments/rescorer.py (fetch → features → evaluate), tests/test_rescorer.py (32 Tier-1 cases). Labels come from app.services.label_matrix.build_cells, not a second implementation, so the training set is the object docs/label-matrix.md describes. ENABLE_ML_ENGINE stays False and no weights are written. The measurement the phase exists for. The sharpest question this corpus can pose is to train on every other study and predict the one that supplies 35% of every negative pair in the database — 347 proteins in, 210 held out: | PET, dominant-paper holdout | AUROC | 95% CI | AUPRC (floor 0.543) | MCC | |---|---:|---|---:|---:| | masked multi-label MLP | 0.621 | [0.54, 0.77] | 0.635 | +0.243 | | random forest | 0.597 | [0.51, 0.76] | 0.624 | +0.240 | | baseline — nn identity, pos − neg | 0.587 | [0.48, 0.66] | 0.603 | −0.061 | | baseline — best single feature | 0.516 | [0.44, 0.65] | 0.553 | +0.000 | | label-permutation null, p95 | 0.568 | | | | 0.034 of AUROC over a trivial baseline. Phase 0's verdict survives contact with four times the label set, a 650M embedding model, a masked multi-label loss and a study-level split.
• Five things the phase settled, none of them a modelling result. 1. Report the leakage, then report the noise beside it. Phase 0's gradient reappears — PET per-fold AUROC 0.699 at 28% median train↔test identity, 0.770 at 63%, 0.808 at 82% — but the fold-to-fold SD is 0.03–0.17, so across four or five folds nearly every model-minus-baseline gap in the experiment is inside one standard error. The honest conclusion is not "the model is bad", it is "407 PET pairs cannot tell". 2. Study leakage is a second axis and no identity split can see it. At the same achieved identity — 0.633 against 0.645 — the PET forest scores 0.770 under an identity split and 0.597 when the held-out fold is one study's screen. 3. **A third gate, discovered by the run: a head's positives and negatives must overlap in family space. contrast_overlap() measures it. PHB's share none** — 17 of its 18 negatives are enzymes that degrade something else — and PCL's share 2 of 10. Their strong-looking numbers are family recognition, and a single profile bit score gets 93–95% of the model's discrimination on them. PET is the only head whose contrast set genuinely overlaps (9 shared families), which is exactly why its question is the hard one. 4. ESM-2 650M — the size §6.2 names, deferred by Phase 0 — changes nothing. Across 34 head × split × model comparisons: mean +0.011, median −0.001, 16 improved against 18 worsened. On the decisive fold the block costs 0.11 AUROC. 5. Pooling out-of-fold scores is unsafe on folds this uneven. On the paper split PET pools to 0.532 while every individual fold sits at 0.60–0.86. The gap is calibration, not discrimination; both figures are published rather than the flattering one being chosen.
• docs/ is now copied into .github/test.Dockerfile. The guard that app/workers/ml_engine.py's design-doc reference does not dangle could not itself run in the job that blocks the merge. That reference dangled for the whole life of the stub, which is the entire reason Phase 6.1 exists.

Changed

• app/workers/ml_engine.py and app/workers/engines.py now say why the engine is off — no polymer head clears the discrimination gate — and point at both documents, rather than at "no model weights have been trained yet". The NotImplementedError is a design output, not an unfinished edge.
• docs/hmm-pipeline-plan.md records status for §6.1–§6.4. Two of §6.4's four ranking steps are not implementable as written: a TM-score >= 0.5 hard gate no ProstT5 hit can clear (Phase 5), and a primary sort on a predicted Tm that DeepTM has never been able to supply. The ban on a composite score is untouched and is now load-bearing in a second place.

Added

• **Phase 5 — the structural tier. Built, measured against the curated corpus, and recommended for deferral: it finds almost nothing the sequence arms missed.** No discovery campaign was run. Code: app/services/hmm_structure.py (pure logic, 45 Tier-1 cases in tests/test_hmm_structure.py), scripts/structural_tier.py (weights → refdb → predict → anchors → search → fidelity → yield → verify → bench). Result document: docs/phase5-structural-tier.md. The measurement the phase exists for. Scored over Phase 1's 94-family partition, in both the production configuration (ProstT5 3Di → our reference models) and an upper bound this project cannot have for a novel screening hit (real structures on both sides). Universe: the 227 deduplicated seeds that hold a model, 10 512 true same-family pairs, 177 queries with a relative. | route | precision | recall | F1 | |---|---:|---:|---:| | profile arm (hmmsearch, 4 profiles, leaked in its favour) | 0.664 | 0.993 | 0.796 | | tail arm (DIAMOND) | 0.965 | 0.899 | 0.931 | | structural arm — production | 0.675 | 0.837 | 0.747 | | structural arm — upper bound | 0.676 | 0.841 | 0.749 | The structural arm is worse than DIAMOND on both, in both configurations, and real coordinates buy it 0.004 of recall. Its exclusive contribution over both sequence arms is 12 true pairs against ~1 850 false ones (0.6% precision) and 3 queries out of 177 — against 4 the sequence arms reach and it does not, so in the production configuration the tier is net −1. In the upper bound the TM ≥ 0.5 gate rejects 409 of 67 539 rows (0.6%), which is the plan's own "every α/β-hydrolase shares the fold" prediction landing as a number; in the production configuration no hit can be gated at all and all 29 282 are structural_analog_unverified.
• **ProstT5 evaluated first, as the plan requires, and adopted — but the plan's §5 gate does not survive it.** Folding a novel sequence is not available to this project (ESM Atlas: cached 110-mer ~1.5 s, novel one 504s at ~29.7 s, repeatably; 128 rows still retry-capped), so ProstT5 was the only route worth evaluating. It runs offline, needs no GPU, is **deterministic across thread counts** — byte-identical 3Di at 1, 4, 8 and 12 threads — and reproduces the structure's own 3Di string at a median 0.597 per-position identity over 227 chains fed their own coordinates' amino-acid sequence. What it cannot do is produce coordinates, so **a TM-score cannot be computed for a ProstT5 query and an active-site geometry check is impossible** — and asking for one anyway makes foldseek die in convertalis, the last stage, after the prediction and the whole search have been paid for. Hence two pinned output formats and a detect_format() that identifies a table by width. fold_gate() therefore passes a row that has no TM-score rather than failing it — failing it would report "below 0.5" for a quantity nobody computed — and such a hit is classed structural_analog_unverified. In place of geometry, anchor_transfer() carries the reference's catalytic anchors through the foldseek alignment and checks the residues, named catalytic_residues_conserved and never "active site intact".
• A third arm, and still no composite score. Phase 4 keeps the profile arm and the tail arm apart because their bit scores are log-odds against different nulls; a foldseek E-value is a third such quantity and a TM-score a fourth. class_order_with_structure() derives the combined ordinal order from Phase 4's CLASS_ORDER rather than restating it, inserting the structural classes below seed_homolog on the plan's own reasoning, and a test asserts every Phase 4 class survives in its original relative order. Ranking is numeric within an arm only.
• Compute, measured and then extrapolated, with the two labelled. ProstT5 runs at 6.30 residues per CPU-second (measured over 227 chains at four threads, uncontended; 3.25 at twelve — thread scaling is poor) against the profile arm's 2.48 × 10⁶, a factor of 394 000. Applied to hits, which is where the tier belongs, that is 16 CPU-minutes for the 19 survivors of Phase 4's real sample — four minutes of wall clock at four threads — and is not a problem. Applied to an nr-sized catalogue it is 1.32 × 10⁷ CPU-hours. **Compute is not what defers this tier — yield is** — but it permanently fixes where the tier could ever sit in the cascade.
• Structure coverage published rather than assumed. 406 proteins hold a structure — 120 experimental, 138 AlphaFold, 148 ESMFold. Among the 227 deduplicated seeds in the experiment, **71% of the reference models are predictions**, and evidence_matrix() measures what that does to the result: only 15.2% of classified hits are experimental-against-experimental — and 0.0% in the production configuration, where the query's 3Di is always predicted. structural_hits() takes the query side's provenance separately so that a ProstT5 string is never reported as an experimental structure merely because the same protein happens to have one on file. Every hit therefore carries the 3Di provenance of both sides, and its evidence kind is the weaker of the two — an ESMFold model against a ProstT5 string is two predictions agreeing with each other, not a structural measurement.
• Determinism, verified so that it can fail. Two credible candidates, both checked: fp16 ProstT5 inference reducing over a thread-dependent partition (byte-identical), and foldseek's output order, which is a function of thread scheduling — one digest 156d868b… across 1, 3 and 8 threads over three different files (canonical_rows() sorts into one total order; structural_digest() hashes the parsed rows, so a printf change is not mistaken for a result change). The Tier-1 suite asserts the digest is invariant across three permutations and that perturbing one bit score by 1.0 moves it — the negative control Phase 4's -Z tolerance check did not have.

Changed

• docs/hmm-pipeline-plan.md §5 carries the status block and is flagged for rewriting: its gate ("TM-score ≥ 0.5, then check active-site geometry explicitly") is not implementable for the only query route available to this project.

Added

• **Phase 4 — the screening pipeline. The cascade is built, proved on the frozen fixture and on a bounded real sample, and benchmarked.** No discovery campaign was run. Code: app/services/hmm_screen.py (pure logic, 41 Tier-1 cases in tests/test_hmm_screen.py), scripts/screen_sequences.py (antifam → orfs → plan → screen → report → bench → verify). Result document: docs/phase4-screening.md. pyrodigal -p meta (in process) → orfs.faa, partial genes flagged and kept hmmsearch --cut_ga AntiFam.hmm → every match discarded hmmsearch --cut_ga --domtblout -Z N --domZ N PlasticDB.hmm → the profile arm coverage / bias / length / Δbits → the §4 post-filters, in order diamond vs no_profile.faa → the tail arm diamond vs seeds.faa → nearest characterised neighbour + paper
• **The two arms are reconciled, never merged — the central design decision of this phase. 55.8% of the corpus sits behind a profile and 105 curated proteins, a quarter of it, are reachable only by searching no_profile.faa**. A profile bit score is a log-odds against that profile's null; a DIAMOND score is a log-odds against one curated sequence. There is no rescaling that makes them one number, so the report does not pretend there is: both arms run on every query, both are reported wherever both fire, ranking is by an **ordinal class** across arms and numeric only within one, and arm_overlap() publishes the split on every run. This is §6.4's ban on laundering unreliable numbers into a confident-looking float, applied one phase earlier. On the real sample the tail arm reached more than the profile arm did: of 19 queries reached, 8 profile-only, 11 tail-only, 0 by both. That settles a product question — /tools/hmm-screen has to lead with the non-profile route rather than footnote it.
• **Competitive Δbits works in raw bits and survives a family with no measurable NC.** M0001's 5 325 hard negatives all scored below the floor, so its NC, its NC→TC gap and its MCC are undefined. Any margin normalised against them is a NaN for that family, which silently drops every M0001 hit or admits every one with nothing in the output saying which. Δbits is therefore a difference of two full-sequence scores from the same search over the same database with the same -Z, the margin is a declared 3.0 bits (one octave of odds), and a family's own Pfam is excluded from its own confusable panel exactly as Phase 3 excludes it from that family's N3. Separability enters the label — family_member_uncalibrated, "read --cut_ga here as near-exact members only" — and never the arithmetic. A library with no calibration.json beside it degrades every hit to that class rather than up to family_member.
• **-Z and --domZ are computed once over the whole input and passed to every shard**, including the single-process run the shards are validated against — every hmmsearch in this phase is built by one argv function, because a one-process path that took -Z from its input while the shards took theirs from the plan would make the acceptance test compare two different searches and pass. Measured on the real sample: **four shards reproduce the single process row for row and E-value for E-value**, and a 4-shard full screen produces the same hit-report digest as the unsharded one. Without -Z, the same four shards produce E-values whose **median ratio to the correct ones is 0.2515 against an expected 0.25** — the classic sharding bug, in a table that concatenates perfectly cleanly either way. assert_comparable_z() refuses to merge shard outputs whose recorded Z disagrees, and the Kubernetes Job is rendered from the plan object so the two cannot drift.
• A hit is homology plus provenance, and never activity. Phase 0 measured profile-HMM AUROC at 0.44–0.47 for PET — below chance — and no polymer earns a calibrated confidence on this build, so every artefact leads with that sentence and every row carries activity_claim: none. The polymer wording is Phase 3's verbatim from polymer_policy.json, which preserves the distinction that is easy to lose in a report template: PET reports *tier cleared, discrimination did not*, while PCL, PLA and PHB report tier cleared, discrimination not run. An unknown polymer gets Tier 3's wording — the weakest claim available — rather than silence.
• The provenance chain reaches the paper. A hit resolves to its nearest characterised member by DIAMOND against the curated seed set, and from there to the per-polymer verdicts and the paper DOI behind them. On the real sample the chain resolved on 21 of 26 rows, and the best hit is IsPETase recovered from its own genome at 100% identity to protein:21 with its paper attached and without a word claiming it degrades anything. Phase 1 gains one stage, python -m scripts.export_seed_fasta provenance, because the seed header carries a paper count and a count cannot be cited; without the sidecar the report says paper_provenance: unresolved rather than inventing a citation.
• make hmm-smoke now runs Phases 1–4 on the frozen fixture: it screens it twice, compares the hit-report digests, checks a 3-shard screen against the unsharded one, and pins that an unpinned -Z would have changed E-values, so the check cannot quietly stop detecting the bug it exists for.

Measured

• Throughput is ~40× what the plan was budgeting from. Two independent measurements on a 1.87 M-residue real ORF set (HMMER 3.4, --cpu 4, CPU-seconds = wall × threads): 2.73 × 10⁹ residue·profile per CPU-second against the 4-profile library (1 100 model positions) and 3.36 × 10⁹ against AntiFam (36 150), versus the ~7.0 × 10⁷ in the plan. Extrapolated — and labelled as extrapolated — an nr-sized protein catalogue (3 × 10¹¹ residues, a stated assumption) costs ≈ 34 CPU-hours against the calibrated library. The profile search is not the bottleneck. ORF calling is: pyrodigal ran at 0.51 Mbp/s, ≈ 545 CPU-hours per Tbp of contigs.
• **AntiFam discarded nothing on real bacterial ORFs, and costs 33× the search it protects.** 0 of 5 612 ORFs from a published draft assembly, and 0 of 8 001 after cutting the same assembly into 1.5 kb fragments (which manufactured 2 389 extra ORFs, 75% of them partial, from 1.9% fewer residues). The seam demonstrably fires — fed AntiFam's own seed sequences it rejects 319 of 320 — so the zero is a measurement. Because an AntiFam verdict is a property of one sequence, --antifam-mode post applies it to the hits instead of to every ORF for an identical surviving hit set at 1/1000 the cost; pre stays the default because it is the plan's order, and the one difference is documented (in post the spurious ORFs are still inside -Z, so E-values are smaller while bit scores are identical).
• Determinism, verified rather than assumed. Two independent screens of the same 5 612 ORFs produced a byte-identical hits.tsv. Phase 4's own non-determinism candidate is shard completion order, so nothing is consumed in arrival order: shard rows are merged into one canonical sort, ORF names are a pure function of contig and gene index, and the report digest is taken over the parsed rows so formatting cannot masquerade as a result change.

Fixed

• **The determinism harness had the phase's one real bug, and it was in the check.** verify compared E-values with an absolute tolerance of 1e-12, which every pair of significant hits satisfies at 1e-86 — so it reported "an unpinned -Z would not change E-values" while printing two differing digests beside it. A tolerance that passes on every real corpus is not a check. It compares the ratio now, which is also the informative number, and the fixture pins the expectation so the regression cannot come back silently.

Added

• **Phase 3 — calibration. All four profiles now carry GA, TC and NC, so hmmsearch --cut_ga inherits our thresholds.** This is the differentiator: no other plastizyme resource holds curated negatives, so none can say what its models do on the hard cases. Code: app/services/hmm_calibration.py (pure logic, 47 Tier-1 cases), scripts/calibrate_profiles.py (fetch → negatives → fit → tiers → report). Result document: docs/phase3-calibration.md; per-profile deliverable: THRESHOLDS.md. | profile | members | GA | TC | NC | gap | CV MCC (folds) | |---|---:|---:|---:|---:|---:|---| | F0001 | 221 | 137.00 | 220.60 | 53.40 | 167.20 | 1.000 ± 0.000 (10) | | F0002 | 26 | 92.80 | 161.30 | 24.40 | 136.90 | 1.000 ± 0.000 (5) | | F0003 | 11 | 174.90 | 270.40 | 79.50 | 190.90 | 1.000 ± 0.000 (4) | | M0001 | 17 | 389.10 | 389.60 | — | — | undefined | An MCC of 1.000 here is a statement about how easy the question is. The best hard negative scores 137–191 bits below the weakest true member, so every threshold in that gap gives the same confusion matrix on every tier and the data cannot choose one — GA's precision is bounded by the width of the gap, not by the cross-validation. Reported beside its achieved leakage, as the house rule requires: **every held-out member of every family has a training relative at ≥25% identity**, which is the corpus Phase 0 described rather than a fold-assignment bug. False-positive rate at GA is 0.00% in all four tiers for all four profiles.
• Four negative tiers, built and cached. N0 = 200 000 Altschul–Erikson di-residue shuffles (implemented in-process and unit-tested to preserve every dipeptide count exactly — esl-shuffle is not in the pinned toolchain); N1 = 10 000 length-matched reviewed Swiss-Prot; N2 = 3 487 proteins of eight obligate parasites and host-associated bacteria; N3 = a 6 009-protein pool of the catalytic panel's families, filtered per profile to exclude that family's own Pfam. Only fetch touches the network and it only writes to a cache, so a UniProt release cannot silently recalibrate the library.
• **§3.4's polymer evidence tiers are computed from the label matrix in pairs and emitted as polymer_policy.json.** Tier 1 = 4 (PET, PCL, PLA, PHB), Tier 2 = 8, Tier 3 = 39. No polymer earns a calibrated confidence, because clearing a tier is necessary and not sufficient: PET clears Tier 1 on the best contrast set in this field and still reports "polymer undetermined" (Phase 0: AUROC 0.628 against a null reaching 0.561), and PCL/PLA/PHB clear their tier and have never been asked — which the artefact distinguishes from having failed. Nylon 6 misses Tier 1 by one negative pair and is promoted automatically when it arrives; a test pins that.

Changed

• canonicalise_hmm gained an explicit allow_thresholds argument. Phase 2 still refuses a GA/TC/NC line by default — fitting a cutoff there from the curated activity negatives is the tempting, quiet mistake — and Phase 3 is the one caller that passes it. Re-canonicalising after writing the three lines is what keeps the calibrated library byte-reproducible too.
• make hmm-smoke now runs Phases 1, 2 and 3, calibrates the fixture twice and compares the digests, and checks the Phase 2 guard in both directions.

Reported honestly

• M0001 cannot be calibrated, and the reason is structural. Not one of its 5 325 N3 negatives scored above the reporting floor, so NC, the gap and MCC are undefined; the artefact reports separable: null, not true. The cause is visible in a tier inversion the pipeline now flags: N1 — the easy tier — reached 256.9 bits and N2 reached 309.4 while the hard tier reached nothing, because M0001's family is coextensive with one Pfam family (PF01425), N3 excludes that family by construction, and unfiltered Swiss-Prot still contains it. There is no hard tier to build for such a family without calling genuine members non-members. Its GA falls back to 389.1 against a TC of 389.6 — "at least as good as the weakest member" — which is a documented limit, not a calibration.
• **The specificity report against the 150 curated all-negative proteins cuts both ways.** 106 are hit at GA against Phase 2's 130 at E ≤ 1e-5, and the difference is not a specificity win: 28 of them are genuine family members that score above the reporting floor and below GA, so a calibrated search now misses them. Both columns are published. The count never enters the fit.
• Determinism verified, not assumed. Two independent fits over the same profiles and tiers produce an identical PlasticDB.calibrated.hmm (sha256:327bd360f82eb324…).

Changed

• The build plan absorbs Phase 2's six measured amendments (docs/hmm-pipeline-plan.md). They were written up in docs/phase2-profile-construction.md and deliberately not applied at the time because the plan file was owned elsewhere; all six are now folded in and the Phase 2 document records that.
  • §2.2 — T-Coffee expresso is out of scope, permanently. It is pinned in the image but was never run and never will be: D3 fixes the aligner at MAFFT L-INS-i for every family, and expresso fetches structural templates over the network, which a build step must not do. The structural evidence it was wanted for is spent in Phase 5, where foldseek already runs locally.
  • §2.3 — the TCS gate does not do family surgery. Phase 1 predicted it would fire on the 130-seed family and budgeted for splitting it. It scored 91 against a gate of 50 — the highest of the four profiles, and the lowest score anywhere on the corpus is 80. TCS measures whether an alignment is internally consistent, not whether a family is biologically one thing, and 130 real polyesterases sharing one α/β-hydrolase core align extremely consistently even when they degrade nineteen different polymers. The gate stays at 50; the expectation is corrected, in §2.3 and in both Phase 1 status notes that carried it. The column-creep check on M is the one that would have caught over-broad clustering, and it also comes back clean.
  • §2.0 — the trim-fragmentation repair is now a named step, with its three conditions. Envelope trimming fixed the large families and broke one real 15-seed family; the repair is worth 4.8 points of seed coverage.
  • §2 deliverables — the tail artefacts are first-class. no_profile.faa / no_profile.tsv cover more of the corpus than the profiles do, and they now appear in the Phase 8 release layout beside PlasticDB-A.hmm.gz.
  • §2.4 — the 25% trim hard-fail never came close to firing (worst case 1.6%). Kept, and noted as not the binding constraint on an envelope-trimmed corpus.
  • **Phase 7's fingerprint must include the family partition, the Phase 2/3 recipe constants and the pinned toolchain versions.** A profile is a function of its seeds and of MAFFT, ClipKIT, T-Coffee and HMMER; a toolchain bump that changes every profile must not leave the digest unmoved. The VALID_POLYMER_NAMES swap is pinned to Phase 7 for the same reason — it needs a build artefact that nothing writes until the family build is wired into reference_build.py.
• Phase 2's result is recorded as a status block in the plan, alongside Phase 0's and Phase 1's.

Added

• **Phase 2 — profile construction. Four profiles over 55.8% of the corpus, and a declared tail that is bigger than the profiles.** hmmbuild now runs per homologous family and never per polymer (D1): app/services/hmm_profiles.py (the decision logic, Tier-1 tested) and scripts/build_family_profiles.py (align → TCS gate → split ladder → trim → build → press → search back). Result document: docs/phase2-profile-construction.md. | profile | seeds | TCS | M | trim | recovery | |---|---:|---:|---:|---:|---:| | F0001 | 130 | 91 | 258 | 1.5% | 130/130 | | F0002 | 18 | 80 | 178 | 0.0% | 18/18 | | M0001 | 15 | 82 | 434 | 1.6% | 15/15 | | F0003 | 10 | 83 | 218 | 0.0% | 10/10 | Every profile recovers every seed it was built from, and for every seed its own profile is the best-scoring one — no seed is better explained by another family's model. Across the whole seed set 322 of 427 curated seeds are reached; all 275 inside a profile-bearing family are found.
• t-coffee 13.41.0.28bdc39 joins the pinned toolchain, for §2.3's TCS gate and nothing else. t_coffee -evaluate rebuilds its consistency library from the sequences, so the gate runs offline; §2.2's **expresso is deliberately not run**, because D3 fixes the aligner at MAFFT L-INS-i and a build step must not fetch PDB templates over the network.
• The no-profile tail ships as a named artefact pair, not as an absence: no_profile.faa (137 tail seeds, headers intact) and no_profile.tsv (family, size, the exact reason there is no profile, and every corpus protein each representative stands for after the 90% dedup). **105 curated proteins — a quarter of the corpus — are reachable only through the phmmer/DIAMOND route**, so /tools/hmm-screen has to lead with it rather than footnote it.

Measured

• **The §2.3 TCS gate did not fire, and Phase 1's expectation that it would was wrong. The 130-seed family scores TCS 91** — the highest of the four, against a gate of 50. TCS measures whether an alignment is internally consistent, not whether a family is one biological thing: 130 real polyesterases sharing an α/β-hydrolase core align beautifully even when they degrade nineteen different polymers. The plan's other check agrees — the model is 258 columns against a median trimmed seed of 258 residues, so there is no column creep either. Keep the gate; stop expecting it to do family surgery. Splitting it was measured anyway (split-probe), and it is expensive: at --min-seq-id 0.40/0.50/0.60/0.70 the family becomes 3/11/31/56 subfamilies retaining 98%/86%/58%/28% of its seeds. At 0.60 — the first rung giving more than two profile-sized pieces — 42% of the family falls into the tail.
• The nylon amidases are re-merged, on evidence, under the same gate. Envelope trimming had cut one genuine 15-seed PF01425 family into 6, 6 and 3 (v0.98.0, reported against ourselves). A group of families is a merge candidate only when every seed shares one best catalytic Pfam accession, every family is below the seed gate, and **the same seeds were a single cluster before trimming** at the same identity and coverage — that last condition is the evidence the trim did it, and it is why pretrim exists. A candidate then faces the same TCS gate, the same 25% trim hard-fail and the same n >= 8 as anything else. On the real corpus the rule finds two groups and admits one: M0001 (TCS 82, M = 434 — the full Amidase envelope, not a fragment — 15/15 recovered, zero hits outside itself) and rejects M0002 as too small. Seed coverage 51.0% → 55.8%; that merge is the entire improvement.
• Nothing else moved the other 44%. Sixty-nine of the 88 tail families are singletons, and Phase 1's panel widening — the intervention meant to dissolve them — left 64 of 76 in place. The tail is permanent at this corpus size.

Changed

• scripts/build_hmm_profiles.py is marked SUPERSEDED and warns on every run. It groups by the polymer= header tag, which is the architecture of the tool this project is trying to beat: F0001, F0002 and F0003 carry 19, 19 and 21 distinct polymers, so a polymer-grouped build cuts each into that many fragments and reassembles them across non-homologous families by label. It is not deleted only because app/workers/pipelines.py still reads the PlasticDB.hmm it writes; it goes when Phase 7 wires the family build into app/workers/reference_build.py.
• make hmm-smoke now runs Phases 1 and 2 — twice. The fixture's profiles are built into two directories and their digests compared, which is the determinism claim checked rather than asserted. It is still the build job in CI, so mafft, t_coffee, clipkit and hmmbuild now execute on every pull request.

Fixed

• Two ways a profile build silently fails to be reproducible, both live. hmmbuild stamps a wall-clock DATE into every model it writes and records the absolute path it was invoked with in COM — through a temp directory, that differs on every single build. Either alone guarantees two builds of the same seeds under the same pinned toolchain produce different files, which makes "is this release reproducible?" unanswerable by comparing checksums. canonicalise_hmm drops the date and rewrites COM to the logical command. Verified: a second independent build of the whole corpus reproduces all four digests.
• T-Coffee rewrites protein:412 to protein_412 in score_ascii, so its per-sequence scores came back keyed on an identifier matching nothing else in the pipeline — silently, because the overall score is still correct. Those per-sequence numbers are the diagnostic that names which members drag a family below the gate. parse_tcs undoes it.

Deliberately not done

• No calibration. Profiles carry no GA/TC/NC and canonicalise_hmm raises if one appears. The tempting move is to fit a cutoff from the 150 curated negatives sitting in the Phase 1 output; it would encode "does not degrade PET" as "is not a member of the polyesterase family". Those are activity negatives and most are genuine family members — **130 of the 150 are hit by these profiles, which is correct behaviour**. Phase 3 fits the cutoffs against all four negative tiers.
• No discovery campaign. The profiles were built from the curated corpus and searched back against it. Nothing was screened against nr or MGnify.

Added

• Matrix B — the engineered-variant Δ dataset. The discovery label matrix excludes engineered variants on purpose: a parent and its mutants are 99%+ identical, and Phase 0 measured AUROC climbing 0.63 → 0.825 on nothing but train/test identity overlap. But variants are the only controlled comparisons the corpus holds — same scaffold, a few substitutions, same assay, same figure — so they now get their own dataset with their own unit: (parent, variant, polymer) → Δ, fold change, direction, built by scripts/export_mutation_effects.py on app/services/mutation_effects.py and written up in docs/mutation-effects-dataset.md. Like the seed export, fetch is the only stage that touches the database, it opens the connection readonly=True, and it never reads, writes or filters on visibility — these are build artefacts, the embargoed rows are a large share of the measured negatives, and the release filter is Phase 8. The exporter refuses to write into a snapshot directory.
• resolve_parent_frame verifies a coordinate frame instead of trusting one. ProteinVariant.numbering_offset indexes the variant's stored sequence and is not guaranteed to index the parent's. Each candidate offset — declared, identity, ± the signal-peptide cleavage site — is applied to the parent sequence and accepted only if every mutation's wild-type residue is found where it claims to be; a variant that resolves on nothing is exported with frame=unresolved and no coordinates at all. BbPETase-CD is why: its declared offset of −156 correctly indexes its own catalytic-domain construct and puts P207 on a cysteine of the parent. Trusting the column would have mis-placed nineteen mutations across five variants, each onto a real residue of the right protein, without raising anything.
• **Stage 3 prototype: measured substitutions projected onto a family-reference column**, so a novel profile hit can be told *"this position is known to have been engineered, and you carry the wild-type residue."* Pairwise Needleman-Wunsch with BLOSUM62 and affine gaps, written out rather than adding biopython to the image for a 24×24 table. 697 family positions over 15 families; only the 116 with single-mutant evidence are labelled attributed.

Measured

• **332 matched (parent, variant, polymer) triples — the feasibility question is answered yes. 959 Δ rows over 300 variants, 42 parents, 43 papers**, of which 156 variants are single mutants, from a corpus of 745 variants over 85 parents. Relaxing temperature, pH and duration from the condition key gives 1341 rows over 395 variants. Every one of the 2742 Mutation rows is fully structured — there is no free-text-only mutation problem.
• The kinetics tables cannot carry this dataset and the observations can. Of 959 rows, 859 come from observations (monomer_released 491, activity_detected 209), 83 from enzyme_properties (61 of them Tm) and 17 from enzyme_kinetics. A kcat/Km-based Matrix B would be a table of seventeen numbers.
• Matrix B is a PET-engineering dataset with a fringe, and says so: PET 218 of 300 matched variants, then PHB 30, PBAT 13, PLLA 13, PCL 11. Directions run 537 improved / 358 reduced / 36 unchanged — the reduced rows are a label no public dataset carries.
• The highest-yield curation target is unit and duration normalisation. 308 variants carry a measurement but find no wild-type twin. The exporter diagnoses each failure field-by-field into summary.json: 1170 unmatched measurements come from papers that reported no wild type of that parent and metric at all (an unrecoverable floor), but **685 have one and disagree on a condition field — 369 of those on nothing but unit and/or duration**. That is a normalisation pass, not a re-extraction, and it would grow the dataset by roughly a third.
• Slightly over half the variant corpus hangs off an embargoed scaffold. The label matrix established that the public subset carries zero curated negatives for any polymer; Matrix B is a different table, so it was measured rather than assumed. **397 variants over 36 held parents against 348 over 49 public parents** — the concentration runs the wrong way because the deeply engineered scaffolds are the ones under hold. A reconstruction from published data loses roughly half the (parent, variant, polymer) triples and all of the surrounding negative evidence. Recorded as what a competitor could rebuild, not as a justification for the embargo: the release filter is still Phase 8.

Added

• **Matrix A — the label set is a (protein, polymer) matrix with three states, not a list of positive and negative proteins.** The plan counted "577 positives / 283 negatives"; both are counts of proteins, and a label lives on a pair. **47 proteins in this corpus are positive on one polymer and negative on another**, so a protein-level label destroys one of those two facts and does not say which. The evidence spine has always stored it correctly (interactions.protein_id × plastic_id × result) — only the ML framing collapsed it. scripts/export_label_matrix.py + app/services/label_matrix.py un-collapse it; written up in docs/label-matrix.md. 903 labelled pairs: 620 positive, 273 negative, 10 conflicting, over 557 wild types and 51 polymers. The states are positive / negative / untested, and untested means absent or masked, never zero: 903 of 29 427 cells were ever measured, so imputing the rest as negatives would fabricate 28 524 labels and teach a model the literature's attention rather than any property of an enzyme. label_matrix.tsv contains no untested rows; the wide grid leaves the cell empty and says so in its header. Four exclusions, each measured rather than argued. **Engineered variants are out entirely** — 745 variants of 85 parents at 99%+ identity, and Phase 0 measured AUROC climbing 0.63 → 0.73 → 0.825 purely as train↔test identity rose 28% → 63% → 81%, so admitting them makes any identity split meaningless. Variant-only evidence does not propagate: 41 cells where the wild type was never tested but a variant was stay untested, carrying a pointer. Organism-level evidence is out (3 476 of the spine's 5 735 interactions). Conflicts are kept as conflicts — all 10, with both arms and both sets of conditions; in none of them does a condition set recur on both arms, so a majority vote would have deleted a real distinction.

Measured

• **Every negative pair in the database is on an embargoed row — for every polymer, not just PET.** Phase 0 established this for PET's 147; across all 51 polymers the public subset is 456 positives and zero negatives, against 273 negatives and 10 conflicts under a hold. A matrix built after a visibility filter would not be a smaller matrix, it would be a matrix with one class and no way to notice. The build reads the full corpus read-only; the release filter stays at Phase 8 and the artefacts refuse to be written under snapshots/ or static/.
• 73.9% of negative pairs carry substrate form, pH and temperature together (92% / 89% / 82% individually), which is what makes a cross-paper negative comparable at all — a negative is a statement about conditions, not about an enzyme. It is uneven where it matters: **PET's negatives are 88% fully specified, PCL's are 23%**. Deeper fields fall off as expected (duration 80%, crystallinity 46%, molecular weight 16%). aerobicity = 'not_reported' counts as absent, because counting it would report the schema being filled in rather than the paper having said anything.
• Evidence tiers, counted in pairs: 4 polymers reach Tier 1 (PET, PCL, PLA, PHB), 8 reach Tier 2, 39 are Tier 3. **Nylon 6 misses Tier 1 by one negative pair** — the cheapest available increase in this pipeline's coverage is one published nylon negative. PBAT, Impranil and Nylon oligomer hold no negatives at all and cannot leave Tier 2 however many positives accumulate.
• Source concentration, re-measured on pairs. One paper supplies evidence for 211 of the 413 PET pairs (51.1%, and it is the sole source for 210 of them), 21% of all labelled pairs, and 34% of every negative pair in the database. 630 positive pairs come from 156 papers; 283 negative pairs come from 38. Leave-one-paper-out belongs beside the identity splits.
• 341 labelled pairs sit on 185 proteins with no stored sequence and are dropped from the trainable set — counted in report.json, never silently, and kept by --include-unsequenced.

Changed

• docs/hmm-pipeline-plan.md — three errors our own measurements exposed. §3.1 no longer calibrates family thresholds against activity negatives. A GA/TC/NC cutoff answers "is this sequence in this family?", and a PET-negative protein is a genuine family member — that is why somebody tested it. The profile should hit it. Calibrating until the threshold rejects our curated negatives would teach it to reject true homologs and encode an activity claim inside a membership model. N3 is now "same-superfamily-but-different-family" (ESTHER non-polyesterases, confusable α/β-hydrolase families, top-BLAST-ratio non-members), and family calibration uses N0–N3 only. The 150 all-negative proteins became a per-profile specificity report ("this profile hits 7 of the 150 tested-inactive proteins"), published in THRESHOLDS.md and never an input to a cutoff. Their real home is Phase 6 / Layer C, where the unit is the pair and "PET negative, PCL positive" survives. §6.2's 25%-identity split is gone — Phase 0 measured that 99.5% of the labelled set falls in one connected component at 25%, so there is nothing to hold out and a plan mandating an impossible split gets an evaluation that quietly does something else. It is replaced by leakage-reported evaluation (every metric prints the achieved train↔test identity beside it, at three points on the leakage axis), leave-one-paper-out, per-polymer heads gated by the evidence tiers, and a masked multi-label loss so one protein contributes PCL = +1 and PET = −1 in the same pass and nothing at all for the polymers nobody tested it against. Phase 1 now records why exactly 150 proteins leave the seed set: they are negative for everything tested and so contribute no positive evidence for any polymer. The old "283" counted proteins carrying any negative — a different set, which would have deleted 51 plastizymes from the seeds over a negative on some other polymer. The label matrix reached the same 150 from the other direction, which is the same statement written in the unit that loses nothing.

Changed

• **The catalytic Pfam panel is 29 profiles, not 11, and the largest thing it was missing was the PETase family itself. Phase 1 measured that 46% of the corpus (266 of 577) finds no catalytic domain** at --cut_ga, keeps its full chain, and therefore cannot meet the family clustering's 80% coverage requirement against a neighbour that was trimmed to a domain — median 320 residues against 169. Rather than guess a wider list, all 577 mature chains were scanned against the whole of Pfam-A (30 134 profiles, same --cut_ga) and the families the missed proteins actually belong to were read off the result: scripts/experiments/catalytic_panel_scan.py, written up in docs/phase1-catalytic-panel.md. PF12740 (PETase) was the best-scoring hit for 153 of the 266 — and for a further 204 that the old panel caught with a weaker, shorter model. A PET-hydrolase database was screening its corpus for cutinases and not for PET hydrolases. Behind it: NylA amidases (PF01425), family VIII esterases (PF00144), NylC (PF03576), the PLA-degrading subtilisins (PF00082), PHB depolymerases (PF10503), the Cbotu_EstA/PlaM4 polyesterases (PF24708). A family is admitted only when it is the best-scoring Pfam-A hit for a protein the panel misses, Pfam's own description makes it the catalytic domain, and that catalytic unit is one domain — so fn3 and HemolysinCabind went to ACCESSORY_PFAM instead (in the catalytic panel, PueA would have been "trimmed" to a 35-residue RTX repeat), and the multi-domain oxidative enzymes (DyP peroxidase, laccase, PQQ β-propeller, ADH) are deliberately left whole. Domain detection 311/577 (54%) → 537/577 (93%); accessory modules excised 84 → 96; trimmed and untrimmed median lengths now 258 and 264 residues, so the corpus is finally compared like with like.

Measured

• **Widening the panel does not buy more profiles, and the singletons were not the artefact.** Same corpus, same parameters, only the panel changed: families 103 → 94, singletons 76 → 69, families clearing n ≥ 8 6 → 3, seeds inside those 49.8% → 51.0%. **64 of the 76 singletons are still singletons** — the long tail is genuine sequence diversity, not a coverage artefact, and Phase 2 must plan for it permanently. What the length mismatch had been doing was splitting the large families: five of the six gate-clearing families were one family cut into pieces by the coverage requirement and are now a single **130-seed family holding 42% of the deduplicated corpus** (PET, PCL, PBAT, Impranil). It stays above 40% at every point of the identity × coverage sweep including --min-seq-id 0.40, so it is real homology rather than a threshold artefact. **Phase 2 should plan for three profiles, not six**, and expect §2.3's TCS quality gate to fire on that family first. Reported against ourselves: trimming also fragmented one genuine family. The 15 nylon amidases that cleared the gate as full chains split into 6, 6 and 3 once trimmed — three members have a ~300-residue Amidase envelope against the others' ~440 (0.67 coverage, below -c 0.80), and the rest separate at 30% identity once the flanking regions carrying the alignment are gone. Envelope trimming is not free. Determinism is unaffected: three consecutive cluster runs give a byte-identical families.json, and two consecutive export runs give byte-identical seeds.faa, negatives.faa and accessory.faa. The run read the full 577-protein corpus through a session the server held readonly, SELECT only; no visibility or visibility_hold_* column was read for filtering or written at all.

Added

• **A microorganism can now be withheld from public view and put back exactly as it was — the hold mechanism proteins have had since 0.88, generalised.** An operator with 296 enzymes under embargo could be certain no promotion path would publish one of them, and could say nothing of the kind about the strain the enzyme came from: Microorganism.visibility is written by the same paths (deposits.create_deposit / release_deposit, the daily release_due_deposits task, submission_promotion._cover) and read by the same gates, but there were no hold columns on microorganisms and the before_flush parking guard read isinstance(obj, Protein) (#431). The five columns are now a VisibilityHoldMixin carried by both entities — same names, same reversibility (prior state stored, batch-tracked, manual beats bulk, both directions idempotent) — and the flush guard keys on the mixin, so the next holdable entity is covered the day it is declared rather than the day someone remembers to add it to a class list. The admin dashboard's Visibility tab grew the matching pair of switches, the bulk one selecting every strain whose evidence includes a negative result through either arm the rest of the codebase credits an organism by: its own whole-organism assays, and the cell-free assays of enzymes purified from it. **A hold does not cascade, deliberately, and the reasoning is written down in app/services/visibility_holds.py.** Withholding a strain leaves its enzymes exactly where they were: an enzyme is an independently accessioned record with its own switch, and a cascade would need a second kind of "prior" — *held because of a parent* — which is a second idiom for the one concept these columns exist to express. The organism list reports how many source proteins stay public per row, so the choice is visible in the UI rather than discovered later. A held organism's name is likewise not scrubbed from the public enzyme rows that point at it: a species name is in NCBI, in the enzyme's GenBank entry and usually in the paper's title, and blanking a column of an otherwise-public record is less honest than either state. This ships the capability only. No record's visibility or hold state is read, moved or inferred by the release; whether any organism is ever held is the operator's decision, made from the dashboard.

Fixed

• **The microorganism detail route, its two page routes and the anonymous autocomplete had no visibility gate at all.** The list filtered, the search filtered, the exports filtered, the release snapshot filtered — and the detail page every one of them links to served an embargoed, private or tombstoned strain to anyone who guessed its tax-id or accession. /microorganisms/names returned the whole table, held rows included, to unauthenticated callers, and /microorganisms/search did the same for anything matching the query. All four now 404 or omit, exactly as the protein and engineering routes do. Found by the first route-level test written against a held organism.

Changed

• Migration 0011_micro_visibility_holds adds the five hold columns and their two indexes to the populated microorganisms table, and renames protein_visibility_hold_reason_enum to visibility_hold_reason_enum — metadata only, no table rewritten and no value changed. The type's two labels never said anything about proteins, and one shared type is what lets the mixin be a mixin rather than a per-table copy. Release order is unchanged (python -m app.db.init_schema, then alembic upgrade head); the downgrade moves the type back so a rollback cannot strand a database between the two revisions. As in 0010, a downgrade forgets what a hold changed rather than releasing it — release from the dashboard first if anything is withheld.

Fixed

• **mmseqs easy-cluster writes *_rep_seq.fasta in a different record order on every run, and the family clustering that reads it is order-sensitive.** The set of representatives is identical — verified by hashing the sorted headers across runs — but greedy set cover (--cluster-mode 0) gives whichever sequence it sees first the chance to become a representative and claim its neighbours. On the full 577-protein corpus, three consecutive runs of identical inputs and identical parameters produced 102, 103 and 104 families, with 5, 5 and 6 of them clearing the Phase 2 admission gate. Nothing reported it. --threads 1 does not fix this; the non-determinism is in the output order, not in the clustering arithmetic. canonicalise_fasta now rewrites the representative FASTA in a fixed order (by typed sequence id) before the family clustering, and three consecutive runs give 103 families and 6 gate passes every time. A profile-HMM release whose family partition changes between two builds of the same corpus is not reproducible in the sense the pipeline claims to be, and the Phase 8 manifest would have recorded a partition nobody could regenerate.

Changed

• docs/phase1-seed-families.md now reports the full corpus. The previous figures came from the 345 public wild types, which was all the build host could reach; the run has been repeated read-only against all 577 sequenced wild types (357 public, 220 under a visibility hold). The public-subset table is kept alongside, clearly labelled, because the contrast is itself a finding. Authoritative numbers: 150 curated negatives separated, 427 seeds, 313 after 90% dedup, 103 families, largest 58 seeds (18.5%), **76 singletons (74% of families), and 6 families clearing n ≥ 8 — holding only 49.8%** of the seeds. Half the deduplicated corpus falls outside any profile-eligible family, which is the number Phase 2 has to plan around. Against the public subset: the embargoed rows add breadth, not depth — 29 more families, 21 more singletons, and no additional profile-eligible family. What moves is the denominator: gate-clearing families cover 62% of the public subset and 50% of the real corpus, so anyone estimating coverage from our published data overstates it by a quarter. The public subset also yields **zero negatives of any kind** (456 positive, 6 inconclusive out of 462 interactions), confirming Phase 0's finding from the outside. D1 holds empirically, and more strongly than on the public subset. F0004 carries 19 distinct polymers, F0002 and F0007 carry 17, F0012 carries 14 — single homologous families whose members were each measured against a different substrate panel. A polymer-grouped build would cut each into a dozen fragments and reassemble them across families by label; the resulting "PET profile" would span eleven non-homologous groups. The converse also holds: F0005/F0011 are clean nylon-hydrolase families and F0006/F0008/F0013 clean PET ones. Signal-peptide stripping finally ran — the public feed carries has_signal_peptide but not the cleavage site, so the earlier run could not apply D2's mature-chain rule at all. All 351 were stripped. A/B on the full corpus: four more sequences collapse at 90% dedup (313 vs 317), eight more proteins find a catalytic domain at --cut_ga (311 vs 303), and two families and one gate pass disappear — a shared secretion motif had been making unrelated secreted proteins look alike, and the gate should not be cleared on the strength of a targeting sequence. Two further measurements that change the plan: **46% of the corpus finds no catalytic domain** and keeps its full chain, so it cannot meet an 80% coverage bar against a trimmed domain — a share of the 76 singletons is that artefact, and widening the Pfam panel is the cheapest available improvement. And **Phase 3.1's "283 curated negatives" is not the measured number**: 150 negative-only proteins (the tier as exported), 201 carrying any negative, 455 negative interaction rows. The 51 proteins carrying both stay in the seed set, because they demonstrably degrade some polyester and putting them in negatives/ would push the calibrated NC up and every profile's GA with it. The run wrote nothing: SELECT statements on a session the server held readonly, and no visibility or visibility_hold_* column touched. Verified afterwards — 296 private proteins, all bulk_negative_results, unchanged.

Fixed

• **The release build had no retry on the three binary downloads, so a transient connection reset from GitHub's release CDN failed the deploy.** v0.95.0 failed twice in a row, both times identically: `curl: (35) Recv failure: Connection reset by peer, 150 s into a 17 MB mmseqs-linux-avx2.tar.gz`. Since v0.94.0 the image pulls MAFFT, MMseqs2 and foldseek from the open internet at build time, and build-and-push.yml is the only path to production — so two third-party hosts became a hard dependency of every deploy, with no retry between them and a failed release. Every download now carries `--retry 5 --retry-delay 5 --retry-all-errors --connect-timeout 30 --speed-limit 10240 --speed-time 30`. --retry-all-errors is the load-bearing flag: plain --retry does not cover curl exit 35, which is the failure we actually hit. The speed floor aborts a stalled transfer after 30 s so it is retried, instead of hanging for 150 s until the far end resets it. The SHA-256 pins are untouched and cannot be weakened by a retry: a truncated or substituted file still fails sha256sum -c. Closes #436.

Added

• Phase 1 of the profile-HMM pipeline: seed export and family definition (docs/hmm-pipeline-plan.md §1.2–1.4). Result document: docs/phase1-seed-families.md. Three pieces — app/services/hmm_seeds.py (pure logic, Tier-1 testable), scripts/export_seed_fasta.py (corpus → seeds, negatives, accessory modules) and scripts/build_seed_families.py (90% dedup → families → structural cross-check → report) — plus make hmm-smoke, which runs the whole thing on a frozen fixture and checks it against a golden partition. Families, not polymers (D1). Nothing groups on a polymer tag; polymer is counted within a family and reported as an annotation, with each verdict attached to its own polymer so that a PET-negative, PBAT-positive enzyme keeps both facts. The measured partition is the argument for that decision: the four largest families each carry three or four different polymers, and PET appears in at least eight separate families. A polymer-grouped build — which is what scripts/build_hmm_profiles.py still does, left in place and not extended — would have cut those families apart and then glued their PET fragments into one "PET profile" spanning four non-homologous groups. The empirical family table (345 public wild types, the full corpus was not reachable from the build host — see the result document): 250 seeds after 90% dedup, 74 families, largest 23.2%, 55 singletons, and only **6 clear the Phase 2 n ≥ 8 gate**, holding 62% of the seeds. The --min-seq-id × -c sweep moves that between 5 and 7 across the whole grid, so the clustering threshold is not what limits this pipeline; corpus diversity is, which is the same conclusion Phase 0 reached from the other direction. No winner is selected — the plan selects by leave-one-out recovery and hard-negative separation, which are Phase 2/3 measurements. Three assumptions in the plan turned out to be false and are handled rather than worked around:
  • There are no InterPro/Pfam coordinates in the database. InterPro data is stored in proteins.cross_references as {id, name, type, db}; the enrichment step drops the per-entry locations, and there is no protein_domains table. --domain-envelope therefore computes the envelope with hmmsearch --cut_ga --domtblout against the catalytic Pfam panel, taking the envelope columns (20/21), not the alignment ones.
  • There is no per-residue pLDDT. protein_structures.confidence is the mean over the model. --plddt-trim reads the B-factor column of the stored structure instead, and leaves a protein untrimmed — and counted — when the model cannot be read, rather than trimming against an average.
  • **The seed export is a new script, not flags on scripts/export_reference_fasta.py.** That one builds the public DIAMOND reference database and filters visibility == "public"; this one reads the full corpus on purpose, because every PET negative is embargoed and filtering at Phase 1 would delete a class rather than shrink the corpus. One wrong default in a shared script publishes 296 embargoed sequences, and this project has already leaked an embargo three times through three different promotion paths. The seed writer additionally refuses to write under a snapshots/ or static/ path.
• make hmm-smoke, wired into the build CI job (.github/hmm-smoke.Dockerfile). The Tier-1 image carries pytest but no bioinformatics binaries and the application image carries the binaries but no pytest, so every test that calls mmseqs would otherwise skip in both and the clustering code would never run in CI at all. The smoke test runs against the image the build job just built.

Fixed

• mmseqs silently reuses an existing tmp directory, keeping a latest symlink and skipping work it believes it has already done. Running the cluster and sweep stages into the same tmp produced two different answers for identical parameters (6 versus 7 profile-eligible families) because the sweep was reusing the previous run's prefilter. Both stages now wipe their tmp directory first. Left alone this would have published a family table that no parameter combination actually produces, with nothing reporting it.

Added

• **The image carries the profile-HMM toolchain: mafft, mmseqs2, foldseek, clipkit, pyrodigal** (docs/hmm-pipeline-plan.md Phase 7). Every version is pinned exactly, and the three downloaded artefacts are additionally pinned by SHA-256, because a profile HMM is a function of the seed set and of the toolchain: mafft --auto picks a different strategy across releases and hmmbuild's default entropy weighting has moved between HMMER minor versions, so two builds of the same seeds under two toolchains produce two different profiles with nothing anywhere reporting it. HMMER is pinned to 3.4 and MAFFT is taken from the author's own .deb at 7.526 rather than Debian's 7.505 — 7.526 is what the Phase 0 experiment measured on. MetaGeneMark is deliberately absent: it is licensed per registered user, so it cannot ship in a redistributable image, and pyrodigal is the substitute — a better one here, since it calls ORFs in-process rather than a subprocess per shard. tests/test_bio_tools.py::test_metagenemark_is_not_shipped keeps it out. Licences were verified upstream, not assumed: MMseqs2 is MIT; **foldseek and pyrodigal are GPL-3.0**, as is the already-bundled DIAMOND. All three ship as unmodified upstream builds invoked as separate processes (or, for pyrodigal, as an independently-licensed library), so they do not relicense PlasticDB; the manifest's source field is the corresponding-source pointer GPL-3.0 asks for. One consequence worth knowing before a deploy: **from release 10 foldseek publishes only an AVX2 build for linux/amd64**, so the image now requires an x86-64 CPU with AVX2. On a CPU without it the binaries die with SIGILL, and the build reports exactly that rather than leaving it to surface inside a Celery task three phases later. Full detail, including how to roll foldseek back to the last SSE2 release: docs/bioinformatics-tools.md.
• **app/services/bio_tools.py — one manifest, three readers, so the toolchain cannot rot silently.** scripts/check_bio_tools.py executes every binary and runs as a RUN step in the image build, so an apt line, a download or a pip pin that only looked like it worked fails docker build — the build job in CI. tests/test_bio_tools.py (Tier 1, no Docker) asserts every pin in the manifest still appears verbatim in the file that installs it, catching the version edited in one place and not the other. Pipeline code resolves binaries through tool_path(), which names the phase that wanted the tool instead of raising a bare FileNotFoundError.

Fixed

• **docker-compose.yml mounted bio_tools :ro on the worker, which is the one service that has to write it.** The worker runs rebuild_reference_databases (app/workers/reference_build.py), which mkstemps into that directory and os.replaces the result into place — the atomic swap readers depend on. Read-only, that is EROFS inside a Celery task, so no worker-driven reference build could be tested locally at all, and the profile-HMM build joins the same _BUILDS list in Phase 7. The stale comment claiming the API needed write access "only" for make bio-build is gone with it. Nothing else is loosened: job scratch still lives in job_files, and it is the atomic-replace discipline that protects a concurrent reader, not the mount flag.

Changed

• The chart-executor sidecar pins numpy==2.0.2 instead of 2.1.1, so one numpy serves the whole project. ClipKIT caps numpy<2.1 on Python 3.12 and requirements-dev.txt pulls the sidecar's pins in so test_chart_sidecar.py runs against the versions the sidecar image ships; at 2.1.1 that resolution is impossible and .github/test.Dockerfile could not build at all. Both are numpy 2.x and matplotlib 3.9.2 supports either.
• .github/test.Dockerfile now copies docker-compose.yml, which is data to the new mount test — the same reason it already copies Dockerfile.

Fixed

• **/api/v1/search returned 500 for any query matching an organism with no NCBI tax-id (#408).** The microorganism arm of the search UNION selected tax_id straight into GlobalSearchResult.route_key, which is non-optional. A null therefore failed Pydantic validation and took down the entire response — one such row anywhere in the result set, and the site-wide search box returned nothing. Fifteen production organisms were live triggers: ?q=tamsuii, ?q=circinatus, ?q=japonica and a dozen more each 500'd, through both /search and /search/grouped. A null tax_id has been a sanctioned state since 0.83.0 — a validly published species NCBI has no node for — so the fix is not the one the protein arm uses (drop the row). A tax-id-less organism does have a page, /microorganisms/accession/{accession}, shipped in 0.82.0 and already linked from the microorganism list. Search now routes on coalesce(tax_id, accession) and carries the accession page in detail_url, so it agrees with the list page instead of contradicting it. The 15 rows are findable rather than merely non-fatal — they are the ones a curator most needs to find, each carrying a recorded cause (absent_from_ncbi, misspelling, obsolete_name, not_validly_published) and an invitation to revisit it. /search/autocomplete never 500'd because it already took the other option — .where(route_key_col.is_not(None)) silently hid these rows from the dropdown. It now suggests them too, with the route the dropdown can follow.
• **Six templates built a dead /microorganisms/null link for the same organisms.** /microorganisms/{tax_id} takes an int, so an unguarded ` /microorganisms/${row.tax_id} ` 422s for a tax-id-less row — and three of the six guarded the displayed tax-id (${m.tax_id || '–'}) while leaving the link unguarded, rendering an en-dash inside a link to nowhere. The fallback the microorganism list had kept to itself is now shared as organismHref / organismLink in static/js/organism_name.js, and used by the protein detail and list, plastic detail, paper detail, taxonomy group and paper footprint pages. Two payloads gained the accession they needed to build it: FootprintMicroorganismRead and the taxonomy group's top_microorganisms. The dashboard search dropdown (base_dashboard.html) also ignored the detail_url the API sends, so it linked engineered variants to /proteins/{id} as well; it now honours it.

Fixed

• **Any authenticated user could publish arbitrary public proteins through POST /api/v1/deposits (#417).** DepositCreate.visibility was a bare str with no validator, and the route copied it verbatim onto the Deposit and onto every Protein and Microorganism it created. So one call from any verified account put rows into the public corpus — the set workers/reference_build builds the DIAMOND databases from, that services/export and services/snapshots read, and that a release snapshot is cut from under a Figshare DOI that cannot be unpublished. No curator saw them. It was also a hole in the operator embargo currently withholding 296 proteins. Two changes, and the second is the one that matters. The schema now validates visibility against the enum (a typo is a 422 instead of reaching the database enum as a 500), sharing one validator with schemas/submissions.py in schemas/common.check_visibility rather than keeping a second copy that was never written. And **moving deposit content into the public set is now a superuser act**, the same right deps.may_publish_ingestion reserves for a contributor's submission — publishing somebody's own unpublished data under their name has always been the narrow right. Enforced at all three doors, because leaving any one open makes the others theatre: POST /deposits with visibility='public' (403); POST /deposits/{accession}/release, previously open to the deposit's owner, so "create private, then release" was the same bypass in two calls (403); and the daily release_due_deposits beat, previously reachable by creating an embargoed deposit dated yesterday, i.e. the same bypass with a 24-hour delay. The beat now releases only deposits carrying a submission_id — the fact that they came through an acceptance a superuser had to make — and reports matured deposits it declines to publish to Slack rather than leaving a depositor waiting in silence. Depositors lose nothing they are owed: minting accessions, holding records private or embargoed, and the anonymous reviewer link all work exactly as before. Verified in the pg tier, where the assertion is made against public_visibility_filter itself — the clause a snapshot is built from — not against a status code. The before_flush guard in models.py was confirmed empirically (not assumed) to park a held protein's publication through the release route as well, so the 296 embargoed rows cannot leave by this door either.

Added

• **Phase 0 of the HMM pipeline: the within-family PET discrimination experiment.** Before any profile-HMM code is written, one question decides what Layer C is allowed to claim — inside a single homologous family, does anything computable from sequence distinguish an enzyme measured to degrade PET from one measured not to? We can ask it honestly, which almost nobody can: every published plastizyme classifier trains its positives against homologues of unknown activity, a near-circular label set. We hold real negatives. scripts/experiments/pet_discrimination.py (fetch / features / evaluate, --help on each) answers it, and docs/phase0-pet-discrimination.md records what it found. Labels come from the evidence spine — interactions.result against the plastic row named exactly PET. Never an ilike: app/services/query.py filters plastics by substring, which is right for a search box and wrong for a label, because %PET% also matches PETG and %PE% matches LDPE and HDPE too. The gate resolved: nothing separates them. Layer C will report "polymer undetermined" for PET and will not ship a calibrated PET discriminator. Across six feature blocks — profile HMM bit score and envelope coverage, the M1→M5 active-site ladder, thermostability, accessory modules, nearest characterised neighbour, ESM-2 embeddings — and two model classes, the best out-of-fold result under the hardest split the corpus permits is **AUROC 0.628 / MCC 0.21 / AUPRC 0.740**, against a prevalence floor of 0.639 and a label-permutation null that reaches 0.561. The profile HMM, the feature the whole pipeline is built on, scores below chance. Two findings the plan did not anticipate. First, **the 25%-identity split the plan mandates does not exist on this corpus**: 68% of the 82,621 labelled pairs are ≥25% identical, and single-linkage clustering at 25% leaves **99.5% of the 407 labelled proteins in one connected component**. There is nothing to hold out. Phase 6.2 mandates the same split for the ML rescorer and cannot have one either, so that requirement is flagged for rewriting. Second, **apparent performance is a monotone function of train/test similarity** — the identical pipeline reports AUROC 0.63 at 28% median train↔test identity, 0.73 at 63%, and 0.825 at 81%, i.e. under the random split most published plastizyme classifiers use. The experiment therefore reports the achieved leakage next to every metric, which is the number that decides whether an AUROC means anything and which almost nobody publishes. Features that could not be run are implemented, marked not_run with a reason, and reported as such — never silently dropped, and never given a fabricated number. DeepTM did not run (no PyPI package, no public weights endpoint); what runs in its place is a panel of published sequence-level thermostability correlates, prefixed thermo_ so it can never be mistaken for a predicted Tm. ESM-2 is the 35M checkpoint, not the 650M named in Phase 6.1 — the larger model loads but needs about an hour per pass on a CPU-only host.
• The corpus-scope rule, now written down. The build reads all 577 sequenced wild types; the release filter runs at Phase 8, not Phase 1. Phase 0 measured why this is not a preference: **all 147 PET negatives in the database are embargoed, and the public subset holds 249 positives and zero** negatives. The hold was applied as a curation batch (visibility_hold_reason = 'bulk_negative_results'), so filtering on visibility before Phase 8 would not shrink the corpus evenly — it would delete one class outright, and every threshold, negative tier and calibration downstream would inherit that silently, with nothing failing loudly. A profile is residue frequencies, not sequences, so nothing is leaked by building over held rows. Recorded in docs/hmm-pipeline-plan.md under "Scope of this build", along with the rule that no build-side query filters on or writes visibility and that sequence-bearing intermediates are build artefacts, never deposits. The experiment reads production read-only — a session opened readonly=True, SELECT statements only, pinned by a test that greps the four corpus queries for write verbs and for any mention of a visibility column.

Changed

• docs/hmm-pipeline-plan.md. Phase 0 marked run, with the re-measured counts (260 positive / 147 negative, against the plan's pre-embargo estimate of 264/151). Phase 1.1's SignalP blocker marked cleared — measured against the live corpus, all 577 now carry a resolved has_signal_peptide (351 true / 226 false) and all 351 signal peptides have a stored cleavage site, so Phase 1.2's --strip-signal-peptide has what it needs. Structural coverage has also moved to 406 of 577, from the 386 quoted in Phase 2.
• requirements-dev.txt gains scikit-learn, so the experiment's threshold-selection and cluster-bootstrap tests execute in CI instead of skipping past themselves — the same reasoning already applied to the chart-executor pins.

Added

• A sequence is now a lookup key. Every accession-keyed enrichment source — UniProt metadata, RCSB, the EBI AlphaFold DB, InterPro — needs a uniprot_accession before it can do anything, and **365 of the 577 sequenced proteins in production have none**. 175 of those have no structure of any kind, and 101 of them are longer than 400 residues — past the public ESMFold endpoint's hard ceiling, which refuses a 450-residue sequence with an immediate 413. For those, AlphaFold was not the preferred route to a structure; it was the only one, and it was closed. The new uniparc rung opens it, from the residues themselves: **CRC64 of the sequence → UniParc entry → active UniProtKB cross-reference**. UniParc is the archive UniProt indexes by sequence, so a protein with no accession still has a key. Once the accession lands in the column, the rest of the ladder works unchanged — and does so in the same pass, because the rung is placed after UniProt and before everything that reads the column (#424). Attribution is exact identity, checked twice. A CRC64 is only an index — 64 bits, and collisions exist — so the UniParc entry's sequence is compared to ours character for character before it is accepted at all. Then only an active cross-reference is used: active is UniParc's way of saying "this accession currently holds this exact sequence", and an inactive one means the entry has since been deleted or re-sequenced. Nothing is ever attributed on a partial or near match. The CRC64 implementation is pinned in the test suite against four checksums UniProt publishes itself, because the wrong variant does not fail — it silently matches nothing, which reads exactly like "there is nothing to find". Every resolution carries its provenance: the enrichment row's detail reads `resolved A0A0K8P6T7 (UniProtKB/Swiss-Prot) via UPI0006A8DE61 on exact crc64 8620A766B69749D2`, and the UniParc id is stored as a cross-reference and linked from the protein page. A value already in uniprot_accession is never overwritten — that came from a curator or an importer, and the rung reports what it found instead.
• scripts/backfill_uniparc_accessions.py — the one-time catch-up over the corpus that predates the rung. Resumable by construction (a protein stops being a candidate the moment its uniparc row settles, so an interrupted pass is continued by re-running it), --limit to bound a pass, one protein per commit with commit_with_deadlock_retry, and --report-fingerprint to show that the reference-database fingerprint has not moved. It covers private records as well as public ones — a structure is internal enrichment, not publication — and writes to no visibility column in any branch.

Fixed

• UniProt now reads an accession that is already in the column. Its only routes in were a GenBank id and a BLAST search that is off by default, so a protein carrying nothing but a UniProt accession never got a name, an EC number, GO terms or active sites. It now fetches the entry directly, and ignores a 200 whose entryType is Inactive — a deleted entry has no sequence and no names, and parsing one would blank the columns it was meant to fill.

Added

• A way back from the retry cap. The enrichment state machine stops attempting a source once its row is transient with retries >= RETRY_CAP. That rule is right — it is what stops the weekly batch spending its whole budget on a record that will never resolve — but it had no exit. So when v0.87.7 fixed the ESMFold rung (the free api.esmatlas.com endpoint answers 504 at its gateway's ~29.6s ceiling for about half of all requests; the fix retries within one attempt and took per-attempt success from ~50% to ~97%), it could not reach a single row it was written for: 231 rows — 66 proteins and 165 engineered variants — were already past the cap, burned through by the ~50% era, and permanently parked (#421). Admin → Enrichment → "Rescue retry-capped rows" re-attempts one source for exactly those rows. The panel says how many there are before you press it, per source, and reads zero when there is nothing left. What it deliberately does not do is what already existed. "Re-enrich all" is force=True across all 8 sources × 762 proteins × 745 variants; it re-attempts the 139 rows already settled permanent: unfoldable — sequences over 400 residues, which the endpoint answers 413 to — and re-runs UniProt, PDB, InterPro and AlphaFold over the whole corpus to reach a few hundred rows. Only transient rows are ever selected here; ok, none and permanent are terminal and are never re-asked. Resumable by construction, not by a checkpoint file. A row's retry counter is reset immediately before its attempt, so a row that has been tried is no longer at the cap and a second pass does not select it again. Interrupt one and start another; it continues from the rows the first never reached. And a row that fails again comes back at retries=1 rather than at the cap, so the weekly batch keeps trying rather than re-parking it — the same trap, one release later, which is the failure this feature exists to end. New enrichment_rescue_runs table (ORM + python -m app.db.init_schema; no migration — create_all creates a table that does not exist yet), reusing the taxonomy_runs pattern shipped in 0.89.0 rather than inventing a third mechanism: a UNIQUE lock_key so two passes cannot contend for the free upstream that is the reason the rows were parked, counters and a heartbeat committed after every row, an automatic release of a lock whose worker died, and a visible history of what each pass recovered. Reasoning in full in the module docstring of app/services/enrichment_runs.py. python -m scripts.rescue_enrichment is the same thing without a browser — same lock, same run rows, same worker code — and reports, writing nothing, when given no --source.
• enrich_entity_sources — the ladder, restricted to named sources. A rescued rung still runs through _enrich_one / _enrich_one_variant, so it keeps the gates that make it correct (the "an experimental structure already exists" check, the AlphaFold-may-still-deliver wait, and the structure_file reconciliation that is what actually makes a recovered fold render).

Fixed

• The dashboard was still reporting v0.87.7's improvement as 522 errors. Busy buckets on endpoint_over_capacity, the tag EsmfoldOverCapacity writes — and that exception did not exist when any of the stored rows were written. All 522 esmfold transient rows in production carry the older shape, and every one of them is a 5xx from the free endpoint: ``` esmfold_failed: ESM Atlas returned 504: {"message": "Endpoint request …"} 416 esmfold_failed: ESM Atlas returned 504: Gateway Time-out 104 esmfold_failed: ESM Atlas returned 502: Bad Gateway 2 ``` So Errors read 522 and Busy read 0 — the exact crying wolf the new bucket was introduced to stop, still on the page. The coverage query now recognises both spellings. This is a reading rule, not a rewrite: rewriting the stored strings would either discard what the endpoint actually said or wrap it in a tag the row was never given, and neither is a thing that was observed. The match is narrow — the ESM Atlas rung by name, and only the status codes _ESM_ATLAS_RETRYABLE treats as "ask again shortly" — so esmfold_failed: ESM Atlas returned no ATOM records is still an error and still counted as one. The Retry-capped column's tooltip no longer advises "rescue with Re-enrich all", which was expensive and wrong.
• **A backfill died at 140 of 577 because it held row locks for two minutes at a time.** scripts/backfill_signal_peptides.py defaulted to --commit-every 10. A record is a ~12-second hosted EBI Phobius call, and upsert_enrichment reads entity.enrichments — a lazy relationship load, which triggers SQLAlchemy's autoflush, which takes the row locks on proteins at the first record of the batch and holds them until the commit at the tenth. Two minutes of held locks for perhaps five milliseconds of writing, against a table reindex_search_vectors bulk-UPDATEs on a schedule. PostgreSQL resolved the deadlock by killing the backfill, and with no retry the run simply ended. Both halves are fixed. --commit-every now defaults to 1, bounding what a deadlock can destroy to one prediction; and the new app.workers.sync_db.commit_with_deadlock_retry re-applies and re-commits that record rather than losing it. The prediction itself is never repeated — it has already happened, outside the transaction, and only the in-session writes are redone. Retries are keyed on the SQLSTATE (40P01, 40001), not on message text: a constraint violation retried four times is still a violation, and retrying it would hide the thing an operator has to see.
• scripts/backfill_plddt.py had the same shape, worse. row.protein inside its loop is a lazy load, so the autoflush took a lock on protein_structures at the first repair — and a single commit at the end of a one-second-per-row loop held it for the entire run. It now commits each repair through the same retry. (backfill_microorganism_synonyms.py was checked and is clean: it accesses no relationship and issues no query inside its loop, so nothing flushes until the final commit. backfill_accessions.py, backfill_submission_ingestions.py, migrate_variant_enrichment.py and enrich_dois.py make no network calls inside a transaction at all.)

Added

• Taxonomy enrichment has a button, and the button has a memory. Refreshing NCBI taxonomy was a person: kubectl exec into a worker pod, call refresh_lineages() by hand, and read the pod's log for the outcome — until the pod was recycled and there was no outcome to read. Admin → Taxonomy now starts the same pass on a worker, shows it moving, records what it did, and refuses to start a second one. Four scopes, cheapest first, because the difference between them is 1 NCBI request and ~1300: | Scope | Rows | When you want it | |---|---|---| | Missing lineage | organisms with a tax-id and no lineage string | after an import — currently 0 of 1282 in production, which is the point of showing the number | | Unresolved names | organisms NCBI had no node for (25 in production) | after NCBI adds nodes; the same rules as the weekly retry job | | Single organism | one | a curator fixed one name and wants it re-asked | | Full refresh | every organism with a tax-id (1282) | a genuine re-sync; ~7 minutes at the anonymous 3 rps, ~2 with an NCBI_API_KEY | No overlapping runs, enforced by the database. taxonomy_runs.lock_key is UNIQUE and non-NULL only while a run is live. Two clicks 200 ms apart both pass any "is one running?" SELECT; only one survives the insert. This is not fussiness — NCBI blocks the shared institutional IP, not the tab that double-fired, so a duplicate crawl is everyone at the university losing E-utilities. The weekly beat tasks take the same lock, so Sunday's 03:17 refresh will not land on top of a run started at 03:16. Progress, not a spinner. Counters and a heartbeat are committed every 25 organisms, so the dashboard reads *340 of 1282 checked · 12 updated · 2 failed* while the crawl is still going — and so a run whose worker was killed is distinguishable from one that is merely slow. A run that stops reporting for twenty minutes has its lock released automatically; a Release the lock button cuts that wait to five, because recovering from a dead worker must not require a terminal either. It is refused while the run is still reporting: nothing in the API can stop a Celery task, so freeing a live run's lock would start a second crawl beside the first. New taxonomy_runs table (ORM + python -m app.db.init_schema; no migration — create_all creates a table that does not exist yet). Deliberately not Job: that is a bioinformatics upload with fifteen DIAMOND-shaped columns, a results file, a user-facing status page, Slack notifications and a file-eviction sweeper, and its job_type is a live PostgreSQL ENUM whose `ALTER TYPE … ADD VALUE` cannot be used in the transaction that adds it. Reasoning in full in the module docstring of app/services/taxonomy_runs.py.
• The orphan sweep's reference map grew a third answer. delete_orphaned_organisms_and_proteins classifies every foreign key pointing at microorganisms.id as either keeping the organism alive or dying with it, and a completeness test refuses a new one that is neither. taxonomy_runs.microorganism_id is genuinely neither: a run about an organism must not stop that organism being reclaimed, and must not be deleted when it is — the run happened. The new _MICROORGANISM_AUDIT_TRAIL list says so out loud, and the FK's ON DELETE SET NULL is what enforces it.

Fixed

• The lineage refresh was pacing itself, badly, in a second place. app/workers/lineage.py built its own efetch URL and slept 1.1s every third request. That copy sent no tool/email/api_key — an unidentified caller is throttled harder and blocked without an email first — ignored Retry-After, never retried a 429, and ignored NCBI_API_KEY entirely, so the 10 rps allowance the key buys was left unused while the shared institutional IP wore the 3 rps one. Every request now goes through eutils_get, the one place that paces, identifies, retries and backs off.
• A throttled NCBI used to read as a clean run. The same loop treated every non-200 as continue, which made "NCBI is rate-limiting us" and "NCBI has no such taxon" the same event: a pass against a throttling NCBI finished successfully having updated nothing, and nothing anywhere said so. The two are now counted apart — NCBIUnavailable is a recorded failure with the organism's name attached, an empty document is a note that the tax-id may have been merged or deleted.

Added

• **Withhold protein records from public view, and put them back exactly as they were. A paired switch on the admin dashboard (Admin → Visibility**): one action withholds every protein carrying at least one negative interaction — the unpublished asset a competitor is not generating — and one action restores that same set. Plus a per-protein pair, and a searchable management list, for the one-off cases. **The enforcement is the existing Protein.visibility column, not a new flag.** Every read path in this codebase already gates on it: the REST list/detail/FASTA/structure routes, the server-rendered pages, FTS and trigram search, the CSV/JSON/FASTA downloads, services/export.py, services/snapshots.py — whose output takes a Figshare DOI that can never be unpublished — and workers/reference_build.py, which builds the public DIAMOND databases from visibility = 'public'. A parallel is_hidden boolean would have needed every one of those patched by hand, and would have missed some. Four new columns on proteins (migration 0010_protein_visibility_holds) do one job: make the action exactly reversible. They record, per row and durably, the visibility the record had immediately before the hold, who applied it, when, and under which batch. Restore puts that back — a record that was already private for an unrelated reason comes back private, never public. The prior value is stored rather than recomputed, because release happens months later: by then curation will have moved which proteins carry a negative result, and recomputing would restore the wrong rows. Precedence, stated plainly in the UI: manual beats bulk. A protein you hide by hand carries reason = 'manual' and the bulk restore leaves it hidden — release it individually. Unhiding one by hand during the embargo takes it out of the batch for good. Both switches are idempotent: restoring twice reports restored: 0 and changes nothing. The fourth door is closed structurally. Embargo has leaked out of this database three times, each through a different promotion path that assigned visibility = 'public' without knowing an embargo applied — and the lesson of those three is that the list of such paths is never complete. So the guard is not at the doors: a before_flush listener parks any visibility write to a held protein in visibility_hold_prior instead of applying it. Nothing leaks, and nothing is lost — an embargo that matures during a hold takes effect the moment the hold is lifted.

Fixed

• **A record that is not public is no longer served by accession, by id, or from the evidence spine.** Auditing the read paths for the switch above turned up pre-existing leaks that would have defeated it, and that already applied to every embargoed deposit:
  • protein_resolution.resolve_accession reported a non-public protein as LIVE, so GET /api/v1/proteins/PB000123 and the page behind it served its full detail — sequence, structures, kinetics, interactions, papers. Now MISSING (404, not 403 and not 410: the existence of the record is part of what is withheld).
  • /proteins/{id}/fasta and /proteins/{id}/structure took a raw id with no visibility or tombstone check at all.
  • /variants/{id}, /variants/{id}/structure, /engineering/proteins and /engineering/proteins/{accession} served a hidden parent's data through its engineered variants.
  • The paper detail page, the public paper footprint (its curator twin keeps the full view), and the plastic page's protein buckets, counts and rows all rendered hidden proteins straight off the evidence spine. The interaction-actor rule that guarded snapshots now lives in services/query.py and is shared by all of them.
  • /search/autocomplete applied no visibility gate — the FTS index covers every row regardless — which made it the discovery mechanism for the rest.
  • Count-only disclosures: /stats/by-country, the taxonomy-group protein totals, and the plastics table's protein_count.
  • The public contribution timeline republished the old and new values of every field an admin edited, including sequences, for any protein.
  • Pathway steps, the pathway graph and the compound page named their linked enzyme regardless of its visibility.
• The chatbot can no longer read non-public rows. POST /chat lets an anonymous visitor run arbitrary read-only SELECTs against an allowlist of tables; the allowlist governs which tables, never which rows, so "list every protein sequence" returned embargoed and private records. Each visibility-bearing relation is now rewritten into a filtered sub-select aliased back to its own name, so the hidden rows are not removed from the answer — they were never in the relation. (Sub-selects rather than database views on purpose: a view over SELECT * pins the base table's column types and would break the next migration that widens one.)
• FROM papers, users is no longer allowed past the chatbot's allowlist. The guard's table scan was \b(from|join)\s+(\w+), which reads the first relation of a list and stops — so a comma-joined second table was never checked against ALLOWED_TABLES at all, and users (hashed passwords, API key hashes) was reachable by writing a comma instead of a JOIN.

Fixed

• Signal peptides: the source had never run, once, in production. has_signal_peptide was NULL on all 762 protein rows and signal_peptide_cleavage_site on all of them too. protein_enrichments held zero rows for source='signalp' — not stale ones, none. Two independent causes, either one fatal: 1. ENRICHMENT_SIGNALP_ENABLED defaults to False, and the rung sits behind it, so nothing was ever attempted. Worse, the disabled branch did not even set result["signalp"] = skipped — the source vanished from the payload entirely, making "switched off" indistinguishable from "does not exist". 2. Flipping the flag could not have helped: the rung shells out to signalp6, which is not in the image and cannot legally be put there. SignalP 6.0 is licensed by DTU per registered user with no redistribution right. Confirmed against the running worker: command -v signalp6 → not found. The fix is not to smuggle SignalP into the image but to stop depending on a tool we may not ship. New app.services.signal_peptides makes the predictor pluggable across three back-ends, chosen by ENRICHMENT_SIGNALP_TOOL: | Tool | Licence | Role | |---|---|---| | deepsig | GPL-3.0 | Local, redistributable, no rate limit. requirements-bio.txt; kept out of the default image only because it pulls TensorFlow. | | ebi_phobius | n/a — we redistribute nothing | EBI's hosted job service. Needs no install, so it is the default and the feature works on a stock image. | | signalp6 | DTU academic | Honoured when an operator has licensed and installed it. Never shipped by us. | ENRICHMENT_SIGNALP_ENABLED now defaults to True, because there is finally a predictor a default install can reach. Both columns are written together and never independently — a True boolean with a NULL cleavage site silently degrades the reference FASTA, every ESMFold model and mature_sequence() to the precursor. The invariant is enforced in SignalPeptideResult itself. All three parsers are pinned against real captured tool output, because the whole feature turns on one integer and the three tools each report it differently: SignalP 6 CS_pos → the first number; DeepSig GFF3 → the feature end; Phobius n6-17c23/24o → the number before the slash. Verified live end-to-end against EBI on four production sequences, e.g. PB264Q MKANLILACASVVAGVSA│APLERR… — a textbook AxA cleavage at 18.
• ESMFold: 522 "errors" were one free endpoint being busy. Every one of the stored details was a 5xx from the keyless ESM Atlas endpoint — 416 × 504 {"message": "Endpoint request timed out"}, 104 × `504 Gateway Time-out, 2 × 502`. The endpoint is not dead and has not moved: the EvolutionaryScale→Biohub acquisition is a red herring here, because no Forge key is configured, so esmfold_api_url is never reached and the keyless path is the only one that runs. Measured against the live service (2026-08-23), which is what the fix is built on: a 78-residue fold returns 200 in 1.7 s; six identical 300-residue folds returned three 200s (1.4–2.6 s) and three 504s at exactly 29.6 s — an AWS API gateway ceiling over a cold backend. It is not length-dependent: a 180-residue sequence failed where a 300-residue one had just succeeded. A 504 therefore says nothing whatsoever about the sequence. The state machine's own retry is THROTTLE_DAYS (6) away and gives up after RETRY_CAP (5), so a ~50 % per-request failure rate produced rows that had burned up to 11 retries over months with no structure. _fold_via_esm_atlas now retries within the attempt (ESMFOLD_MAX_ATTEMPTS, default 5, linear backoff), which takes ~50 % to ~97 %. Sequence-level rejections (400/413/422) are still never retried — the answer cannot change. New EsmfoldOverCapacity, distinct from EsmfoldUnfoldable and from a plain error, so "a free service is popular" stops being reported as a defect. Separately confirmed that ESMFOLD_PUBLIC_MAX_RESIDUES=400 is correct rather than arbitrary: 450 residues gets an immediate 413, in under a second.

Changed

• The admin enrichment table stops crying wolf. It had one Errors column fed by every transient row, and it counted pending nowhere at all — so esmfold read "522 errors" (520 of which were an upstream being busy) and alphafold's 498 pending rows were simply invisible. Every one of those 498 is a protein with no UniProt accession for AlphaFold to key on, and 183 proteins carry no sequence at all: curation gaps that no number of retries can close. Four columns now separate what needs a human from what does not — Errors (real, actionable), Busy (upstream at capacity), Blocked (waiting on data a curator must enter), Waiting (may still resolve itself) — plus Not run, which distinguishes "ran and found nothing" from "never ran", the precise ambiguity that let SignalP read as a tidy 0 % for years. A dormant badge marks a source with no rows at all. Colour is now carried only by the columns that mean act on this.
• The status/detail vocabulary moved to app.services.enrichment_status, so the dashboard buckets on the same constants the workers write. It previously restated retries >= 5 as a literal, which would have desynced silently the moment RETRY_CAP moved.

Added

• scripts/backfill_signal_peptides.py — dry-run by default, --apply to write, resumable (candidacy is a property of the query, not a checkpoint file), batch-committed, and rate-limited on the hosted predictor only. --force rescues rows whose last attempt failed; --report-fingerprint prints the reference-database digest before and after. ⚠ A successful run invalidates the public DIAMOND databases, by design. current_fingerprint() hashes (has_signal_peptide, cleavage_site, sequence) because export_reference_fasta writes the mature chain — a signal peptide landing on an unchanged sequence really does change the FASTA. Three new tests in test_reference_rebuild.py pin that pairing, including one that asserts over mature_sequence()'s actual attribute reads, so adding a fourth input to the trimming rule fails the build rather than silently letting the databases on disk disagree with the digest that claims to describe them.
• requirements-bio.txt for the optional local predictors, with the licensing rationale for what is in it and what can never be.

Fixed

• The tree's leaf count is counted, not subtracted (#415, follow-up to #308). v0.87.4 started reporting how many organisms actually became tips, and derived it as organism_count - skipped_no_lineage. Verified against production immediately after the deploy: the footer said 780 taxa over a tree with 776 rendered tips. The number whose entire job is to be honest about what was drawn was itself wrong by 4. Two things stop an organism becoming a tip and only one of them is countable up front. _build_newick places a tip with node.setdefault(m.name, {}), and _lineage_to_internal_path drops a lineage's last rank as the organism's own — so a genus-only organism hangs its tip one level above its congeners, under the label that is precisely the genus node they created there. setdefault returns that node and the organism silently becomes internal. Live, on the whole corpus: Micromonospora, Rhodococcus, Microbispora, Roseateles. count_newick_leaves() counts tips from the Newick string, which is also the only thing that works on a Redis cache hit (where _build_newick never runs). result_metadata now carries merged_into_ancestor alongside skipped_no_lineage, and the invariant `leaf_count + skipped_no_lineage + merged_into_ancestor == organism_count` is asserted. The footer names each reason rather than lumping them: *"776 taxa (of 795 matched — 15 have no lineage, 4 share a name with an ancestor)"*, because the two need different answers from a curator. The loss itself is not fixed here — what a genus-level record should look like on a tree that also holds its species is a topology decision, and the robust form of it (key _build_newick by organism id, not by name) touches a label contract that the tree page's parser and the plastic matrix both depend on. Tracked in #415. Making the count honest is what surfaced it, and is the half that must not wait.

Changed

• **The sulfonated copolyester row is now the family row: one plastic for the twelve ionic phthalic polyesters, with the diol carried in substrate_composition** (curator Ruling A, 2026-08-20). 0.87.3 added poly(ethylene terephthalate-co-5-sulfoisophthalate) — one of the twelve copolyesters 10.1016/j.jbiotec.2017.01.012 (Haernvall et al. 2017) assays — and recorded the open question in REJECTED_TERMS: eleven more rows, or one family row with the diol in substrate_composition. **The curator chose the family row.** These copolyesters are too niche to grow a 108-row plastics vocabulary by ~10% for a single paper's independent variable, and none of the eleven names is used in any literature outside these two papers. Row 105 is renamed in place to poly(terephthalate-co-5-sulfoisophthalate) rather than joined by a family row: a specific member and the family that contains it are the near-duplicate this module exists to prevent, since a producer naming either would be right and the two would split every query that touches the material. The name elides the diol and keeps both diacids, which is what makes it a family without making it a bucket — nothing non-sulfonated and nothing built on ortho-phthalate can drift into it. The literature's own term, "ionic phthalic polyester" (the J Biotechnol title), leads the full_name so it is searchable without being a key. The PET parent is detached, and that is the substantive half of the change. 0.87.3's parenting rested on "the backbone is still ethylene glycol and terephthalate" — true of exactly one of the twelve members, and of none of the three glycol-ether ones, so keeping it would make a PET-and-descendants filter return an enzyme acting on poly(tetraethylene glycol terephthalate-co-5- sulfoisophthalate). PEN is the precedent for an aromatic polyester whose relation to PET is a resemblance rather than a derivation. **The rename was measured against production before it was written, because renaming a referenced row is a different and riskier operation than renaming an unused one.** Row 105 carried 0 microorganisms, 0 proteins, no degradation links, no children, no monomers, no components and no containing blends; all 395 ingestions were scanned and none names it in extracted_data — the one prose mention, ingestion 288 (10.1021/acs.est.7b00062), files its fourteen analogues under the placeholder Plastic. The only artifact naming it anywhere is the not-yet-submitted extraction of the paper this ruling is about. The seeders could not express a rename at all: they are name-keyed upserts, so changing the literal would have inserted the new name and left the old row standing beside it. RENAMED_PLASTICS closes that, runs before the upsert, and refuses three cases quietly rather than aborting a deploy over the state of the data — the old name absent (nothing to do), both names present (a merge only a curator can make), or the old row carrying references (a rename re-labels evidence filed under the old meaning; whether it survives the new one is a curatorial question). The reference check is discovered from the mapper metadata rather than from a list of tables, so a new foreign key onto plastics.id cannot make it quietly incomplete. The PreSync job now prints the rename count, because a rename is the one thing that step does that is not purely additive. full_name and abbreviation are cleared by the rename so the upsert refills them. This is the one point where "never clobber an existing value" is deliberately suspended: that rule protects a better description of the same material, and a re-purposing changes which material the row is, so a description of the old one is not a correction to keep but a statement that has become false.

Added

• **Nine diols, so the family row's distinguishing component can actually be named.** 1,5-Pentanediol, 1,6-Hexanediol, 1,8-Octanediol, 1,12-Dodecanediol, 1,2-Propanediol, 1,4-Cyclohexanedimethanol, Diethylene glycol, Triethylene glycol and Tetraethylene glycol. Ethylene glycol (legacy v1 import), 1,3-Propanediol and 1,4-Butanediol were already present. Collapsing twelve plastics onto one row only holds if every member stays distinguishable, and substrate_composition[].component is validated against the compounds vocabulary — so a missing diol does not fail loudly, it just leaves nine of the twelve substrates unidentifiable while the composition still validates. That would collapse the paper's independent variable silently, which is worse than the vocabulary growth the ruling was avoiding. The convention is the one this corpus already runs on: the diol is a null-fraction component beside the diacid mole fractions, as 1,4-Butanediol is in 9 papers and 75 occurrences. is_additive=False throughout — a diol is half of a polyester backbone, never something dispersed in it. This is the field most likely to be got wrong from outside the paper, since several of these are also sold as solvents, humectants and antifreeze; the flag records the role in this material. 1,2-Propanediol is a separate row from 1,3-Propanediol despite sharing C3H8O2, on the Bis(2-ethylhexyl) phthalate / Dioctyl phthalate reasoning: the paper assays both and the difference is a finding — the branched C3(1,2) polyester released 5.46 mmol TA/mol against the linear C3's 12.7.
• PlasticTerm.no_abbreviation — a row that has no abbreviation and none to be derived. 0.87.3 used the column width as a proxy ("a 51-character name will not fit VARCHAR(50), so NULL"), and this release is where the proxy and the real question come apart: the family name is 42 characters and fits, so without the flag the rename would have quietly filled in an abbreviation column whose NULL was a deliberate decision — an "abbreviation" that abbreviates nothing, and a side effect of a rename rather than a decision anyone made.

Fixed

• **The phylogenetic tree carries every organism matching the filter — the cap was in the browser, not in the query (#308).** v0.87.3 removed the .limit(500) from _resolve_ids() and the reported symptom did not move: the Microorganisms page still showed 795 organisms and its "View as Tree" button still produced a 500-leaf tree. Reproduced against production by driving the real button and capturing the requests: ``` GET /api/v1/microorganisms?polarity=positive&limit=500&offset=0 → 500 of 795 POST /api/v1/tree/build {"microorganism_ids": [ …500 ids… ]} ``` limit's maximum on that endpoint is 500, deliberately (B2 — one request must not be able to materialise a table), and build_tree uses microorganism_ids verbatim when they are given. So the page had one page of ids, the tree had one page of leaves, and _resolve_ids() — the function that had just been fixed — never ran at all. **The root cause was app/templates/landing/microorganisms.html:562, qs.set('limit', '500')**, and no amount of server-side work could have reached it. The page now sends the filter and the server resolves it, which is the only arrangement with no number in it to outgrow. Three parts:
  • TreeBuildRequest carries the whole microorganism browse filter set — q, plastic, year range, temp_class, environment, location, polarity, include_negative, evidence_scope, every taxonomic rank. A shorter list would have re-created the bug quietly: the page can filter by all of those, so a request that could not express them would force the page back to resolving ids itself. microorganism_ids stays and still wins when present — it is the only way to ask for a set no organism filter describes (the proteins page's organisms-behind-these-proteins tree, and /tools/build-tree?ids=).
  • _resolve_ids() answers with microorganism_list_filters() — the definition of "a microorganism matching these filters", already shared by the list endpoint, /api/v1/stats's headline and the exports. That is the part that cannot silently re-break: the tree and the table now answer one question, so a filter added to the list reaches the tree with it.
  • app/static/js/microorganism_filters.js reads the form once for both the table and the button, and app/static/js/tree_from_filters.js posts those filters. Shipped modules rather than inline template script, because the browser tier can only guard a file the page loads. One deliberate behaviour change: with no filters at all, a tree used to mean every row in the table and now means every row the browse list shows — 795 rather than 1,308 — because the shared definition reads an unqualified request as "has at least one positive result", the ruling that makes the list total and the home page headline one number. include_negative: true still reaches every public record. And the signed-out path is gone rather than fixed. The button used to bounce a visitor with no token to /tools/build-tree carrying only name/lineage/rank, silently dropping plastic, polarity and evidence scope — this same defect in miniature. POST /api/v1/tree/build takes an optional user by design (building a tree ingests no user data), so there was nothing to gate. A failed build now reports the failure instead of navigating to a tree of a different question.
• The tree page counts the leaves it drew, not the rows it was handed. _build_newick skips an organism with no lineage string — it has nowhere to hang — with a warning in a worker log, while the page's footer counted the plastic matrix, which has a row per organism whether or not it was placed. In the live corpus 16 of the 795 organisms have no lineage, so the honest leaf count is 779 and the footer would have said 795 over it. result_metadata now carries organism_count, leaf_count and skipped_no_lineage, and the footer reads "779 taxa (of 795 matched — 16 have no lineage and cannot be placed)". Same failure family as the truncation above: a phylogeny missing organisms renders exactly like a complete one, so the number has to be carried out to the reader.

Testing

• tests/e2e/tree_from_filters.spec.js — **the tier that would have caught this.** It clicks the real button against a fake backend that enforces the real 500-row list cap and resolves filters uncapped, then asserts the resulting tree's leaf count equals the number of matching organisms (795 > 500). Verified red first: against the v0.87.3 handler 4 of its 6 cases fail, the headline one reporting a 500-id POST body.
• tests/test_tree_build_from_filters.py — the server half and the contract between them: the request model is a **superset of the list endpoint's own filter parameters** (so the page can never again have a reason to resolve ids itself), the resolver returns 520 of 520 and every one becomes a leaf, polarity reaches it, and over PostgreSQL the tree's id count equals the total the list reports for the same filter. Verified red first against a reinstated .limit(500) (assert 500 == 520) and a reinstated client-side id fetch.
• tests/test_tree_no_leaf_cap.py's fixtures now seed real degraders, because the resolver answers the browse list's question and an organism with no interaction is in no list.

Added

• **PBA — poly(butylene adipate), the aliphatic endpoint of a series whose other members are already here.** 10.1016/j.nbt.2015.11.004 (Perz et al. 2016, *Substrate specificities of cutinases on aliphatic-aromatic polyesters and on their model substrates*) is blocked on this one name: two positive interactions, HiC and Thc_Cut1, each carrying the adipic acid released after 72 h at 50 °C. The paper synthesises a graded Ada:Ta series and this is its Ta0 endpoint (Ada100_Ta0, "PBAda" in Fig. 5) — adipic acid and 1,4-butanediol, no terephthalate in it at all. It must not be folded into PBAT, the one tempting move because both come out of the same table: the Ada:Ta ratio is the experiment's independent variable, so filing the Ta0 sample under the aliphatic-aromatic copolyester collapses the axis the study was built on and asserts terephthalate in a substrate that contains none — the PEN-into-PET argument one family over, and it lands harder here because Thc_Cut1's largest product release on any substrate in the paper is on this sample. Nor PBS/PBSA: same diol, different diacid, and PBSA is the dangerous near-miss since it does contain adipate and would additionally assert a succinate comonomer. kind='homopolymer', polymer_class='polyester', unparented — PBAT, PBS and PBSA are siblings on the diacid axis, not ancestors. The abbreviation is shared with poly(butyl acrylate) in the wider literature, which is why the full_name is load-bearing.
• **poly(ethylene terephthalate-co-5-sulfoisophthalate) — a sulfonated PET copolyester — and 5-Sulfoisophthalic acid as a compound.** 10.1016/j.jbiotec.2017.01.012 (Haernvall et al. 2017, *A new arylesterase from Pseudomonas pseudoalcaligenes can hydrolyze ionic phthalic polyesters*) fails validation with 26 errors: 12 plastic names and 14 substrate_composition components. Parented to PET: the Apexa shape, not the PEN shape. The backbone is still ethylene glycol and terephthalate, with 9–30 mol% of the diacid replaced by sodium 5-sulfoisophthalate. PEN was left unparented because it swaps the diacid outright, and the stated objection was that a PET-and-descendants filter must never return an enzyme that has never released terephthalate — here PpEst releases terephthalic acid and MHET, and the paper reports no 5-sulfoisophthalic acid in any hydrolysate. Equally it must not be folded into PET: PET is not hydrolysed under these conditions and the ionic comonomer being what opens the chain is the whole result. kind='copolymer', which the model expresses directly. The diacid is a compounds row and refused as a plastics one: nothing acts on it, and a plastics row would make a monomer selectable as an actor's substrate. Unlike 2,6-Naphthalenedicarboxylic acid, refused last release because PEN is a homopolymer and nothing needed its diacid, this one has 14 substrate_composition entries waiting on it whose mole fractions are the paper's second independent variable. **The paper is not unblocked by this, and the shortfall is recorded rather than papered over. It assays twelve** copolyesters, one per diol; the curator ruled on the ethylene member. The compounds row clears all 14 composition errors and this row clears 1 of the 12 plastic errors, leaving 11. The other eleven diols are a deferral awaiting a ruling — including whether they want eleven rows or one family row with the diol carried in substrate_composition — and are recorded in REJECTED_TERMS with that question stated.
• **Three aromatic medium-chain-length PHAs — and their published abbreviations deliberately not used as keys.** 10.1074/jbc.m608119200 gives poly(3-hydroxyphenylhexanoate), poly(3-hydroxyphenylheptanoate) and poly(3-hydroxyphenyloctanoate), all genuine bacterial polyesters assayed as polymers by a purified depolymerase. They are classified exactly as PHHx and PHD are — homopolymers on polyhydroxyalkanoate, parented to PHA; the phenyl side chain is what stops them being filed under PHO/PHN/PHHx. **The paper's own abbreviations are PHPhH and PHPhh, which differ only by the case of the final letter, and they cannot be keys here.** Measured against this codebase rather than assumed: PostgreSQL's unique index on plastics.name is case-sensitive, so both rows would store without complaint — while every lookup is case-insensitive. plastic_resolve.resolve_plastic compares func.lower(name) and func.lower(abbreviation) and returns .first(), an arbitrary pick between the two; its normalised fallback lower-cases as well; and ingestion_promotion matches with ILIKE. Two distinct polymers would therefore resolve onto one row with no constraint violated and no error raised — an observation filed against the wrong material, silently. So the canonical name is the chemical name and the abbreviations are case-fold-safe: PHPhHx, PHPhHp and PHPhO, taking the hexanoate/heptanoate stems PHHx already uses. The cost is stated rather than hidden: neither literal from the paper is a key, so an extraction written against its abbreviation list resolves to nothing until re-extracted. A miss is visible and recoverable; a silent merge is neither.
• **Bis(2-ethylhexyl) phthalate (DEHP) as its own compound, by curator ruling.** compounds already holds Dioctyl phthalate, and the two share the formula C24H38O4, which is exactly why folding one onto the other looks free. It is not: "dioctyl phthalate" strictly names the di-n-octyl ester (DnOP, CAS 117-84-0) while DEHP is the branched 2-ethylhexyl ester (CAS 117-81-7), and DEHP is the one actually in pPVC. Mapping a paper's DEHP onto the DOP row would put a compound the paper never used into substrate_composition — wrong, not merely coarse. is_additive=True, on the Rhodamine 6G precedent. The existing Dioctyl phthalate description read "Plasticiser (DOP / DEHP)", which the ruling overturns, and the literal is corrected here. **That correction reaches a fresh database only** — seed_compounds fills NULLs and never overwrites, deliberately, so the deployed row keeps the old wording until a curator edits it. Flagged here rather than forced, because overwriting a description on every deploy is the curator-clobbering this seeder exists not to do.

Fixed

• **A vocabulary name too long to be an abbreviation is dropped rather than copied.** seed_plastics set abbreviation = name for every row, which was free while every name was an abbreviation. The sulfonated copolyester's name is 51 characters against a VARCHAR(50) column, so copying it would have aborted the PreSync seed job on PostgreSQL — a deploy-time failure invisible to every SQLite-backed test, since SQLite does not enforce the width. Truncating would be worse than NULL: resolve_plastic matches on abbreviation, so a mangled prefix becomes a second resolvable identifier for the same material. The limit is read off the column so the two cannot drift, and NULL is what the promotion path already writes for a row with no abbreviation. An explicit abbreviation on a term now wins over the name, which is what lets the three aromatic PHAs above be keyed by chemical name and still carry a short handle.

Fixed

• A filter-built phylogenetic tree carries every organism it matched (#308). _resolve_ids() ended in a bare .limit(500) with no ORDER BY, so a filter matching more organisms than that built the tree from an arbitrary 500 of them — and not even the same 500 twice — while nothing told the caller. The failure mode is not an error message: a clipped phylogeny renders exactly like a complete one, so someone filtering by a phylum drew conclusions from a tree that was quietly missing organisms. At 795 organisms in the live corpus the cap was reachable today, not hypothetically. The cap is gone rather than made visible. The issue proposed keeping it and reporting "showing 500 of N"; Victor overruled that — *"it should have no limit. we dont have that many and it generates bad side effects"* — and the measurement agrees. Seeded on PostgreSQL with 795 organisms, 40 plastics and 2,769 interactions: resolving the whole corpus takes 23 ms (a domain: filter over 531 of them, 23 ms; a plastic_id filter, 7 ms), the worker's build_taxonomy_tree takes 67 ms, and the poll payload is **593 KiB raw / 16 KiB gzipped** for an 18 KB Newick plus the 40 × 795 plastic matrix. Nothing there justifies discarding data. The 500 arrived with the original tree feature (36b431e, 2026-05-26) carrying no stated reason, so nothing was traded away that had been argued for. The ORDER BY stays on its own merits, and is on both queries. Rows are ordered by lineage, then name, then id — taxonomic grouping first, and a total order so two identical requests resolve to the same list. It also has to be on the worker's re-fetch (id IN (...) has no inherent order), because _build_newick inserts nodes as it walks the rows and therefore lets row order decide sibling order in the Newick.

Fixed

• The four landing-page stat cards are the same height again (#316). They carried no h-100, so each card was only as tall as its own text — and the four no longer carry the same amount of text. #402 gave Microorganisms a 545 tested whole · 171 via their enzyme breakdown and Proteins a wild-type · engineered one, while Plastic Types and Papers have none, so the pair the reporter photographed sat 17 px apart at desktop and the grid read as ragged. Measured across eight viewport widths, the cards sitting beside each other were unequal at seven of them; they are level at all eight now — including 390 px, where 1,284 wraps and Papers was 68 px taller than Plastic Types. The class is the one this page already uses: every .feature-card below the hero carries h-100, as do the dashboard's own .stat-cards. Where an anchor wraps a card it needs d-block h-100 too — height: 100% does nothing to an inline box, so the card would stretch to the anchor rather than to the column. No CSS change and no new dependency. The two rows still differ from each other, and that is deliberate. A wrapped Bootstrap card grid sizes each flex line to its own content, and levelling the lines would take a fixed min-height — measured, the right constant would be 314 px at 320 px wide and 126 px at 768 px, so any single number is wrong at most widths and shows as dead space under the short cards.

Added

• **accession_notes on ExtractedProteinVariantActor — the caveats are more load-bearing on a variant than on a wild-type, and it was the one actor with nowhere to put them.** The occasion is a correction on 10.1002/anie.202525215: ten sequence-bearing nodes held a His-tagged pET26 construct where ruling R-1 requires the deposited PDB 8OTA record verbatim, and six of the ten were protein variants. The note explaining the swap — including a numbering_offset corrected −32 → −34, silently wrong by two residues against the new frame — could not be written where it belonged. accession_notes was declared on ExtractedProteinActor and on nothing else, so a note on a variant went onto an undeclared key and was discarded between the 200 and the database. A variant's sequence is derived — the parent record plus this variant's substitutions, gated by a wild-type-residue checksum — and it carries a numbering_offset (R-7) that has to be reconciled against the paper's own numbering. Every one of those steps can go uncertain in a way a curator must be told about, and none of them exists for a wild-type. Writing the note on the parent instead is not a workaround: the parent record is shared by every sibling variant, so it would attribute a statement about one derived sequence to all of them. The producer had been sending it all along. Swept across the 397 extraction JSONs on the pipeline's disk, wire-shaped exactly as the submit stage sends them: 29 notes on 3 papers (10.1038/s41467-024-45523-5, 10.1016/j.polymdegradstab.2017.01.006, 10.1038/s41467-025-61599-z) landed on the undeclared key and were dropped without a word — *"the paper assays this variant but never lists its substitutions; the mutation set is taken from the originating study it cites"*, *"it differs from BAV72205.1 at exactly one position, precursor residue 111 = paper residue 77 + 34"*. That is exactly the prose R-1 and R-7 exist to preserve. It travels as far as the wild-type's does and no further, deliberately: declared in the contract (so it reaches GET /openapi.json and the producer's preflight), stored verbatim in extracted_data, and rendered on the curation form as its own control. **Promotion writes it into no column — for either actor.** proteins has no accession_notes column either; this is prose for the curator reading the ingestion, and giving the variant path a column the wild-type path does not have would be a new asymmetry in place of the one being removed. So no migration: the field lives inside a JSON column that already exists. A control was not optional. approve_ingestion overwrites extracted_data wholesale, so a leaf the form does not render is deleted the first time a curator presses Approve — which is why tests/test_curation_form_coverage.py derives the expectation from the schema in both directions. The exportable-leaf count goes 102 → 103, in the Python tier and in the Playwright pin.

Fixed

• **An unrecognised key in an extraction is now named, in the log and in the response. It is still accepted.** Pydantic's default extra="ignore" meant an undeclared key was dropped between the wire and model_dump() with no error, no log line and a 200 — which is why the variant accession_notes above went unnoticed for months, and why nothing complained when the producer emitted "accession_notes": null on those six nodes. app.services.unknown_fields.unknown_field_paths walks the raw request body against ExtractedData — the only moment the dropped keys still exist — and POST /ingestions and PATCH /ingestions/{id} report what it finds two ways: a WARNING naming the ingestion id, the DOI and every path, and an unrecognised_fields list on the response. Paths use app.services.payload_diff's notation (interactions.0.protein.ec_number, a 422 loc minus the envelope), so one resolves through the curation form's existing error locator with no second mapping to keep in step. extra="forbid" was considered and rejected, and so was emitting additionalProperties: false into the OpenAPI document, which looks like the cheaper version of the same idea and is worse. Forbidding 422s any payload carrying a stray key, including the stored extractions the submit stage replays. The OpenAPI route would not change what this server accepts — but the producer's own preflight validates against GET /openapi.json and refuses to submit what fails there, so it would convert "we dropped a key" into "this paper is blocked", one repository away and with nobody watching. A blocked paper mid-batch is worse than a warning. tests/test_unknown_extraction_fields.py pins both decisions. **Noise is the failure mode this had to survive, so it was measured before it was built. Over those same 397 extractions: 288 carry a top-level supplements_required** — a producer control flag the extract stage consumes locally, never a claim about the paper — so reporting it would fire on 73% of submissions on one uninteresting key and the mechanism would be ignored inside a week. It is allow-listed by exact name at the top level only, with extraction_model (pipeline provenance, delivered as its own PATCH field); the same word on an interaction is not that flag and is still reported. After that allowance and the variant field above, 4 extractions in 397 trip the warning at all, on six keys, every one of them a real loss: observations[].observation_type and interactions[].confidence (both retired in Feature 2.5), microorganism.sample_source (the field lives on the interaction), microorganism.thermophilic, kinetics.notes, and protein.cross_references (owned by V2 enrichment, never extracted). Four papers in four hundred is a signal; two hundred and eighty-eight would have been noise. A free-form dict field is a leaf and is never walked — descending into additional_conditions or culture_collection_ids would report every user-chosen key as unknown and destroy the signal on any paper that used them as intended. The walk is bounded at 200 paths for the reason payload_diff.MAX_PATHS is bounded. Scoped out deliberately: POST /ingestions/{id}/approve is not instrumented. Its payload comes from a form whose controls are derived from the schema and asserted exact in both directions by tests/test_curation_form_coverage.py, and the record it re-posts was itself produced by model_dump(), so there is no undeclared key for it to carry.

Note for operators

Nothing in this release requires a migration. accession_notes on a variant is a field of a Pydantic contract stored inside the existing extracted_data JSONB column; no table changes. Extractions already stored are unaffected — the key was never persisted, so there is nothing to backfill, and the three papers that sent it will deliver their notes on their next submission.

Added

• **PEN in the plastics vocabulary — the fourth member of a series three quarters of which was already here.** 10.1021/acscatal.0c05126 (Cui et al. 2021, *Computational Redesign of a PETase for Plastic Biodegradation under Ambient Condition by the GRAPE Strategy* — the DuraPETase paper) is blocked in NEEDS_HUMAN_REVIEW against an extraction of 161 interactions on this one name. PET, PBT and PTT are all present; PEN's absence was arbitrary, which is the argument PTT itself was added on. It must not be folded into PET, and this paper is the one that proves it. It runs the three side by side and the products differ: DuraPETase hydrolyses PEN to 2,6-naphthalenedicarboxylic acid and its mono-/bis(2-hydroxyethyl) esters, where PET gives TPA/MHET/BHET and PBT gives TPA with its mono-/bis(4-hydroxybutyl) esters. Filing the PEN arm under PET would put two arms of one specificity experiment on one row and assert that the enzyme released terephthalate from a substrate that contains none — and the rates separate them in the direction that makes the collapse worst, PEN degradation markedly enhanced by the engineered variant while PBT stays slow. kind='homopolymer', polymer_class='polyester', unparented. Unparented deliberately, and unlike Apexa: Apexa hangs off PET because it is modified PET (the same two monomers plus an undisclosed third), while PEN replaces the diacid outright — 2,6-naphthalenedicarboxylic acid for terephthalic acid. PBT and PTT are the same substitution on the diol side and both are unparented, so four siblings beats three siblings and one child distinguished by nothing. The full_name names the 2,6 isomer rather than reading "polyethylene naphthalate", because the legacy PEF row — poly(ethylene furanoate) — is the furan analogue on that same diacid axis and one letter away in the abbreviation.
• Rhodamine 6G in the compounds vocabulary — a dopant, not a plastic. The occasion is an extraction of a fluorophore-doped PET / PA6 specimen: the polymer resolved, the dye had nowhere to go, so substrate_composition fell back to the base polymer alone and the extraction logged a data gap saying so. Nothing is blocked by this and it carries no scope implication — it is a fidelity fix, and what it repairs is that a doped specimen and an undoped one were the same record. is_additive=True, unambiguously — a tracer dye is never a blend partner or a comonomer at any loading, so it does not get starch's NULL. Formula is the chloride salt (CAS 989-38-8), the form sold and the form a paper means when it writes "Rhodamine 6G". Refused as a plastics row, which is the one way this could have gone wrong: a dopant is not a material under study, nothing acts on it, and a plastics row would make it selectable as an actor's substrate. compounds is reachable from an extraction only through experiment.substrate_composition, which is exactly and only what it claims. Two further refusals are recorded in REJECTED_TERMS with their arguments: parenting PEN onto PET, and 2,6-Naphthalenedicarboxylic acid as a compound — the counterpart of the existing Terephthalic acid row, refused on the 3-Mercaptobutyrate rule (PEN is a homopolymer, so no substrate_composition needs its diacid, and the PET-pathway compounds are there for the reaction-step subsystem rather than as a monomer set).

Added

• **P(3MP) and P(3MB) — the two homopolythioesters, and the half of a v1 paper that has been missing since v1.** 10.1007/s00203-004-0715-z (Elbanna, Lütke-Eversloh, Jendrossek, Luftmann & Steinbüchel 2004) is blocked in NEEDS_HUMAN_REVIEW with **32 validator errors, every one of them one of these two names, against an extraction of 125 interactions and 219 observations**. Ruled in scope by the curator, 2026-08-13. What they are. Polythioesters are the sulfur analogues of the polyhydroxyalkanoates: the backbone ester oxygen replaced by sulfur, giving a thioester linkage -S-C(=O)- where a PHA has -O-C(=O)-. The paper measures the substitution rather than asserting it — thioester C=O at 1,688 cm⁻¹ against the oxoester's 1,735 cm⁻¹ by IR, plus elemental sulfur, ¹H-NMR and MALDI-TOF. Both are biosynthesised, accumulated in vivo as intracellular granules by recombinant E. coli JM109 (pBPP1) — buk/ptb from *Clostridium acetobutylicum* feeding a Thiocapsa pfennigii PHA synthase — and purified from the cells. PHA synthases are unspecific about the linkage type; PHA depolymerases, which is this paper's finding, are not. R-16 was settled first, because if the actors had been assayed on a thioester monomer the right answer was to delete 32 interactions, not to mint two terms. All 32 are against the polymer: purified homopolythioester granules as a suspension (turbidity), as an agar-overlay emulsion (clear zone), or as sole carbon source. The paper's free-3MP and 3,3'-thiodipropionate growth tests are small-molecule carbon sources and correctly produce no interaction. This is the missing half of an existing entry, not a new class. P(3HB-co-3MP) has been a plastics row since v1 — and both of its degraders_list.tsv rows cite this paper. v1 recorded the copolymer positives and had nowhere to put the homopolymer negatives, so the study's title and conclusion were lost. Every result on the two new rows is negative, and that is the point: no PHA-degrading bacterium grew or cleared a plate, and none of the five purified depolymerases showed activity, while the same enzymes hydrolyse the copolymer in the same tables with activity falling as 3MP content rises (25,000 U/mg on PHB → 9,500 on the copolymer → zero). A negative on a named polymer head-to-head against a positive is substrate-specificity evidence — the BR/CR argument verbatim. Both kind='homopolymer', polymer_class='polythioester', unparented. Also adds the compound 3-Mercaptopropionate, without which the paper's three copolymer batches (12, 23 and 36 mol% 3MP) are one indistinguishable P(3HB-co-3MP). Five refusals are recorded in REJECTED_TERMS with their arguments: a generic PTE row (the class belongs on polymer_class, and PTS/PTC carry no full_name so reading either as "polythioester" would be a PHC-style guess); parenting onto PHB/P3HP/PHA (the IR-under-NR mistake, and an "all PHB" query would return a polymer no PHB depolymerase touches); merging the two into one row (different monomers, and Table 4 scores poly(3MB) - where poly(3MP) reads nd — a measured negative and an untested pair are not the same record); the P3MP/P3MB spelling (one interior character from P3HP/P3HB, and that character is the finding); and 3-Mercaptobutyrate as a compound (nothing to add it for — 3MB's polymer is a homopolymer, so no substrate_composition needs it). The legacy P(3HB-co-3MP) row is not reclassified: the seeder fills NULLs, legacy rows are a curator's (the NR precedent), and that backbone is genuinely half oxoester, so neither class is honest for it.