What's new in PlasticDB

Added

• Standalone binding assays and typed endpoints with explicit units, censored and qualitative results, provenance and preparation links, separate from degradation.
• Public binding API and browse page, dedicated dataset/release tables, and revision/withdrawal handling that preserves existing degradation labels.
• Binding vocabulary and adsorption-isotherm, TIRF and FRAP methods.

Added

• Explicit reviewed parent-to-variant reclassification, with pinned source evidence, actor definitions, complete preparation coverage and exact assay conservation.
• Typed, durable reclassification audits and direct public redirects to the variant, including visibility checks and subsequent variant merges. Historical endpoints survive orphan cleanup and source-paper deletion.

Fixed

• Curator-reviewed parent consolidation can preserve explicitly mapped terminal constructs without broadening automatic sequence matching. Sequence hashes, source evidence and variant definitions are checked again before applying.
• Identical parent reference coordinates can retain existing fusion variants without inventing sequences or treating fusions as point mutants. Conflicted structures can remain on archived records with an explicit audit trail.
• Reviewed variant folds can compare tagged sequences in a common core frame, checking mutation coordinates while retaining each original sequence and offset.
• Paper revisions retain historical sequence-frame records rehomed from merged accessions, while still retracting dropped assertions about current records.

Fixed

• Parent protein merges leave previously retired variants on their original parents, preserving their redirects, sequences and merge audits. Retired variants no longer create active fold collisions or move to the survivor.
• A retired variant already on the survivor still reserves its name; incoming live collisions remain explicitly refused under the existing unique index.

Added

• Separate source-scoped protein constructs and enzyme preparations, with explicit reference/expressed/assayed sequence roles, coordinate frames, separate chains, tags/linkers/fusions, modifications and paper evidence. Missing sequences and unknown tag status remain explicit.
• Experiment-to-preparation associations preserve roles and amounts for multiple preparations, including standalone enzyme characterisations. Curator editors and extraction contracts retain the complete material descriptions.
• Public construct/preparation views, actor-page material panels, and four additive downloadable/snapshot tables, filtered by published actor/evidence.
• Exact legacy variant sequence snapshots before approved variant folds, with source attribution and explicit structure-reference associations.

Fixed

• Promotion, revision, withdrawal, orphan cleanup and merges preserve or retract material evidence with its source. Revising one ingestion no longer removes a shared interaction still owning another ingestion's experiments.

All new tables ship through the existing schema-init release step. No automatic corpus-wide reinterpretation or merge is performed. See docs/construct_preparation_layer.md for compatibility and migration policy.

Identity sources, sequence frames and curator-only correspondence. **Migration 0032_identity_sources** (three new tables; python -m app.db.init_schema then alembic upgrade head). Deploy before the producer starts sending the fields; the producer strips them until /openapi.json advertises them. See DECISIONS.md §9.1.

Added

• identity_sources on protein, protein_variant and protein_variant.parent_protein in the extraction contract: where an identity value (sequence, accession, parent, mutations) came from when it was not the paper — cited_paper, author_correspondence, reference_record, genome_recovery, database_lookup (plus paper/supplement), with who/when/reference, verified_against and an optional evidence file + sha256. Absent means "from the paper"; every existing payload stays valid. Vocabulary at GET /api/v1/vocabularies (identity_source) and the vocabulary catalog.
• sequence_frame (precursor / mature / construct / pdb_chain), construct_span and construct_tags on the same three objects. Recorded for curators; not used for dedup.
• Promotion records both per (entity, asserting ingestion) in identity_sources / sequence_frames, with the entity's accession and the asserted values frozen on the row. A replace-mode revision retires the superseded runs' rows; protein merges and variant folds re-home them onto the survivor; orphan cleanup removes them with their entity.
• Correspondence upload (correspondence on POST /ingestions/{id}/files) into ingestion_correspondence_files under a private prefix — never a supplement. Curator-only list and download (GET /ingestions/{id}/correspondence[/{file_id}], attachment, private, no-store). Linked from an entry by name and hash at approval.
• Audit: GET /api/v1/admin/identity-sources (curator; filter by source, accession — current or as asserted) and GET /api/v1/admin/sequence-frames.
• Curation form controls for every new leaf (121 → 157), and a small "Not from the paper: supplied by the corresponding author (sequence)" marker under the sequence on protein and variant pages — the source type only.

Security

• Identity provenance is redacted for non-curators. POST /ingestions, PATCH /ingestions/{id} and POST /ingestions/{id}/upload-markdown accept any signed-in account and echo extracted_data; every such response now goes through one helper (IngestionResponse.for_viewer → redact_identity_provenance), which cuts each identity_sources entry to {source, fields}. Curators still see everything, and the stored record is untouched. A structural test fails if a non-curator route builds an unredacted ingestion response. Follow-up, out of scope here: those producer routes still accept any CurrentUser; they should require an ingest-capable role.

Fixed (review)

• Linking and replacing evidence are serialised. Approval's evidence lookup, the correspondence replace and the delete all lock the file row (FOR UPDATE), so a replace can no longer swap bytes under a link being made. Replaced objects are deleted only after the request commits, so a failed multi-file upload never destroys an existing file. Every write gets its own object key (content hash plus a per-write version), cleanup re-checks after the commit that no row references a key before deleting it, and one request naming the same correspondence file twice is refused (422). A request that fails before committing (a later part's 413/409/422, a storage error) deletes what it wrote. A commit that raises may still have committed, so then nothing is deleted: the keys are logged and left for reconciliation. POST /ingestions/{id}/files sets 30 s SET LOCAL statement/lock/idle-in-transaction timeouts as its transaction opens and accepts at most 10 correspondence parts, so the window from the first correspondence write to the commit is at most (10 × 4 + 2) × 30 s = 21 min — under the reconcile script's 1 h minimum age. python -m scripts.reconcile_correspondence_objects (dry run by default, --apply, --older-than-hours ≥ 1, default 24, each key re-checked in a fresh query right before deletion) lists and removes any object under private/correspondence/ that no row references, e.g. after a crash. Key versions are a full uuid4 — so neither a duplicate in one request nor two interleaved replacements can delete the object the final row points to.
• Distinct assertions stay distinct. The per-payload dedup key includes the normalised asserted values, so two different accessions asserted for one sequence-folded protein are two audit rows.
• Linked evidence is immutable. Re-uploading a correspondence file under the same name with different bytes is a 409 once an approved entry links it; the same bytes again are a no-op. An unlinked file is replaced and its old object removed.
• DELETE /ingestions/{id}/correspondence/{file_id} (curator): removes an unlinked file and its object; a linked one is refused (409).
• Non-Latin-1 filenames download. Content-Disposition carries an RFC 5987 filename* and an ASCII fallback; an em dash in an email subject was a 500.
• Clearing a construct span in the form ({start: null, end: null} or {}) means "no span" instead of a 422; spans are bounded (≤ 100 000) so an overflow is a 422, not a database error.
• Approve reports evidence that did not link (identity_evidence_unlinked, plus a log line).
• A revision that drops an entry clears the public marker: the superseded rows' proteins are invalidated in the cache.
• 0032's downgrade documents and logs that it drops the provenance rows.

Security: the public chatbot's SQL runs as a least-privilege database role that can read only public-rows-only views, so the regex SQL guard is no longer the security boundary. Migration 0031_chat_public_views (creates schema chat_public and its views; touches no table). Needs a deploy (API + migrate job), then a one-time role provisioning and a new secret — until then the chat keeps today's path and warns loudly. See DECISIONS.md §11, "The chatbot reads views, as its own role".

Security

• Chat SQL runs as plasticdb_chat, a dedicated login role whose only privileges are USAGE on schema chat_public and SELECT on its views (via the NOLOGIN plasticdb_chat_reader). It has no privilege on any table in public, so tables added later — including the identity-provenance tables of #605 — are unreachable by default. The role is read-only by default, statement-time-limited, and pinned to search_path = chat_public.
• The guard had been bypassed through PostgreSQL lexical features (escape-string literals, unicode-escaped identifiers, schema-qualified relation names, and SQL-looking text inside string literals); one class allowed an anonymous chat request to read non-public rows. With the role in place a guard bypass reaches nothing private. The guard also now refuses those lexical forms outright, refuses quoted identifiers and the XML table functions, and scans with string-literal contents blanked.

Added

• app/db/chat_public.py: the chat_public views (security barrier; public rows only, mirroring app.services.query; only the columns the chat needs). Built by migration 0031, by create_schema on a fresh database, and rebuilt by python -m app.db.init_schema on every deploy.
• app/services/chat_db.py: a separate NullPool engine from CHATBOT_DATABASE_URL, read-only transaction per query. Chat SQL never uses the application session when it is set.
• CHATBOT_REQUIRE_RESTRICTED_ROLE (default false): when true, chat SQL is refused unless CHATBOT_DATABASE_URL is set. Flip it after the role is deployed. CHATBOT_STATEMENT_TIMEOUT_MS (default 15000).
• scripts/chat_restricted_role.sql (idempotent; superuser; after the release is deployed) and python -m scripts.provision_chat_role (dry run unless --apply; password from --password-stdin or CHATBOT_DB_PASSWORD, sent as a SCRAM verifier; connection errors are reported without the URL or exception text). Provisioning repairs unexpected memberships, PUBLIC/chat table, column and sequence grants, CREATE on schemas (revoked from PUBLIC), default ACLs, PUBLIC EXECUTE on application and extension functions, and the implicit PUBLIC EXECUTE default for new functions; grants SELECT on the approved views by name; and then verifies the boundary — memberships (direct and transitive), ownership, the exact chat_public inventory and each view's definition against app.db.chat_public, column and sequence privileges, catalog objects by provenance (stock or allowed extension only) plus an explicit sensitive-catalog deny list, executable functions, effective function defaults, large objects and default ACLs — failing the whole run on anything unexpected. Its output never includes anything derived from the connection URL.

Changed

• The chatbot's schema hint describes the chat_public views (same names and columns as the tables they front).
• With no CHATBOT_DATABASE_URL, chat SQL keeps the previous path and logs a SECURITY: warning at startup and on every query.

Removed

• scripts/chatbot_readonly_role.sql, which was never wired in, granted SELECT on base tables and carried a placeholder password. The new script disables its plasticdb_chatbot role if it exists.

Review fixes to the containment detector (1.53.0). Report-only, no migration, no write path. Needs a deploy (API).

Fixed

• Per-request all-pairs cost. The containment report is now cached, keyed by a digest of the report inputs (sequences, tags, stored cleavage sites), and the all-pairs search runs in a worker thread, off the event loop.
• TEV scar (ENLYFQ[GS]) is stripped only alongside a His run, so a real C-terminus is no longer trimmed.
• Leading Met is kept on post-SignalP chains; core_sequence agrees with frame_rows after a cut.
• Stored cleavage sites are used as residue coordinates on the normalised chain; offsets are computed on the normalised string.
• Containment plan nulls the survivor, classification and move fields it cannot honestly fill.