Skip to content

Screening keyword highlighting

Project administrators configure two keyword lists, inclusion keywords and exclusion keywords. When a reviewer screens a study on the stage review page, matches for those terms are highlighted inside the PDF shown by the Study Source panel's integrated viewer. The feature is visual assistance only: it never records, suggests or changes a screening decision, and it performs no ranking or classification.

Related issues: #671 (epic), #1366 (highlight keywords from inclusion/exclusion criteria).

Scope and slices

Slice Content Status
A Domain model + pmProject persistence, project-admin endpoint, feature flag, Screening Settings UI, docs Implemented in #3491
B Text layer + highlight engine in StudyPdfViewerComponent, panel/stage-review wiring, ADR-014 addendum Implemented in #3492
C Title/abstract highlighting in the stage review Study Source card and legacy study card (KeywordHighlightedTextComponent) Implemented in #3522
D Popup viewer parity: optional keywords field on the popup state message (#3504) Implemented in #3668
E Title/abstract highlighting in the detached study source window: optional keywords field on the source-window state message (#3504) Implemented in #3673
Follow-ups (#3504) Popup viewer parity (slice D), whole-document match counts / jump-to-match, diacritic folding (#3682), title/abstract highlighting in the detached source window (slice E) Not started unless noted

Out of scope by design: OCR for image-only PDFs, automatic include/exclude, AI classification, ranking, any change to screening answers or the decision card.

Flag decision

New flag screeningKeywordHighlighting (default off everywhere; featureFlags.services: [api], web half tsName: screeningKeywordHighlighting). It gates:

  • the Screening Settings "Keyword highlighting" section (web),
  • the PUT api/projects/{projectId}/screening-keywords endpoint ([FeatureGated], 404 while off),
  • the highlight layer in the viewer (web; the viewer itself is additionally behind integratedPdfViewer),
  • title/abstract highlighting on the stage review page (web; slice C, not behind integratedPdfViewer),
  • title/abstract highlighting in the detached study source window (web; slice E): the stage review page sends no keywords to the window while the flag is off.

Why flagged: the settings surface would otherwise appear for every project administrator in production while the viewer that consumes it stays behind integratedPdfViewer; the flag keeps the two surfaces coherent and provides a kill switch. Reading the lists is not gated (they ride on the project DTO; absent fields deserialise as empty). No environment enablement is part of this work.

Domain model (slice A)

  • Project.ScreeningInclusionKeywords and Project.ScreeningExclusionKeywords: IReadOnlyList<string> backed by private List<string>? fields, mapped with MapField(...).SetElementName(...).SetIgnoreIfNull(true) in ProjectRepository.RegisterProjectAndJobMappings, so existing documents need no migration and store no element until first set. Included in the summary projection so every project read carries them.
  • Single mutator Project.SetScreeningKeywords(IEnumerable<string> inclusion, IEnumerable<string> exclusion) applying ScreeningKeywordRules:
  • trim, collapse internal whitespace to one space, drop empties;
  • de-duplicate case-insensitively within a list (first occurrence wins, original casing kept);
  • limits: 100 characters per term, 200 terms per list (ArgumentException → 400);
  • a term that appears in both lists (case-insensitive) is rejected (400); the reviewer-facing semantics of such a term are undefined, and overlapping phrases are handled at render time instead.
  • Existing Project.Keywords (project metadata tags, JSON-Patch path /keywords, anonymous GET api/projects/keywords) is unrelated and untouched; the new names are deliberately explicit.

API (slice A)

  • PUT api/projects/{projectId:guid}/screening-keywords, body ScreeningKeywordsUpdateDto { inclusionKeywords: string[], exclusionKeywords: string[] }, [Authorize(ProjectAuthorization.ProjectDesignPolicy)] (the existing project-administrator gate, Administrator group by default), [FeatureGated(Flag.ScreeningKeywordHighlighting)]. Returns ScreeningKeywordsDto { projectId, inclusionKeywords, exclusionKeywords } with the normalised lists.
  • Read: both lists added to ProjectDbBaseDto, so the generated web IProject gains screeningInclusionKeywords / screeningExclusionKeywords and the stage review page's existing selectCurrentProject signal carries them with no extra request.
  • Modelled on the category-guidance endpoint (commit ffadc82ec).

Settings UI (slice A)

Screening Settings page (project/:projectId/admin/screening-settings, already guarded by projectDesignGuardFn): a new "Keyword highlighting" section under the criteria block, flag-gated, with two app-chip-input lists (inclusion, exclusion), save/discard, a dirty guard integrated into the page's existing canDeactivate, and explanatory copy: highlights are visual only; whole-word matching; trailing * for prefix matching; highlights need a PDF with extractable text. Save goes through a new projectDetailActions.updateScreeningKeywords effect → generated client → reducer merges the returned lists into the project entity.

Matching model (slice B, pure TypeScript, unit-tested)

  • Input: the page's text-content items (TextLayer.textContentItemsStr), joined in order; an item with hasEOL contributes a newline. Positions map back to the item index and offset.
  • Folded matching: text and terms are both folded one code point at a time: NFKD (compatibility forms such as fi and full-width letters expand; accents split off), combining marks (\p{M}) removed, lower-cased with full case folding for ß → ss (and ẞ) and final ς → σ, then NFC. So cafe matches café (precomposed or decomposed), fish matches fish, ＲＡＴ matches rat and strasse matches Straße, in both directions. The fold is linear, with a per-code-point cache.
  • Index map: every folded code unit records the [start, end) range of the original code point it came from. An expansion (fi → fi, ß → ss) gives each produced unit the whole source range. A code point that folds to nothing (a combining mark) extends the end of the preceding character's range. Match ranges are therefore always whole original code points: a surrogate pair is never split, and a match ending on a base character includes its trailing combining marks. The mask and segments below use original offsets.
  • Whitespace inside a phrase matches any run of whitespace, including line breaks.
  • Whole-word by default: a match must not be preceded or followed by a Unicode letter or digit (evaluated on the folded text). A trailing * on a term removes the trailing boundary (prefix match). No other operator syntax; all other characters are literal.
  • Overlaps and duplicates: matches from both lists are painted into a per-character mask; the mask is then cut into segments classified inclusion, exclusion or both. Nested, overlapping and repeated terms therefore merge without double-wrapping.
  • Cap: if a page yields more than 5,000 matches the highlighter stops and reports the cap (defensive).

Rendering (slice B)

  • After each successful page render in StudyPdfViewerComponent._renderPage, when a keyword config is present: page.getTextContent() → new TextLayer({ textContentSource, container, viewport }).render() into a <div class="textLayer"> that is a sibling of the canvas inside a position: relative wrapper sized to the canvas CSS width, with --scale-factor set to the render scale. The text layer is cancelled/cleared on the same paths that cancel the canvas render (request token, _settleActiveRender, _destroyDocument), and a cancelled or superseded render never paints.
  • Highlights wrap matched ranges in <mark class="syrf-keyword syrf-keyword--inclusion|--exclusion|--both"> inside the text-layer spans. Text-layer text is transparent; marks use mix-blend-mode: multiply so the PDF glyphs underneath stay readable. Elements are created through the container's ownerDocument so Dockview panel windows (live DOM relocation) work.
  • Accessibility: colour is never the only cue. Inclusion marks carry a solid 2px bottom border, exclusion marks a double bottom border, both shows both; each mark has aria-label="Inclusion keyword" / "Exclusion keyword". Colours are theme tokens (--syrf-keyword-inclusion-*, --syrf-keyword-exclusion-*) defined in both the light and .global-dark-theme blocks of syrf-theme.scss; the PDF page stays white in both themes, so the tokens are tuned for a white page.
  • Status line under the page navigation: "Keywords: N inclusion, M exclusion on this page" plus a legend swatch; when the page has no text items: "This page has no extractable text, so keyword highlights cannot be shown." SyRF does not perform OCR and the copy does not promise it.
  • Live configuration: the viewer takes keywordConfig = input<ScreeningKeywordConfig | null>(null); a change re-runs the highlighter on the existing text layer without reloading the document or altering _applyState, revision, retry or heartbeat semantics. The panel's _pdfIdentity key is unchanged, so a keyword change never remounts the viewer.
  • Study switches: the viewer is recreated per study by the host; all highlight state is input-derived.
  • Manual search and ordinary interaction: no key handling is added; the text layer makes browser find-in-page and text selection work on the rendered page, which they did not before.
  • Popup viewer (/pdf-viewer): shows no highlights in slice B; slice D below adds them.
  • Protocol: the syrf-study-pdf-viewer-state message gains an optional keywords?: { inclusionKeywords: string[]; exclusionKeywords: string[] } | null. It sits on the message, not on StudyPdfViewerStudy, so it plays no part in the popup's study comparison and a keyword change can never look like a study change.
  • Rolling-deploy compatibility: an absent field is accepted and means "no highlights", as directUrl did before it. A stage-review tab loaded before the deploy keeps working with a new popup (no highlights), and a new stage-review tab keeps working with an old popup: the old guard ignores unknown properties, so the popup shows no highlights.
  • isStudyPdfViewerMessage stays strict: when present and non-null, keywords must be an object with two string arrays; anything else rejects the whole message.
  • Coordinator: StudyPdfViewerCoordinatorService.setKeywords(config) normalises the lists and posts the change at the current revision. A revision bump would make the popup treat the state as new and reload the document. Study changes still bump the revision and carry the current keywords. null (flag off) and a config whose lists are both empty send no field at all, so the message is the same as before this slice. Unchanged lists post nothing.
  • Stage review: an effect forwards the same pdfKeywordConfig signal the inline panel uses (so it rides screeningKeywordHighlighting) to setKeywords; ngOnDestroy resets it to null before clear().
  • Popup: the routed /pdf-viewer page is StudyPdfViewerComponent itself, so no host binds its keywordConfig input there. The component keeps the keywords from the latest authenticated state in a private signal. Its normalised _keywords computed reads keywordConfig() ?? received and drives the same untracked re-highlight effect as the inline viewer. The popup adopts keywords before the revision checks, because they are not revisioned and messages from the one opener arrive in order. _applyState, revision adoption, retry backoff and heartbeat handling are unchanged.

Title and abstract (slice C)

  • syrf-keyword-highlighted-text (src/app/shared/keyword-highlight/) runs the slice B matchKeywords on one string and renders the result as text plus <mark class="syrf-keyword syrf-keyword--…"> chunks with the same aria-labels as the viewer ("Inclusion and exclusion keyword" for both). With a null config it renders the bare text, so flag-off output is unchanged.
  • Stage review renders the title and abstract through it with the same pdfKeywordConfig signal the PDF panel uses, in both the redesigned Study Source card and the legacy study card. The abstract is matched on the newline: 2 pipe output, so wrapping is unchanged. The header band title is not highlighted.
  • Quote fidelity: the component sits inside the elements carrying data-quotable / data-quote-source, and marks add no characters, so the AF2 quote toolbar resolves the same element and quotes the same text. The component adds no host styles; white-space is inherited from the host element (.preline on the abstract).
  • Mark styles for ordinary text live in global-styles/_keyword-highlight.scss (no transparency or blending); the viewer's text-layer rules stay component-scoped and more specific.

Detached study source window (slice E)

The redesigned stage review can move the Study Source card into its own window (/study-source-window, src/app/study-source-window/). The window receives its content over postMessage from a component-scoped StudySourceWindowCoordinator; before this slice it showed the title and abstract without highlights.

  • Protocol: the source-state message gains an optional keywords?: { inclusionKeywords: string[]; exclusionKeywords: string[] } | null. It sits on the message, not on StudySourceSnapshot, so it plays no part in the coordinator's source comparison and never bumps the revision.
  • Rolling-deploy compatibility: an absent field is accepted and means "no highlights". A review tab loaded before the deploy keeps working with a new window (no highlights), and a new review tab keeps working with an old window: the old guard ignores unknown properties.
  • isSourceWindowMessage stays strict: when present and non-null, keywords must be an object with two string arrays; anything else rejects the whole message. Like the rest of that guard it also bounds sizes (at most 1,000 terms per list, 1,000 characters per term; generous against the server's 200 terms / 100 characters, so a rules change there does not break the window).
  • Coordinator: setKeywords(config) normalises the lists and posts the change at the current revision. The window clears the reviewer's captured selection and any pending quote when the revision changes, so a revision bump would lose the reviewer's work. null (flag off) and a config whose lists are both empty send no field at all, so the message is the same as before this slice. Unchanged lists post nothing. Every later state (study change, target change, handshake) carries the current keywords.
  • Stage review: an effect forwards the same pdfKeywordConfig signal the inline card uses (so it rides screeningKeywordHighlighting) to setKeywords; ngOnDestroy resets it to null, and the coordinator's own ngOnDestroy drops it too.
  • Window: the title and abstract render through syrf-keyword-highlighted-text with the keywords from the latest accepted state. The keyword signal compares structurally, so a state that resends the same lists (for example a comment-target change) does not re-render the text under the reviewer's selection. The "Abstract not available" placeholder is not highlighted.
  • Quote fidelity: the window captures quotes from [data-source-kind] regions (its own equivalent of the inline data-quotable). The component sits inside those elements and marks add no characters, so a selection that starts or ends inside a mark resolves the same region and quotes the same text; the coordinator's normalizedSourceText check is unchanged.
  • Styles: the window is a route of the same Angular app, loaded from the same index.html, so the global styles.scss (which includes _keyword-highlight.scss) and the --syrf-keyword-* tokens on html / .global-dark-theme in syrf-theme.scss apply there without changes.

Decisions made here that Chris may want to revisit

  1. Whole-word matching with trailing * as the only wildcard (alternative: substring matching).
  2. Rejecting a term present in both lists at save time; overlapping phrases render as both.
  3. Keyword lists live on the Screening Settings page under the project-design permission, not on General Settings and not per stage.
  4. Highlight colours and the non-colour cues (solid vs double underline).
  5. Popup-viewer parity shipped as slice D; empty lists send nothing to the popup (no text layer), while the inline viewer still builds a text layer for an enabled config with empty lists.

Testing

  • .NET: ScreeningKeywordRules unit tests; BsonClassMapTests round-trip and absent-element tests; controller authorization/route attribute test; RuntimeFeatureFlagCatalogTests and ProjectAuthorizationActivityParityTests remain green.
  • Web: matcher unit tests (case, phrase whitespace, word boundaries, prefix wildcard, overlaps, duplicates, cap); highlighter DOM tests against a fake text layer; viewer spec additions (render → text layer → marks; cancelled render paints nothing; config change re-highlights without reload; no-text page notice); panel and stage-review wiring specs; popup protocol guard (absent/valid/malformed keywords), coordinator (keyword change keeps the revision, study change bumps it, clear() drops keywords) and popup forwarding specs; source-window protocol guard (absent/valid/malformed keywords), coordinator (keyword change keeps the revision, omitted when off or empty) and window rendering specs (marks from received keywords, none when absent, selection kept on a keyword change, quote text identical across marks); settings component spec; effects/reducer specs; guard specs (syrf-theme.spec.ts, no-hardcoded-help-urls.spec.ts).
  • Browser check on a preview environment with the seeded "Ready for Annotation" project (fixtures under assets/seed-pdf-fixtures/seed/ contain selectable text).