Export template source from be30143a326d5d7d7b0cbabea5bc5ac0301ef5b9:examples/local-first/packages/language-reader-runtime

README.md

@@ -0,0 +1,398 @@
+# Reading package
+
+This package implements the Chinese `language-reader` workspace. Its independent
+contracts are LanguageReader, ReadingReader, ReadingDocument, ReadingFeature, ReadingJob,
+ReadingVocabulary, ReadingWord, and ReadingOwner. Baseline Studio, the workspace agent, and checked
+source activation remain available. Reading entities expose AgentDocumentation;
+read `guide` to find this resource and its extension points from an entity.
+
+## Run the workspace
+
+From the Quixos repository:
+
+```sh
+nix run .#camino-dev -- start --template language-reader --dir /home/timothy/qx-language-reader --agent
+```
+
+The command prints a local browser launch link. Reusing the directory resumes
+the same workspace and its independent authoring workbench. A different
+directory creates a fresh workspace from the Chinese template.
+
+## Entities
+
+LanguageReader owns documents, annotation features, vocabulary, and durable jobs.
+Documents preserve their exact original text and source attribution. Their
+Canonical sourceRevision is a content hash; variant revisions identify an immutable
+adaptation. All ranges use original UTF-16 offsets in that document.
+Creating a rewritten document retains the original document. Features contain
+presentation preferences and a processor key; schemas and implementations belong
+in code. Feature changes use expectedRevision to protect concurrent edits.
+
+Inference is admitted through a persisted narrow Agent. Each scope request owns
+a thread and request; managed graph edges retain those associations. Completion
+subscriptions validate structured output and record results without an open
+reader tab. Reading annotations default to Luna without reasoning and fast
+mode. Cache keys include source revision, semantic configuration, scope, instructions,
+output schema and actual context. Model routing/profile and scheduling preferences
+do not invalidate completed semantic results. Results persist indefinitely until
+explicit regeneration or deletion; attempt history is separate from the indexed
+canonical successful result. Declarative model processors retain
+results across unrelated runtime and presentation edits. Local processors and
+prompt closures conservatively include the package source fingerprint. `ReadingFeature.processorRevision` exposes the checked processor identity
+without copying its implementation into graph state. Old processor results are
+hidden and old in-flight completions become stale after source activation.
+Duplicate requests reuse the job, including a recorded
+failure; explicit retries are bounded. Old feature results cannot replace newer
+configuration. The document source is untrusted data, not developer instructions.
+
+## Add an annotation
+
+Edit `src/processors.ts` or import a new implementation there. A processor
+declares its label, computation strategy, scope, and instructions. A local
+processor returns source ranges directly. Section processors produce at most
+180 ranges in one pass over a bounded section. Focus processors explain a word
+or sentence and use neighboring text as context. Chinese initial features are
+in `src/examples/chinese.ts`; French implementations are source examples in
+`src/examples/french.ts`. Choosing another language is a workspace template or
+source change, not an in-app preset switch.
+
+Generation and reveal are independent. A single section-wide tagging pass can
+prepare before reading. A feature requiring one model call per word must be
+focused, cached, and lazy. Heavy hover requests should wait for deliberate dwell;
+cached or local annotations reveal immediately. Never start a per-word background
+fan-out to satisfy a precomputation setting. Leaving hover discards interest in
+its result; the durable request may still finish and populate its cache.
+
+Reader settings use a large modal with a reader selector, General settings,
+annotator navigation, and breadcrumbs. A document attaches to a ReadingReader;
+settings apply to all its attached documents. Cloning copies settings and creates
+independent annotator objects. General settings include size, maximum width
+(0 is unbounded), CSS font family/fallback list, and background.
+
+Premades insert enabled copies of existing processors. New annotators start with
+a visual type, optional title and type-specific color/font suggestions, then a
+natural-language behavior request. There is no model/content chooser. The agent
+creates checked code, generates a title if needed, chooses preparation strategy,
+and supplies any implementation-specific configuration controls.
+
+One processor can present a cached item through several rendering hooks. Declare
+`Processor.renders`, for example:
+
+```ts
+renders: [
+  { style: "replace", reveal: "always" },
+  { style: "tooltip", reveal: "hover", content: "source" },
+]
+```
+
+Each hook keeps the original source range and data source. A replacement's color
+already controls its ink. A source tooltip slices the original French text
+locally, with no extra inference. `annotation-display.ts` filters/decorates the
+cached facts before all hooks; `render-hooks.ts` projects their appearances.
+Separate annotator objects can also share facts through `cache.identity` and the
+same semantic scope/options, as pinyin and hover pinyin do. Generation stays with
+the processor; reveal belongs to each rendering hook.
+
+Premades currently create one feature from code-owned catalog metadata. QX object
+initialization can install several edges from supplied references, including their
+reciprocal relationships. It does not clone neighboring objects. To instantiate
+a processor/renderer bundle as separate graph objects, create all of them with
+distinct creation keys in one checked transaction and pass references between
+seeds. `addFrenchReader` and `addReader(copy)` are working examples. No second
+construction mechanism is needed for multi-edge initialization.
+
+Text content loads before annotation history. Preparation waits for that history
+to prevent cache misses while it is still loading. Planning and display-mode
+changes have checked JavaScript implementations and run against the local graph;
+server validation remains the same. Measurement and segmentation caches survive
+page and annotation changes. Package source changes do not invalidate declarative
+model results unless instructions, schema, semantic options or actual context
+change. Update the task version in `processor-identity.ts` when changing the model
+request wire shape or shared task construction; the text planner is fingerprinted
+automatically.
+
+Styles are overhead, highlight, underline, color, tooltip, sidebar, replace, bold,
+and italic. Hover must not reflow text: labels, replacement, bold, italic, font
+changes, sizes and spacing use permanent or click activation instead. This also
+applies to custom render hooks. Tooltip/sidebar use hover or click. The renderer
+preserves canonical source mapping for replacement spans:
+selection/copy returns source text. Fitting words and short labels move together;
+oversized spans may wrap. Processing contours cover intervening rows and paragraph
+gaps with straight connectors, without diagonal spikes. The embedded feature author makes actual checked
+source changes, activates a successful candidate, and registers the new feature
+through LanguageReader.addFeature. It has source and graph capabilities; normal
+annotation inference has neither. Their prompts and capability settings are
+captured by the existing Agent admission implementation.
+
+## Check and activate
+
+Inside an authored workbench, use `qx-workspace check` in each changed resource
+and then `root/`. Activate its successful report with `qx-workspace activate`.
+Inspect the selected identity and new feature afterward. Preserve unrelated
+source and concurrent edits. Never report successful activation after merely
+writing files. Failed compilation or activation must preserve the working app.
+
+Renderer code and source-selection tests live in
+LanguageReaderPresentation/src/canvas and tests/canvas.test.ts. Follow its
+RENDERER_PROVENANCE.md when maintaining the recovered measured canvas approach.
+Output schema changes require updating validation and consumer code together.
+
+## Sources and read aloud
+
+Import pasted text or UTF-8 text files, Chinese Wikipedia articles, or opening
+excerpts from the existing Gutenberg catalog. Wikipedia imports retain their
+source URL and CC BY-SA attribution; requests use TextExtracts, a descriptive
+User-Agent, bounded results, caching, and timeouts. Difficulty rewriting creates
+a new document, leaving the imported original intact.
+
+Read-aloud settings store an ElevenLabs key and voice once in private workspace
+state. The public graph returns configured status, never the key. Playback uses
+the streaming text-to-speech endpoint for a selected passage, limited to 1,200
+codepoints per request; it does not automatically narrate an entire book. The
+network implementation is covered with mocked API responses; live paid audio
+requires a configured key and voice.
+
+Primary API references: [MediaWiki TextExtracts](https://www.mediawiki.org/wiki/Extension:TextExtracts),
+[Wikimedia User-Agent policy](https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy),
+and [ElevenLabs streaming text-to-speech](https://elevenlabs.io/docs/api-reference/text-to-speech/stream).
+
+## Verification
+
+`nix run .#language-reader-qualification` from the monorepo runs native graph
+integration and real Chromium interaction tests. Package checks also run
+processor/output validation, mocked network tests, canvas collision/Unicode
+selection tests, and hover scheduling tests. The browser suite checks actual
+CJK canvas metrics, pointer and keyboard copy, scrolling, bounded hover
+admission/cache reuse, the visual editor, imports, and the embedded feature
+conversation at desktop and narrow widths.
+
+## Inline annotator configuration
+
+Configuration belongs to annotator code and appears on its page in Reader settings.
+`Processor.config` is a list of `ConfigField` controls, or a function returning that
+list. Supported controls: `color`, `number`, `select`, `toggle`, `text`, with a key,
+label, typed default, optional numeric bounds and select choices. The grammar
+processor generates one color control per part of speech; names does the same for
+entity types. Declare every option in code so the server can validate it.
+
+`ReadingFeature.options` persists the JSON object. `configureOptions` checks
+`expectedRevision`, validates options and increments `configRevision` atomically.
+The UI edits a draft and saves all processor options together; appearance/reveal
+controls save directly when the processor explicitly declares them. `addFeature` requires `options` (use `"{}"` for defaults) and optional `authoring` (use null for premades).
+Concurrent edits reject cleanly. Changing display settings preserves raw cached results; only semantic inputs invalidate computation.
+
+`Processor.prompt(config)` can include validated config in developer instructions.
+Local processors receive config as their third argument.
+The presentation `annotationDisplays` registry can return per-item `color`, `fontFamily`, or
+`fontSize`, applied after strict model output validation. Model output stays the
+four-field range/text/tag contract; rendering choices are owned by checked code.
+Common options `fontFamily` and `fontSize` configure labels. Their style remains
+`overhead`; `options.placement` chooses `"above"` (default) or `"below"`. A
+`Processor.renders` hook can set `placement` independently. Both positions use
+fixed source lanes, and display changes reuse semantic caches. Wide labels gain
+faint alignment guides; hovering a label emphasizes its source without invoking
+word-hover tooltips or processors. Different
+fonts and sizes in the same annotation source stay on one row; each physical line
+reserves its maximum ascent/descent and shares the resulting baseline.
+
+The presentation renders declared fields automatically. For richer controls, add
+a React component to `annotatorConfigViews[processor]` in
+`LanguageReaderPresentation/src/annotator-config.tsx`; it receives `fields`,
+`values`, `disabled`, and `change(key, value)`. This component edits the same draft
+and validated options. Built annotators have no universal output-type editor.
+`Processor.settings` explicitly opts into color, font, or timing controls;
+`description` explains the behavior and `preset` defines a premade output.
+The New annotator panel can reuse any installed processor type with its code-owned
+controls, or describe a new type to the author. `LanguageReader.archiveType` takes
+`{processor, archived}` and updates `archivedTypes`, hiding a type from the creation
+library without changing its instances, implementation or caches. Restore it to
+make it discoverable again. `ReadingFeature.remove` removes only an instance.
+Creation choices are authoring instructions, not a universal runtime schema.
+Set `addFeature.authoring` to the supplied `authoringJob` when registering a new
+annotator; this links its original request and followups to that annotator.
+`customize.feature` links a change request to an existing annotator.
+`ReadingFeature.setEnabled` runs locally and preserves configuration revision
+and cached annotations across toggles.
+Do not bypass persistence or create per-word inference from a config component.
+
+The French word-gloss and passage-translation processors declare `profile: "cerebras-light"`
+to use GPT OSS 120B with low reasoning. Processor-owned profiles override the
+workspace default and participate in cache identity. Cerebras must be connected.
+
+The default inference profile is `quick-fast`: GPT-6 Luna without reasoning and
+fast mode. `LanguageReader.inferenceProfile` is configurable through the graph and
+captured for each request. Model-backed cache identity includes the effective
+profile; local processors use `local`. Feature authoring remains on balanced.
+
+Pending jobs mark the exact text segment with animated contours. This is
+a rendering overlay, independent of layout, and respects reduced-motion settings.
+Hover word feedback uses cached segmentation/layout immediately; heavyweight hover
+work still waits for dwell and reuses durable results.
+
+## Text processing and reading windows
+
+`src/text-processing.ts` is the reusable source-indexed planner for words,
+sentences, chunks and context. `ReadingDocument.plan` returns target ranges for
+`kind: "focus"` or `"nearby"`; it never starts inference. The UI reports visible
+and nearby source ranges, then uses the same admission/cache path for automatic
+preparation, clicks, dwell hover and retries. Pages are the default view; the
+Reading layout control also offers scrolling. Navigation persists mode and source
+position independently of pixel layout. Selection and cache ranges always refer
+to the original document, across pagination, font changes and reflow.
+
+Defaults: focus targets at most 400 graphemes; context includes up to two sentences
+before and after, capped at 1,600 graphemes. This also bounds unpunctuated paragraphs.
+Automatic preparation uses stable chunks of at most 800 graphemes, preferring
+sentence/word boundaries. Pages prepare the current page plus its neighbors;
+scrolling prepares roughly one viewport above and below. Only two background
+model jobs are admitted concurrently. Independent scopes from the same annotator
+may use both slots; duplicate scopes reuse their existing requests. Unsent queued
+scopes and admitted work both show processing outlines.
+Moving away replaces unsent plans; admitted results finish and remain cached.
+Local passes use the same chunks without model admission. Errors do not cause an
+automatic retry loop. Focused interactions are independent of preparation slots.
+
+Authors can override the defaults with `Processor.processing`, a partial policy
+or `(config) => policy`. Available fields: `chunkChars`, `focusChars`,
+`contextChars`, `beforeSentences`, `afterSentences`, `prefetchChunks`, and
+`context: "nearby" | "document"`. For long-range references, a processor might
+choose `{ context: "document", contextChars: 12000, chunkChars: 2000 }`.
+Sizes have a generous 80,000-grapheme ceiling and radii a 20-unit ceiling.
+These are author choices, not extra user settings. Large prompts/results still
+obey transport and output bounds; choose batch outputs responsibly. Generation
+rejects oversized targets so callers cannot accidentally bypass the planner.
+
+`passageTranslation` demonstrates lazy article translation, producing per-sentence
+sidebar notes in nearby chunks. Add it from Premade annotators. Adapt text creates
+a linked lazy text variant for either translation or difficulty changes.
+Hover feedback changes source ink immediately and never draws a word outline.
+
+## Tooltip presentation
+
+Features can use `style: "tooltip"` with `reveal: "hover"` or `"click"`.
+Choose Tooltip in the annotation editor, then choose its reveal trigger. Existing
+processors and generated/local content work unchanged; this is a presentation
+choice. `always` is unsupported for tooltips. Hover inference retains dwell,
+nearby preparation and cache behavior. Ready hover content opens immediately.
+The source word remains highlighted by ink on hover, without changing layout.
+
+`LanguageReaderPresentation/src/canvas/Tooltip.tsx` renders selectable React
+content in a native manual popover (the browser top layer). It escapes canvas,
+scroll-pane and component clipping while retaining component styles. Positioning
+uses source glyph rectangles in viewport coordinates, flips above/below and clamps
+to the viewport. Cards combine matching annotations and identify their sources.
+Hover cards allow a brief pointer crossing to select/copy their text. This grace
+period does not extend model hover intent. Click cards remain until dismissed or
+the anchor leaves view. Escape, outside clicks and the close button dismiss them;
+pagination, scroll and resize update or clear the anchor. No model work is owned
+by the tooltip component and no HTML from model output is interpreted.
+
+## Lazy text variants
+
+ReadingDocument.rewrite creates a variant linked by canonical; it does not start
+inference. adaptation stores immutable canonical ranges and completed outputs.
+prepareVariant admits only the next section, bounded to 1400 graphemes, with
+nearby context. Requests deduplicate and explicit retries are bounded to three.
+Validated completions append in canonical order. Output length changes do not
+change canonical section identities or invalidate ready prefix annotations.
+The reader prepares ahead near the current page; ordinary annotators then work
+on generated text. This path serves both difficulty adaptation and translation.
+A failed section retains all completed text and offers an explicit retry. Closing
+the UI does not discard an admitted request. Variants are separate documents and
+inherit the source's reader, which can subsequently be changed.
+
+## Embedded authoring
+
+customize captures reader, requested configuration, and prose intent separately.
+Jobs retain the reader link and a read-only configuration snapshot. Their threads
+are collapsed beneath that reader's settings; opening one shows a React request
+card rather than raw orchestration text, then ordinary follow-ups and questions.
+ConversationTranscript separates rendering from its graph controller, and the
+standard agent ThreadView also accepts renderItem for embedded request views.
+Before follow-ups, the author must re-read current reader settings and feature
+options; a request card records history and is not authoritative live state.
+
+## Product language
+
+Use concrete labels and direct copy. Avoid slogans, inspirational filler,
+decorative all-caps branding, and boilerplate AI disclaimers. Explain empty states
+with a useful next action. Caveats should identify a specific limitation briefly.
+These directions also live in the default Workspace Agent prompt and harness.
+
+## Semantic annotation caching
+
+Raw model/local facts are cached separately from display preferences. The pure
+cache.ts wrapper serializes processor identity, scope, generation and explicitly
+selected semantic options. Source revision/range and model profile complete the
+cache identity. Processor.cache.inputs opts into semantic fields; the default
+selects non-color config fields unless computation:false. prompt(config) sees
+only these semantic values. Changes to color, fonts, enabled state or display
+filters must never admit another model request. Install custom filtering/styling
+in LanguageReaderPresentation/src/annotation-display.ts, not model output.
+
+Pinyin batches every word in stable 160-grapheme nearby chunks, with tone marks
+and an estimated HSK level per token. The HSK slider filters cached facts locally.
+Hover pronunciation uses the same computation identity, including sentence context,
+so it does not request inference per word or duplicate the overhead pass. Two
+background slots bound preparation; visible chunks precede neighboring chunks.
+Context-sensitive annotation caches stay scoped to the canonical source/range;
+only introduce bare-word reuse for a processor whose context-independence is
+explicit. For ambiguous pronunciations/definitions retain context.
+
+## Text library and generation
+
+The library lists texts; readers are reusable configurations attached to them.
+`ReadingDocument.archived` hides a text from the active collection without deleting
+its source, variants, annotations or threads. Restoring it changes only that flag.
+`LanguageReader.generateDocument` captures a reader and a bounded prose request,
+creates a durable `ReadingJob` of kind `text`, and uses the quick-fast inference
+agent. Only a validated `{title,text}` result creates and attaches the new text.
+The UI can close while the request runs; completion survives independently.
+
+`addFrenchReader` creates an independent reader with two click-only overhead
+sources and a small French sample. `src/examples/french.ts` supplies per-word
+contextual glosses for the selected sentence and a separate translation lane for
+a few sentences. Focus planning bounds the source range and supplies neighboring
+context. Clicking another passage replaces both visible scopes immediately.
+Ready results remain cached; revisiting a passage reuses them. This scope-based
+exclusivity belongs to click/focus annotations; ordinary word/sidebar annotations
+continue to follow their source word. Hover tooltip dismissal clears only native
+selection owned by the tooltip, so it does not intercept later canvas copying.
+
+Model annotations echo `source` alongside `start`/`end`; it must equal the exact
+original UTF-16 slice, including whitespace and punctuation. `text` remains the
+rendered pronunciation, gloss or replacement. `validateAnnotations` collects all
+indexed errors and strips the echo after validation. The job calls the shared
+`AgentRequest.correct` operation for at most two same-thread corrections, staying
+pending until valid or exhausted. Invalid results never enter the display cache.
+Local processors still receive range/content checks without an echo. An author
+may document a code-owned `verifySource:false` exception; `annotationSchemaFor`
+then supplies the corresponding schema. This affects semantic cache identity.
+
+The feature author uses `balanced-fast` (Sol, medium reasoning, fast mode).
+Inference-based features must be tried on actual reader text or representative
+sample text. Start with `cerebras-light` when connected; try Luna fast, then Sol
+fast if the output needs a stronger model. Check returned ranges and content,
+not just request completion. These are authoring choices, not reader settings.
+
+## Reader extensions and vocabulary
+
+Sidebar panels and word/selection context-menu actions are ordinary React reader
+extensions, separate from annotators. `src/extensions.tsx` in the presentation
+provides a shared `ReaderContext` and `ReaderAction`; `reader-extensions.tsx` is the
+installation point. Actions can inspect the exact original range or cached annotation
+tags. Context menus use the browser top layer and keyboard focus/dismissal. Drag
+selection does not generate click annotations. Panels can include arbitrary UI.
+
+`ReadingVocabulary` is a named graph list, shared through each reader’s optional
+`vocabulary` edge (otherwise the library default). Its standalone editor supports
+search, add/remove and rename. The SDK `ObjectPicker` presents checked paginated
+lists and explicit selection, creation, preview and open-in-new-tab. Existing words
+remain in their original storage; `ensureVocabulary` attaches unlisted entries to
+the first default list through public contracts. Cloning a reader shares its list.
+
+Settings show the current text alongside configuration. This preview projects
+cached facts through current appearance/display options, never initiates inference.
+Author new data models simply, and refactor to richer domain entities when needed
+while preserving user values via supported transformations. Independently scoped UI
+work can use bounded Codex subagents; join/check before activation.