Universal Value Extraction Pipeline

We didn't invent values.
We extracted them.

Human values are behavioral patterns — observable in text, extractable with deterministic computation, verifiable through accumulation. No LLM in the extraction stack. No hallucination in the training data.

Built from the full human spectrum — saints, monsters, and the complex majority in between. That middle ground is where the real signal lives.

See the spectrum →
System Class UVEP
LLM in Stack None
Classification Deterministic
Values Tracked 15
Phase 0 Complete ✓
Output Labels P1 · P0 · APY
Evidence Model Append-only
Doc Types 5 (weighted)
The Spectrum Principle

Most ethics datasets train on the poles.

The far-positive corpus teaches models to recognize virtue performance, not virtue. The far-negative corpus teaches recognition of monsters, not moral drift. The most useful training signal lives in the complex middle.

Most Useful Training Signal
Virtuous ← Complex Middle → Villains
Gandhi
integrity · patience · resilience
Strong P1
Lincoln
commitment · courage · humility
Strong P1
Churchill
high courage · complex integrity
Mixed Profile
MLK Jr.
extraordinary courage · asymmetric
Asymmetric
Malcolm X
value evolution documented
Evolution
JFK
high public · complex private
APY-rich
Nixon
documented yield under pressure
P0 Present
Oppenheimer
moral weight · responsibility
Conflicted
Marcus Aurelius
private journal evidence — action
Journal P1
Any Figure
no pre-labeling at ingestion
Data Decides

"A value stated in comfort is weak signal. A value demonstrated at real cost — under threat, under pressure, against interest — is strong signal. The resistance score is the measurement of that cost."

No figure is pre-labeled positive or negative at ingestion time. Classification emerges entirely from the data. The same extraction code processes Lincoln and Nixon identically. The resistance scores and marker patterns determine the label.

Extraction Pipeline

Ingest. Extract. Score. Classify.

A four-stage deterministic pipeline. Same input, same thresholds, identical output every time. Reproducible means auditable. Auditable means trustworthy as training data.

Step 1
Ingest

UTF-8 source text is segmented into sentence-bounded passages (≤450 chars). Each passage is stored in documents.db with doc_type — the authenticity signal that flows through all downstream scoring.

cli/ingest.py
→
Step 2
Extract

Each passage is scanned against a 15-value keyword vocabulary. One observation per matched value per passage. Watermark-gated: only new passages since last run are processed. Never repeats.

value_extractor.py
→
Step 3
Resistance Score

Each observation is scored for the cost of holding that value in that passage. Formula: base + doc_type bonus + significance + text markers. Action-type passages score highest — documented deeds over words.

resistance.py
→
Step 4
RIC Classify

Each observation is classified P1 (held under resistance), P0 (failed/corrupted), APY (yielded under pressure), or AMBIGUOUS. Exported as JSONL: per-figure files + universal positive/negative training sets.

cli/export.py
Resistance Scoring

Measuring the cost of holding a value.

Resistance scores the authenticity of a value signal — how much it cost to hold that value in that documented moment. Range: 0.0 → 1.0. Additive formula with four independent signals.

Base +0.25 (always)
Floor contribution. Every observation receives this regardless of other signals.
Document Type Bonus +0.10 → +0.40
Authenticity of the source. Action (+0.40) → Journal (+0.35) → Letter (+0.30) → Unknown (+0.20) → Speech (+0.10). A private journal cannot be performance — the audience is zero.
Significance Bonus up to +0.30
Passage significance score × 0.40, capped at 0.30. Configured at ingestion time, applied uniformly across passages from that source file.
Adversity Text Markers +0.20 (if present)
Language detected in the passage: "despite", "even though", "but I still", "at a cost", "scared but", "might lose", "risk losing" — and 10 other adversity patterns.
Maximum Achievable
Action + full significance + adversity markers
1.0 (clipped)
resistance = clip(base + sig_bonus + doc_type_bonus + text_bonus, 0.0, 1.0)

base = 0.25
sig_bonus = min(significance × 0.40, 0.30)
doc_type = action:0.40 / journal:0.35 / letter:0.30 / speech:0.10 / unknown:0.20
text_bonus = 0.20 (if adversity phrase pattern matches)
Document Types — Authenticity Weights
action
+0.40
Export: 1.5×
Documented real-world behavior — highest stakes. Deeds, not words.
journal
+0.35
Export: 1.4×
Private writing. Zero audience pressure. Cannot be performance.
letter
+0.30
Export: 1.2×
Directed correspondence. Lower performance pressure than public address.
unknown
+0.20
Export: 1.0×
Default when source type is ambiguous. No authenticity adjustment.
speech
+0.10
Export: 0.8×
Public address. Highest performance pressure — lowest authenticity score.
RIC Classification

Relational Integrity Coefficient

Every value observation is classified into one of four labels. Applied deterministically during export — the same observation, the same thresholds, the same label every time.

P1
Held Under Pressure
Value demonstrated under meaningful resistance. The cost was real. The value held. The strongest training signal in the corpus.
  • despite · even though
  • stood firm · refused to give
  • nevertheless · persevered
  • maintained · stayed true
Confidence: 0.75 – 0.95
P0
Value Failed
Value yielded, rationalized, or corrupted in this passage. Documented failure — including self-confessed failures flagged correctly as P0.
  • gave in · gave up · yielded
  • i lied · i deceived · i caved
  • backed down · compromised my
  • i rationalized · i pretended
Confidence: 0.55 – 0.85
APY
Answer-Pressure Yield
External pressure explicitly detected and the value failed. The most structured training signal for how moral drift actually happens — under specific coercive conditions.
  • under pressure · when pressed
  • forced to · compelled to
  • to avoid punishment
  • or face consequences
Confidence: 0.95 (APY-resistance → P1)
Classification logic (priority order):
1. APY pressure detected? YES + failure markers → APY (0.95) | YES + no failure → P1 (0.95, APY-resistance)
2. Failure markers present? → P0 (0.85)
3. resistance ≥ p1_threshold (0.55) AND hold markers? → P1 (0.90)
4. resistance ≥ p1_threshold alone? → P1 (0.75)
5. resistance < p0_threshold (0.35)? → P0 (0.55)
6. Otherwise → AMBIGUOUS (0.40)
Constitutional Invariants

Rules that cannot be violated.

These constraints are structural, not stylistic — they define what the pipeline is and what makes it trustworthy as a training data source.

No LLM in Extraction
All classification is keyword regex + SQLite arithmetic. No model dependency, no API key, no rate limits, no variance. Deterministic → reproducible → auditable.
Deterministic Output
Same input + same thresholds = identical output every time. No randomness anywhere in the extraction or classification stack.
Append-Only Observations
value_observations is never updated or deleted. Only appended. The evidence record is immutable once written.
No Pre-Labeling
No figure is labeled positive or negative at ingestion time. Classification emerges entirely from resistance scores and marker patterns. Reputation bias cannot enter.
Watermark Continuity
Processing never repeats. Watermarks advance monotonically. Re-ingesting a figure only processes the new passages since the last run.
Doc-Type Transparency
Document type is recorded at ingestion and flows to every output field. Authenticity weighting is explicit in every observation, not applied post-hoc.
Roadmap

Where we are. Where we're going.

Phase 0 ✓

Foundation — Complete (Mar 2026)

Standalone pipeline operational. DocumentStore, ValueStore, resistance formula, value extractor, CLI ingest + export. 15 values, P1/P0/APY classification, JSONL output. Zero external dependencies.

Phase 1

Semantic Extraction

Replace keyword vocabulary with embedding-based clustering. Passage embeddings via sentence-transformers (BGE-base). FAISS/Qdrant ANN search against value cluster prototypes. Backward-compatible with Phase 0 baseline.

Phase 2

API Layer

FastAPI service wrapping the pipeline. POST /figures/{name}/ingest, GET /figures/{name}/profile, GET /figures/universal, GET /export/ric. Batch processing support.

Phase 3

Web Dashboard

Figure browser with value radar charts, corpus upload UI, universal registry heatmap, training set builder with filters by figure/value/doc_type/label.

Phase 4

Corpus Scale + HuggingFace Export

Batch ingestion CLI, multi-file figure support, corpus statistics, dataset card generation, and datasets-library-compatible format for load_dataset() integration.

Phase 5

SRL Integration

Port AiMe's Self-Reflection Layer — claim_extractor, ric_gate, trait_compiler, inflection_engine. Enables AI model output evaluation against the Ethos value corpus.

"We didn't invent values.
We extracted them from people who lived them —
and people who didn't."
Ethos · Universal Value Extraction Pipeline · Phase 0 Complete