# EmemoX full model evaluation · 2026-10-23 Run ID: `20261022-answer-recovery-v27-full`. Started: 2026-10-22T14:09:00.894809+00:00. Completed: 2026-10-22T16:58:28.819799+00:00. Source revision: `30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3`. Working tree dirty: `true`. Source patch SHA-256: `6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07`. These are project measurements under the recorded conditions, not an official leaderboard submission. Answer correctness, multiple-choice accuracy and evidence retrieval are different metrics. | Dataset | Metric | Value | Denominator | | --- | --- | ---: | --- | | LoCoMo | Answer correctness · LLM judged | 68.1% | 1049 / 1540 selected; 1540 judged | | LoCoMo | Evidence Hit@10 | 95.3% | 1982 scored / 1986 selected; 4 without evidence | | LongMemEval-S | Answer correctness · LLM judged | 80.0% | 400 / 500 selected; 500 judged | | LongMemEval-S | Evidence Hit@10 | 96.4% | 500 scored / 500 selected; 0 without evidence | | PersonaMem | Multiple-choice accuracy | 73.9% | 435 / 589 selected | ## Query decomposition LoCoMo: `rule_based`; enabled: `true`. LongMemEval-S: `memory_query_expansion`; enabled: `true`. PersonaMem: `rule_based`; enabled: `true`. ## Stage measurement dates and reuse Report completion is not a new measurement date for reused stages. Each stage retains its original measurement period and conditions. | Stage | Measurement | Started | Completed | Source run | | --- | --- | --- | --- | --- | | locomo.retrieval | Measured in this run | 2026-10-22T14:09:00.899373+00:00 | 2026-10-22T14:35:29.811041+00:00 | 20261022-answer-recovery-v27-full | | locomo.answers | Measured in this run | 2026-10-22T16:25:17.078737+00:00 | 2026-10-22T16:51:07.830340+00:00 | 20261022-answer-recovery-v27-full | | longmemeval.retrieval | Measured in this run | 2026-10-22T14:09:00.900520+00:00 | 2026-10-22T16:25:13.127917+00:00 | 20261022-answer-recovery-v27-full | | longmemeval.answers | Measured in this run | 2026-10-22T16:51:07.858213+00:00 | 2026-10-22T16:58:28.805164+00:00 | 20261022-answer-recovery-v27-full | | personamem.personamem | Measured in this run | 2026-10-22T14:09:00.899696+00:00 | 2026-10-22T15:13:31.756718+00:00 | 20261022-answer-recovery-v27-full | Stage `locomo.retrieval` source bundle SHA-256: `not applicable`; source manifest SHA-256: `3a79a55c0d253d02e61b49dccf48f3281ec7f35b89c4253eb46afbe4ad09c17e`. Recorded stage conditions: {"environment": {"buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "memoryGib": 48.0}, "execution": {"concurrencyNote": "Real retrieval under the original recorded process/worker settings.", "durationSeconds": 1588.911668, "retrievalProcesses": 1, "workersPerProcess": 8}}. Stage `locomo.answers` source bundle SHA-256: `not applicable`; source manifest SHA-256: `079e13f880f3025f463419c2845661a175c1e59cd477ae61611d311fcf7517d4`. Recorded stage conditions: {"environment": {"buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "memoryGib": 48.0}, "execution": {"answerProcesses": 8, "concurrencyNote": "Saved-evidence answer replay; process limit is shared across datasets when a global controller is recorded.", "durationSeconds": 1550.751603}}. Stage `longmemeval.retrieval` source bundle SHA-256: `not applicable`; source manifest SHA-256: `3a79a55c0d253d02e61b49dccf48f3281ec7f35b89c4253eb46afbe4ad09c17e`. Recorded stage conditions: {"environment": {"buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "memoryGib": 48.0}, "execution": {"concurrencyNote": "Real retrieval under the original recorded process/worker settings.", "durationSeconds": 8172.227397, "retrievalProcesses": 4, "workersPerProcess": 2}}. Stage `longmemeval.answers` source bundle SHA-256: `not applicable`; source manifest SHA-256: `079e13f880f3025f463419c2845661a175c1e59cd477ae61611d311fcf7517d4`. Recorded stage conditions: {"environment": {"buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "memoryGib": 48.0}, "execution": {"answerProcesses": 8, "concurrencyNote": "Saved-evidence answer replay; process limit is shared across datasets when a global controller is recorded.", "durationSeconds": 440.946951}}. Stage `personamem.personamem` source bundle SHA-256: `not applicable`; source manifest SHA-256: `e3a985c1cef5a9ea053bae25cbcd53ab70d8ac417f566c5ffd51ab21cee500ff`. Recorded stage conditions: {"environment": {"buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "memoryGib": 48.0}, "execution": {"answerProcesses": 1, "concurrencyNote": "Original one-process live retrieval and multiple-choice generation.", "durationSeconds": 3870.857022, "retrievalProcesses": 1, "workersPerProcess": 8}}. ## Method and limitations Measurement conditions: Dataset phases ran under their recorded process/worker settings; sum process durations include overlap. All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed. The same configured model generated and judged answers for LoCoMo, LongMemEval-S. Self-judging bias remains; this is not an independent human evaluation. LoCoMo and LongMemEval answers replay saved evidence. Generation latency is reported separately; replay lookup latency and replay answer latency are not live retrieval or end-to-end latency. Real retrieval latency comes from each dataset's retrieval stage. PersonaMem reports Recall latency plus generation latency. This sum has no independent wall-clock timer and excludes ingestion, embedding preload and full HTTP request timing. The JSON field meanEndToEndLatencyMs retains the native schema name for compatibility. Failed generation and judging remain in the selected-question answer denominator. Quality-gate failure does not imply an incomplete run; both states are recorded separately. Token totals and per-selected-question means are published only when every selected question has reported usage for that token field. Partial coverage remains unknown. Answer and judge usage are separate; judge usage is not user-answer token usage. Context bytes are not tokens. Cached and reasoning token categories are preserved separately when recorded. Parallel execution conditions differ from historical serial runs. This report makes no automatic latency-improvement claim. These dialogue-memory benchmarks do not measure robot perception or control. ## Recorded measurements and provenance The accompanying JSON contains the allowlisted measurements, models, failure counts, execution conditions and stage-specific binary/source hashes. ```json { "schemaVersion": "ememox-public-benchmarks-v1", "report": { "runId": "20261022-answer-recovery-v27-full", "startedAt": "2026-10-22T14:09:00.894809+00:00", "completedAt": "2026-10-22T16:58:28.819799+00:00", "date": "2026-10-23", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "build_type": "Release", "cpu_count": 12, "memory_gib": 48.0, "concurrencyNote": "Dataset phases ran under their recorded process/worker settings; sum process durations include overlap. All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed." }, "provenance": [ { "stage": "locomo.retrieval", "runId": "20261022-answer-recovery-v27-full-locomo-retrieval", "binarySha256": "7a534f0109a9603df097ce0fc9f42211ec347f20cff3f07999ff81acec6a5bb1", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "measurementMode": "measured", "startedAt": "2026-10-22T14:09:00.899373+00:00", "completedAt": "2026-10-22T14:35:29.811041+00:00", "sourceRunId": "20261022-answer-recovery-v27-full", "sourceBundleSha256": null, "sourceManifestSha256": "3a79a55c0d253d02e61b49dccf48f3281ec7f35b89c4253eb46afbe4ad09c17e", "conditions": { "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "memoryGib": 48.0 }, "execution": { "concurrencyNote": "Real retrieval under the original recorded process/worker settings.", "retrievalProcesses": 1, "workersPerProcess": 8, "durationSeconds": 1588.911668 } } }, { "stage": "locomo.answers", "runId": "20261022-answer-recovery-v27-full-locomo-answers", "binarySha256": "7a534f0109a9603df097ce0fc9f42211ec347f20cff3f07999ff81acec6a5bb1", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "measurementMode": "measured", "startedAt": "2026-10-22T16:25:17.078737+00:00", "completedAt": "2026-10-22T16:51:07.830340+00:00", "sourceRunId": "20261022-answer-recovery-v27-full", "sourceBundleSha256": null, "sourceManifestSha256": "079e13f880f3025f463419c2845661a175c1e59cd477ae61611d311fcf7517d4", "conditions": { "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "memoryGib": 48.0 }, "execution": { "concurrencyNote": "Saved-evidence answer replay; process limit is shared across datasets when a global controller is recorded.", "answerProcesses": 8, "durationSeconds": 1550.751603 } } }, { "stage": "longmemeval.retrieval", "runId": "20261022-answer-recovery-v27-full-longmemeval-retrieval", "binarySha256": "7a534f0109a9603df097ce0fc9f42211ec347f20cff3f07999ff81acec6a5bb1", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "measurementMode": "measured", "startedAt": "2026-10-22T14:09:00.900520+00:00", "completedAt": "2026-10-22T16:25:13.127917+00:00", "sourceRunId": "20261022-answer-recovery-v27-full", "sourceBundleSha256": null, "sourceManifestSha256": "3a79a55c0d253d02e61b49dccf48f3281ec7f35b89c4253eb46afbe4ad09c17e", "conditions": { "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "memoryGib": 48.0 }, "execution": { "concurrencyNote": "Real retrieval under the original recorded process/worker settings.", "retrievalProcesses": 4, "workersPerProcess": 2, "durationSeconds": 8172.227397 } } }, { "stage": "longmemeval.answers", "runId": "20261022-answer-recovery-v27-full-longmemeval-answers", "binarySha256": "7a534f0109a9603df097ce0fc9f42211ec347f20cff3f07999ff81acec6a5bb1", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "measurementMode": "measured", "startedAt": "2026-10-22T16:51:07.858213+00:00", "completedAt": "2026-10-22T16:58:28.805164+00:00", "sourceRunId": "20261022-answer-recovery-v27-full", "sourceBundleSha256": null, "sourceManifestSha256": "079e13f880f3025f463419c2845661a175c1e59cd477ae61611d311fcf7517d4", "conditions": { "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "memoryGib": 48.0 }, "execution": { "concurrencyNote": "Saved-evidence answer replay; process limit is shared across datasets when a global controller is recorded.", "answerProcesses": 8, "durationSeconds": 440.946951 } } }, { "stage": "personamem.personamem", "runId": "20261022-answer-recovery-v27-full-personamem", "binarySha256": "7a534f0109a9603df097ce0fc9f42211ec347f20cff3f07999ff81acec6a5bb1", "sourceRevision": "30825f8a1e5c5bf0e17fac4a03694b37e2ec66f3", "sourceTreeDirty": true, "sourcePatchSha256": "6811b10d85089a88d08d19a70c278cbe9b1f80a001b85b1e212df5e7b3490c07", "measurementMode": "measured", "startedAt": "2026-10-22T14:09:00.899696+00:00", "completedAt": "2026-10-22T15:13:31.756718+00:00", "sourceRunId": "20261022-answer-recovery-v27-full", "sourceBundleSha256": null, "sourceManifestSha256": "e3a985c1cef5a9ea053bae25cbcd53ab70d8ac417f566c5ffd51ab21cee500ff", "conditions": { "environment": { "label": "macOS-27.0.1-arm64-arm-64bit-Mach-O", "buildType": "Release", "concurrencyNote": "All retrieval, QA and PersonaMem stages are newly measured from the pinned original datasets. The initial wave overlaps LoCoMo 1x8, PersonaMem 1x8 and LongMemEval initially 4x2 (declared maximum 4x2). QA then runs per dataset under 8 processes, without overlap with the initial wave. Per-process model pacing is not a global rate limiter. Host is shared with desktop applications; sampled resource usage and controller intervals describe actual overlap. No controlled latency improvement is claimed.", "cpuCount": 12, "memoryGib": 48.0 }, "execution": { "concurrencyNote": "Original one-process live retrieval and multiple-choice generation.", "retrievalProcesses": 1, "workersPerProcess": 8, "answerProcesses": 1, "durationSeconds": 3870.857022 } } } ], "bundleSha256": "07c9946f115cb0c8f46814fa74540961e630f4988b2f404ab5f4295c26df7532", "href": "/research/ememox-20261022-answer-recovery-v27-full.md", "jsonHref": "/research/ememox-20261022-answer-recovery-v27-full.json" }, "datasets": [ { "id": "locomo", "dataset": "LoCoMo", "complete": true, "datasetSha256": "79fa87e90f04081343b8c8debecb80a9a6842b76a7aa537dc9fdf651ea698ff4", "selection": { "totalQuestions": 1986, "answerQuestions": 1540, "answerExcludedQuestions": 446 }, "retrieval": { "schemaVersion": "evidence-retrieval-v2", "topK": 10, "questions": 1986, "scoredQuestions": 1982, "questionsWithoutEvidence": 4, "hitRate": 0.9525731584258325, "turnRecall": 0.8547644349982289, "sessionRecall": null, "mrr": 0.7614544792017043, "latency": { "meanMs": 547.1151108733603, "samples": 1982, "p50Ms": 190.427333, "p95Ms": 1472.781375, "p99Ms": 6409.060709 }, "meanContextBytes": 26098.9404641776, "maximumContextBytes": 29400, "contextTokens": null, "embeddingModel": "kinfra-text-embedding-4b", "qualityPassed": true, "failures": { "projection_attempt_failures": 0, "failed_projections": 0 }, "slices": [ { "questionType": "1", "scoredQuestions": 282, "hitRate": 0.9468085106382979, "turnRecall": 0.7126696832579186, "mrr": 0.6365698525272993, "questions": 282, "questionsWithoutEvidence": 0 }, { "questionType": "2", "scoredQuestions": 321, "hitRate": 0.9439252336448598, "turnRecall": 0.9226666666666666, "mrr": 0.7232359194976017, "questions": 321, "questionsWithoutEvidence": 0 }, { "questionType": "3", "scoredQuestions": 92, "hitRate": 0.8152173913043478, "turnRecall": 0.6298076923076923, "mrr": 0.5120902346445825, "questions": 96, "questionsWithoutEvidence": 4 }, { "questionType": "4", "scoredQuestions": 841, "hitRate": 0.9690844233055886, "turnRecall": 0.9642857142857143, "mrr": 0.815621520110224, "questions": 841, "questionsWithoutEvidence": 0 }, { "questionType": "5", "scoredQuestions": 446, "hitRate": 0.9596412556053812, "turnRecall": 0.9608695652173913, "mrr": 0.8172227560680475, "questions": 446, "questionsWithoutEvidence": 0 } ], "queryDecomposition": { "enabled": true, "strategy": "rule_based" } }, "answers": { "schemaVersion": "answer-quality-v4", "mode": "saved-evidence-replay", "selectedQuestions": 1540, "judgedQuestions": 1540, "correctAnswers": 1049, "overallCorrectRate": 0.6811688311688312, "judgedCorrectRate": 0.6811688311688312, "answerModel": "deepseek/deepseek-flash", "judgeModel": "deepseek/deepseek-flash", "answerPromptVersion": "grounded-answer-v27", "judgePromptVersion": "semantic-answer-judge-v7", "generationLatency": { "meanMs": 2839.522494209091, "samples": 1540 }, "inputTokens": 27294469, "outputTokens": 361554, "tokenUsage": { "answer": { "inputTokens": { "value": 27294469, "mean": 17723.681168831168, "reportedQuestions": 1540, "totalQuestions": 1540 }, "cachedInputTokens": { "value": 4298996, "mean": 2791.555844155844, "reportedQuestions": 1540, "totalQuestions": 1540 }, "outputTokens": { "value": 361554, "mean": 234.77532467532467, "reportedQuestions": 1540, "totalQuestions": 1540 }, "reasoningTokens": { "value": 331533, "mean": 215.28116883116883, "reportedQuestions": 1540, "totalQuestions": 1540 }, "totalTokens": { "value": 27656023, "mean": 17958.456493506492, "reportedQuestions": 1540, "totalQuestions": 1540 } }, "judge": { "inputTokens": { "value": 7488244, "mean": 4862.496103896104, "reportedQuestions": 1540, "totalQuestions": 1540 }, "cachedInputTokens": { "value": 4150198, "mean": 2694.933766233766, "reportedQuestions": 1540, "totalQuestions": 1540 }, "outputTokens": { "value": 320467, "mean": 208.09545454545454, "reportedQuestions": 1540, "totalQuestions": 1540 }, "reasoningTokens": { "value": 221993, "mean": 144.1512987012987, "reportedQuestions": 1540, "totalQuestions": 1540 }, "totalTokens": { "value": 7808711, "mean": 5070.591558441559, "reportedQuestions": 1540, "totalQuestions": 1540 } } }, "qualityPassed": false, "failures": { "projection_failures": 0, "generation_failures": 0, "judge_failures": 0, "missing_evidence_questions": 0, "degraded_retrieval_queries": 0 }, "tokenF1": 0.5582581968181818, "hallucinationRate": 0.3142857142857143, "slices": [ { "questionType": "1", "selectedQuestions": 282, "judgedQuestions": 282, "correctAnswers": 102, "overallCorrectRate": 0.3617021276595745, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "2", "selectedQuestions": 321, "judgedQuestions": 321, "correctAnswers": 233, "overallCorrectRate": 0.7258566978193146, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "3", "selectedQuestions": 96, "judgedQuestions": 96, "correctAnswers": 35, "overallCorrectRate": 0.3645833333333333, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "4", "selectedQuestions": 841, "judgedQuestions": 841, "correctAnswers": 679, "overallCorrectRate": 0.807372175980975, "generationFailures": 0, "judgeFailures": 0 } ], "evidenceRunId": "20261022-answer-recovery-v27-full-locomo-retrieval" }, "execution": { "concurrency_note": "Retrieval 1 process(es) x 8 worker(s); answers 8 processes. Parallel timings are not comparable to archived serial timings.", "retrieval_processes": 1, "workers_per_process": 8, "answer_processes": 8, "duration_seconds": 3139.663271 } }, { "id": "longmemeval", "dataset": "LongMemEval-S", "complete": true, "datasetSha256": "d6f21ea9d60a0d56f34a05b609c79c88a451d2ae03597821ea3d5a9678c3a442", "selection": { "totalQuestions": 500, "answerQuestions": 500, "answerExcludedQuestions": 0 }, "retrieval": { "schemaVersion": "evidence-retrieval-v2", "topK": 10, "questions": 500, "scoredQuestions": 500, "questionsWithoutEvidence": 0, "hitRate": 0.964, "turnRecall": 0.8761160714285714, "sessionRecall": 0.9641350210970464, "mrr": 0.8897690476190475, "latency": { "meanMs": 1120.164258056, "samples": 500, "p50Ms": 1027.731167, "p95Ms": 1847.2345, "p99Ms": 2106.851 }, "meanContextBytes": 27980.574, "maximumContextBytes": 46471, "contextTokens": null, "embeddingModel": "kinfra-text-embedding-4b", "qualityPassed": true, "failures": { "projection_attempt_failures": 0, "failed_projections": 0 }, "slices": [ { "questionType": "knowledge-update", "scoredQuestions": 78, "hitRate": 1.0, "turnRecall": 0.9583333333333334, "mrr": 0.967948717948718, "questions": 78, "questionsWithoutEvidence": 0 }, { "questionType": "multi-session", "scoredQuestions": 133, "hitRate": 0.9849624060150376, "turnRecall": 0.8134556574923547, "mrr": 0.8765664160401002, "questions": 133, "questionsWithoutEvidence": 0 }, { "questionType": "single-session-assistant", "scoredQuestions": 56, "hitRate": 0.9642857142857143, "turnRecall": 0.9642857142857143, "mrr": 0.9553571428571429, "questions": 56, "questionsWithoutEvidence": 0 }, { "questionType": "single-session-preference", "scoredQuestions": 30, "hitRate": 0.9333333333333333, "turnRecall": 0.9090909090909091, "mrr": 0.6825396825396824, "questions": 30, "questionsWithoutEvidence": 0 }, { "questionType": "single-session-user", "scoredQuestions": 70, "hitRate": 0.9714285714285714, "turnRecall": 0.9545454545454546, "mrr": 0.9404761904761904, "questions": 70, "questionsWithoutEvidence": 0 }, { "questionType": "temporal-reasoning", "scoredQuestions": 133, "hitRate": 0.924812030075188, "turnRecall": 0.8648648648648649, "mrr": 0.8495614035087719, "questions": 133, "questionsWithoutEvidence": 0 } ], "queryDecomposition": { "enabled": true, "strategy": "memory_query_expansion" } }, "answers": { "schemaVersion": "answer-quality-v4", "mode": "saved-evidence-replay", "selectedQuestions": 500, "judgedQuestions": 500, "correctAnswers": 400, "overallCorrectRate": 0.8, "judgedCorrectRate": 0.8, "answerModel": "deepseek/deepseek-flash", "judgeModel": "deepseek/deepseek-flash", "answerPromptVersion": "grounded-answer-v27", "judgePromptVersion": "semantic-answer-judge-v7", "generationLatency": { "meanMs": 3942.187693048, "samples": 500 }, "inputTokens": 9628949, "outputTokens": 182578, "tokenUsage": { "answer": { "inputTokens": { "value": 9628949, "mean": 19257.898, "reportedQuestions": 500, "totalQuestions": 500 }, "cachedInputTokens": { "value": 1932541, "mean": 3865.082, "reportedQuestions": 500, "totalQuestions": 500 }, "outputTokens": { "value": 182578, "mean": 365.156, "reportedQuestions": 500, "totalQuestions": 500 }, "reasoningTokens": { "value": 170061, "mean": 340.122, "reportedQuestions": 500, "totalQuestions": 500 }, "totalTokens": { "value": 9811527, "mean": 19623.054, "reportedQuestions": 500, "totalQuestions": 500 } }, "judge": { "inputTokens": { "value": 1922247, "mean": 3844.494, "reportedQuestions": 500, "totalQuestions": 500 }, "cachedInputTokens": { "value": 1376974, "mean": 2753.948, "reportedQuestions": 500, "totalQuestions": 500 }, "outputTokens": { "value": 98377, "mean": 196.754, "reportedQuestions": 500, "totalQuestions": 500 }, "reasoningTokens": { "value": 69918, "mean": 139.836, "reportedQuestions": 500, "totalQuestions": 500 }, "totalTokens": { "value": 2020624, "mean": 4041.248, "reportedQuestions": 500, "totalQuestions": 500 } } }, "qualityPassed": false, "failures": { "projection_failures": 0, "generation_failures": 0, "judge_failures": 0, "missing_evidence_questions": 0, "degraded_retrieval_queries": 0 }, "tokenF1": 0.5542271738000001, "hallucinationRate": 0.226, "slices": [ { "questionType": "knowledge-update", "selectedQuestions": 78, "judgedQuestions": 78, "correctAnswers": 70, "overallCorrectRate": 0.8974358974358975, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "multi-session", "selectedQuestions": 133, "judgedQuestions": 133, "correctAnswers": 86, "overallCorrectRate": 0.6466165413533834, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "single-session-assistant", "selectedQuestions": 56, "judgedQuestions": 56, "correctAnswers": 53, "overallCorrectRate": 0.9464285714285714, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "single-session-preference", "selectedQuestions": 30, "judgedQuestions": 30, "correctAnswers": 21, "overallCorrectRate": 0.7, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "single-session-user", "selectedQuestions": 70, "judgedQuestions": 70, "correctAnswers": 62, "overallCorrectRate": 0.8857142857142857, "generationFailures": 0, "judgeFailures": 0 }, { "questionType": "temporal-reasoning", "selectedQuestions": 133, "judgedQuestions": 133, "correctAnswers": 108, "overallCorrectRate": 0.8120300751879699, "generationFailures": 0, "judgeFailures": 0 } ], "crossCuttingGroups": { "abstention": { "questionType": "abstention", "selectedQuestions": 30, "judgedQuestions": 30, "correctAnswers": 25, "overallCorrectRate": 0.8333333333333334, "generationFailures": 0, "judgeFailures": 0, "overlapping": true } }, "evidenceRunId": "20261022-answer-recovery-v27-full-longmemeval-retrieval" }, "execution": { "concurrency_note": "Retrieval 4 process(es) x 2 worker(s); answers 8 processes. Parallel timings are not comparable to archived serial timings.", "retrieval_processes": 4, "workers_per_process": 2, "answer_processes": 8, "duration_seconds": 8613.174347999999 } }, { "id": "personamem", "dataset": "PersonaMem", "complete": true, "datasetSha256": "cccd34cf53e0bc4d9536c04cff5ca045156d9a4e227e83327112482840bbc93c", "selection": { "totalQuestions": 589, "answerQuestions": 589, "answerExcludedQuestions": 0 }, "personamem": { "schemaVersion": "personamem-mcq-v1", "mode": "live-retrieval-and-generation", "questionsSha256": "cccd34cf53e0bc4d9536c04cff5ca045156d9a4e227e83327112482840bbc93c", "contextsSha256": "217247ebfec9e8442fc53570c795ab69f21aad08745f7de78d9beab51b122d4a", "selectedQuestions": 589, "correctAnswers": 435, "accuracy": 0.7385398981324278, "answerRequests": 589, "answerModel": "deepseek/deepseek-flash", "embeddingModel": "kinfra-text-embedding-4b", "meanRetrievalLatencyMs": 1493.199034, "meanGenerationLatencyMs": 3304.803267, "meanEndToEndLatencyMs": 4798.0023, "meanContextBytes": 8701.327674, "contextByteReductionRate": 0.938398, "contextTokens": null, "inputTokens": 1363972, "cachedInputTokens": 1224886, "outputTokens": 247615, "reasoningTokens": 246437, "totalTokens": 1611587, "tokenUsage": { "answer": { "inputTokens": { "value": 1363972, "mean": 2315.7419354838707, "reportedQuestions": 589, "totalQuestions": 589 }, "cachedInputTokens": { "value": 1224886, "mean": 2079.6027164685906, "reportedQuestions": 589, "totalQuestions": 589 }, "outputTokens": { "value": 247615, "mean": 420.39898132427845, "reportedQuestions": 589, "totalQuestions": 589 }, "reasoningTokens": { "value": 246437, "mean": 418.39898132427845, "reportedQuestions": 589, "totalQuestions": 589 }, "totalTokens": { "value": 1611587, "mean": 2736.1409168081495, "reportedQuestions": 589, "totalQuestions": 589 } } }, "qualityPassed": true, "failures": { "projection_attempt_failures": 0, "failed_projections": 0, "generation_failures": 0, "degraded_retrieval_queries": 0, "empty_retrieval_queries": 0 }, "queryDecomposition": { "enabled": true, "strategy": "rule_based" }, "slices": [ { "questionType": "generalizing_to_new_scenarios", "selectedQuestions": 57, "correctAnswers": 41, "accuracy": 0.7192982456140351 }, { "questionType": "provide_preference_aligned_recommendations", "selectedQuestions": 55, "correctAnswers": 41, "accuracy": 0.7454545454545455 }, { "questionType": "recall_user_shared_facts", "selectedQuestions": 129, "correctAnswers": 107, "accuracy": 0.8294573643410853 }, { "questionType": "recalling_facts_mentioned_by_the_user", "selectedQuestions": 17, "correctAnswers": 12, "accuracy": 0.7058823529411765 }, { "questionType": "recalling_the_reasons_behind_previous_updates", "selectedQuestions": 99, "correctAnswers": 83, "accuracy": 0.8383838383838383 }, { "questionType": "suggest_new_ideas", "selectedQuestions": 93, "correctAnswers": 54, "accuracy": 0.5806451612903226 }, { "questionType": "track_full_preference_evolution", "selectedQuestions": 139, "correctAnswers": 97, "accuracy": 0.697841726618705 } ] }, "execution": { "concurrency_note": "One process; live retrieval and multiple-choice answer generation.", "retrieval_processes": 1, "workers_per_process": 8, "answer_processes": 1, "duration_seconds": 3870.857022 } } ] } ```