Ground · Research benchmark

EviDial

Submitted · ICLR 2027

Updated

When correct depression predictions lack evidence.

A model can match a symptom label even when an interview does not provide enough evidence for that assessment. EviDial makes textual support a separate dimension of depression-dialogue evaluation.

01 · The question

A correct label can still leave a question unanswered.

Consider a sleep assessment. The interview never discussed sleep, but the benchmark label says there is no sleep disturbance. A model can receive credit for predicting that label without finding evidence for it in the conversation.

Illustrative example · sleep disturbance

  1. 01

    Dialogue

    Sleep was never discussed.

  2. 02

    Prediction

    “No sleep disturbance.”

  3. 03

    Label agreement

    The prediction matches the benchmark label.

  4. 04

    Textual support

    The dialogue does not justify that judgment.

EviDial asks both questions: did the prediction match the label, and did the dialogue support its direction? This helps identify cases that warrant abstention or further inquiry.

02 · A separate evidence dimension

What does the dialogue actually support?

Each participant–symptom cell receives an evidence state under a defined contract. Multiple judges contribute candidate judgments; reference construction and measurement audits expose uncertainty in those judgments.

  • 01

    Absent

    No eligible evidence remains after checking subject, time window, scope, and retractions.

  • 02

    Partial

    Relevant evidence exists, but cannot resolve whether the target symptom is present or absent.

  • 03

    Conflicting

    Eligible evidence contradicts itself, preventing a consistent judgment.

  • 04

    Sufficient

    Eligible evidence supports one clear direction: symptom presence or absence.

Four checks before evidence qualifies

Subject
Does the statement concern the participant?
Time window
Does it fall within the instrument’s target period?
Assertion scope
Does it cover the relevant symptom components?
Withdrawal
Was the statement subsequently retracted?

Missing evidence does not mean a symptom is absent. These states describe the available text, not a participant’s clinical status.

03 · Benchmark scope

Two languages. Three sources. Explicit units.

EviDial brings together Chinese and English dialogue records from PDCH, E-DAIC, and TalkDep. The main clinical reference uses one PDCH visit per participant and excludes simulated personas.

Dialogue records
382
Includes repeated visits and 12 simulated dialogues.
Clinical participants
343
Unique participants in the main reference pool.
Participant–symptom cells
2,744
Eight source-specific symptom dimensions per main-pool participant.
Source / instrumentRecordsMain participantsScope
PDCH · HAMD-179972Chinese interviews; 27 repeat visits excluded from the main pool.
E-DAIC · PHQ-8271271English interviews; audit uses the processed model-visible text.
TalkDep · BDI-II120Simulated personas; excluded from the main clinical reference.

The shared inventory has nine symptom dimensions; each main source contributes eight. PDCH excludes concentration and E-DAIC excludes suicidality. Source, language, instrument, and preprocessing differ together, so source differences do not establish a language or causal effect.

Label agreement and evidence support diverge.

The findings distinguish corpus evidence, an exploratory model audit, and human validation reported in the manuscript. Each percentage retains its own denominator and evidence scope.

04a · Candidate evidence reference

Most main-reference cells lack sufficient textual evidence.

The candidate reference assigns absent or partial evidence to 2,273 of the 2,744 main cells. Sufficient-evidence shares vary sharply across sources; this is a descriptive result about the processed dialogues and the working evidence reference.

Absent or partial evidence
82.8%
2,273 / 2,744 main-reference cells under the candidate reference.
Aggregate evidence landscape: 1,170 absent, 1,103 partial, 468 sufficient-positive, and 3 unresolved cells, with source-specific symptom evidence shares.
Figure

Candidate-reference evidence states across 2,744 main cells. Ribbon widths encode counts; source–symptom shares use all corresponding cells. NA marks dimensions excluded by design.

Open original figure

04b · Exploratory development audit

A label-correct prediction may still lack support.

The model audit covers 87 development cells from 76 participants, three model families, and four prompting strategies, with one run per condition. Among label-correct predictions, 69.2–96.9% lack sufficient textual support for their predicted direction.

Unsupported-Correct Rate (UCR) instead measures unsupported label-correct predictions per planned prediction. The two rates are reported separately because their denominators differ. Abstention and technical failure can reduce UCR without improving support for the predictions that remain.

Unsupported among correct
69.2–96.9%
Denominator: resolved label-correct predictions within each model–method condition.
Unsupported-Correct Rate
20.7–54.0%
Denominator: planned predictions, excluding unresolved references; none were unresolved in this audit.
Exploratory audit of three model families and four methods: supported correct, unsupported correct, incorrect, abstained, and failed predictions, with UCR intervals and unsupported-among-correct shares.
Figure

87 development cells per condition, 76 participants, and 12 model–method conditions. UCR intervals are conditional on the candidate reference. These results do not establish definitive model rankings.

Open original figure

04c · Manuscript-reported human validation

The reported human check preserves the main finding.

The manuscript reports a blinded 120-cell validation with two independent annotators and a third adjudicator. RE-LCA reaches 76.5% exact agreement with human consensus on comparable cells.

Rescoring the overlapping model audit against the human reference leaves an average 79.1% of label-correct predictions unsupported across 12 conditions. Each condition has its own correct-prediction denominator. These human-validation results are manuscript-reported and have not been independently reproduced here.

Exact agreement
76.5%
RE-LCA versus human consensus; reported by the manuscript.
Mean unsupported among correct
79.1%
Human-reference rescoring, averaged across 12 model–method conditions; manuscript-reported.
Manuscript-reported human validation: reference agreement, perturbation-state recovery, and unsupported predictions under automated and human references.
Figure

Human-validation results reported in the final manuscript. This subset check does not constitute a complete human audit across all cells and symptom dimensions.

Open original figure

05 · Interpretation boundaries

What these findings can substantiate.

EviDial measures whether an assessment is supported by its observable input. The current findings concern processed dialogues, a candidate evidence reference, and a small development-set audit.

Textual support
Evidence states characterize the dialogue, not diagnostic truth. A mismatch does not show that the native clinical label is wrong.
Exploratory scope
The 87-cell audit uses lightweight models and single runs. Broader held-out evaluation with stronger models and repeated runs remains future work.
Reference uncertainty
Automated judge coverage is uneven, reference choices remain sensitive, and sufficient evidence for symptom absence is harder to establish under the contract.
Source composition
E-DAIC contributes 79% of the main cells. Source, language, clinical instrument, and preprocessing are confounded; preservation of clinical meaning after reformatting has not been independently verified.

06 · Materials

Read the manuscript. Inspect the aggregate figures.

Research status: submitted to ICLR 2027. The materials document the study’s measurement contract, findings, and limitations.

Manuscript recompiled from final sources

PDF · 16 pages. This rendering follows the final source files; it is not confirmation of the exact file uploaded to the conference.

Explore all research