Event-source validation study

You are helping evaluate the reliability of source-backed event records used to assess forecasting systems. The research is conducted by the research team.

What you will do

You will review event claims. For each item, answer: Does the article support the full event claim shown above? You are not being asked to forecast an outcome.

Study procedure

Read the complete event claim and article, then use the annotation rubric to judge entity match, event-date validity, and source support. The full rubric and examples appear above the response controls on every item. Provide one supporting excerpt and explain every rejected or qualified judgment. Do not use outside sources.

Information collected and research use

  • Your Prolific ID, annotation choices, excerpts, explanations, response timing, and quality-check results are recorded.
  • Your Prolific ID is used to administer and resume the study. Research analysis and reporting will use pseudonymized records and aggregate results.
  • Progress is autosaved. Unsubmitted records are retained for resumption but excluded from completed-study analysis.

Voluntary participation and withdrawal

Participation is voluntary. You may stop before final submission by returning the study on Prolific. You may contact the research team to request deletion of an unfinished record. Payment and approval follow the duration, reward, and completion terms shown on Prolific.

Confidentiality and contact

This interface does not ask for your name or personal contact details. Questions about the study or a withdrawal request should be sent through Prolific messaging, or to [email protected]. Do not include your annotation answers in email.

Content notice

Articles may discuss politics, conflict, disasters, illness, death, or other sensitive real-world events. You may stop and return the study if you do not wish to continue.

Important instructions

Let’s try an item together

Read a claim, find what the article supports, and record your judgment.

  1. 1. Worked example
  2. 2. Common distinctions
  3. 3. Your turn

Follow one example

Why make three separate judgments?

Entity match checks that the article concerns the same people, organisations, places, or products. Event-date validity checks when the event occurred, which may differ from the article's publication date. Source support checks the substantive event and its material details, excluding occurrence-date correctness. Judge them separately: an article can describe the right event while giving a different date or supporting only part of the claim.

Event claim to validate

The city council approved the clean-air policy on 12 May.

Claimed event date
12 May
Domain
Politics
Does the article support the full event claim above?
Article

Council approves clean-air policy

Local News · Published 13 May

On 12 May, the city council voted to approve the clean-air policy after a three-hour meeting.

The policy will take effect in July.

Your judgment
1. Entity match
CorrectAmbiguousIncorrect
2. Event-date validity
CorrectNear matchIncorrectUnclear
What establishes the date?
Article states dateArticle contextPublication date onlyNo date evidence
3. Source support
FullPartialNoneContradictory
Article evidence
“On 12 May, the city council voted to approve the clean-air policy...”
Judge the event and its date separately
If the article instead said the vote occurred on 14 May, select Correct entity, Incorrect date, and Full support. Enter 14 May as the corrected date and briefly explain the mismatch.
Reference: full annotation rubric
1. Entity match
  • Correct: the article refers to the same people, organisations, places, and products as the claim.
  • Ambiguous: the article does not provide enough information to determine whether they are the same.
  • Incorrect: the article identifies a different entity. An unnamed or unidentifiable subject is Ambiguous, not Incorrect.
2. Event-date validity
  • Correct: the claimed occurrence date is supported.
  • Near match: the only difference is one or two days and can reasonably result from timezone or reporting differences.
  • Incorrect: the article establishes a different occurrence date. Enter that date. An absent date is Unclear, not Incorrect.
  • Unclear: the article cannot establish when the event occurred.
What establishes the date?
  • Article states date: the article explicitly states the event's occurrence date.
  • Article context: relative wording or surrounding context establishes the occurrence date, including using the publication date to interpret "today", "yesterday", or "on Monday" when the context is clear.
  • Publication date only: only publication metadata is available, without article wording connecting it to when the event occurred; choose Unclear.
  • No date evidence: the article provides no usable date information.
3. Source support
  • Full: the substantive event and its material details are supported; assess its occurrence date separately.
  • Partial: some substantive part of the event is supported, but other material details are missing. Shared names or topic alone are not enough.
  • None: the claimed action or event is not supported, even if the article discusses the same people or topic.
  • Contradictory: the article gives information incompatible with the event claim. Missing details or additional compatible facts are not contradictions.
Evidence and explanations
  • Provide one exact article excerpt for Full, Partial, or Contradictory support.
  • Briefly explain every qualified or rejected judgment, including Ambiguous, Near match, Incorrect, Unclear, Partial, None, or Contradictory.
  • Do not use outside sources.
Step 1 of 3

Rubric version: 2026-09-06. The full rubric is also available beside each annotation item.

Event claim to validate

Claimed event date
Domain
Does the article support the full event claim above?
Likely entities from the claim

Automatically extracted as a reading aid; judge every named entity in the full claim.

Article

Highlighted date text is a reading aid only; verify what it refers to in context.

Quick attention check

Study complete

Your responses were submitted successfully.

You may now return to Prolific.

Please return this study

The tutorial checks were not completed successfully after two attempts. Return the submission through Prolific; your annotation responses have not been submitted.