Guidelines

Guidelines

You will be presented with news articles to assess according to your professional judgment. For each article complete the following steps:

Step 1: Factual Takeaways

Note TWO spans in the article that you consider well-supported and unproblematic. To the extent possible, focus on takeaways that are related to the core themes of the article, rather than more tangential details. When marking spans, select a minimal, self-contained segment. For each span, briefly describe in your own words what is the main factual takeaway from it.

Step 2: Points of concern

Note ALL points in the article that prompt doubt, discomfort, or that require critical reflection to discern their true implications. These are texts where a surface-level reading leads to one takeaway, but a more critical reading would not cause one to not believe the original takeaway with the same level of certainty.

This may include issues such as misleading word choices, logical fallacies, misleading framing, missing but important context, unanswered questions that might have bearing on the interpretation of information in the article, situations where critical information is omitted etc.

For each point, you will annotate:

  1. The relevant span of text UNLESS it applies to the entire document.
  2. What takeaway a reader might reach when reading the content without applying critical thinking or upon a non-careful reading.
  3. What question(s) should a reader ask to better assess the takeaways from this span?
  4. Rate the severity of this concern from 1 (minor issue) to 5 (major issue) based on how much a critical reflection on the text alters your trust in its main takeaway.
    • Minor issue(1): I still believe the original takeaway. This issue is small and does not meaningfully change my trust in it.
    • Small issue(2): I still mostly believe the original takeaway, but I have some doubts because something is slightly misleading, missing, or unclear.
    • Moderate issue(3): I am uncertain whether to believe the original takeaway. There is not enough support for the takeaway in the article.
    • Serious issue(4): I mostly do not believe the original takeaway. There is weak support for the takeaway, or important information is missing.
    • Major issue(5): I do not believe the original takeaway at all. It is clear that something is misleading.

Important instructions:

  1. Read the entire article first before marking points of concern, as sometimes information that seems misleading or unclear at first is clarified later in the article.

  2. You may consider whether the speaker is relevant or qualified to make the claim. However, your decision should not be based on personal opinions about the speaker.

  3. Quotes are often included in news articles, such as "X said, 'the incident was completely unexpected.'" They consist of a speaker (X) and their statement ("the incident was completely unexpected."). For quotes, consider a full quote (speaker + statement) as the factual takeaway only when the fact that it was said is what matters, meaning it is important that this particular speaker made the statement. Otherwise, consider the underlying information itself as the factual takeaway, as long as it is presented as reliable.

    Span: "The university president said, 'A new scholarship program will begin on Monday.'"

    Factual takeaway: A new scholarship program will begin on Monday.

    Why? The university president is a credible and relevant authority on this matter. The statement is presented as reliable information about the start of the program, so the takeaway should focus on the program beginning.

    Span: "The famous chef said, 'I think traditional cookbooks are becoming obsolete.'"

    Factual takeaway: The famous chef said that traditional cookbooks are becoming obsolete.

    Why? The relevant fact is that the chef expressed this view, not whether cookbooks are actually becoming obsolete.

Examples:

Factual Takeaways

Example 2

Article link: Morning coffee may lower risk of heart disease-related death

Relevant span of text from the article

The study found that people who drank coffee in the morning had a lower risk of dying from cardiovascular disease and had a lower mortality risk than all-day coffee consumers

Takeaway

Drinking coffee in the morning may be better for heart health than drinking it at other times of the day.

Points of Concern

Example 1

Article link: Fast food goes GLP-1: Shake Shack joins protein craze

Relevant span of text from the article

Shake Shack’s spokesperson added that many of the items can fit into eating patterns often followed by people using GLP-1 medications such as Ozempic, Wegovy and Zepbound, with an emphasis on higher-protein, more balanced meals that do not compromise on flavor.

Potential takeaway

The new products in Shake Shack’s menu are appropriate for people using GLP-1 drugs.

Question to ask

Is this claim supported by experts, or is it just the company’s perspective?

Severity4I mostly do not believe the original takeaway.

Example 2

Article link: Deadly 'superbug' is spreading across US

Relevant span of text from the article

Approximately 7,000 cases were identified across dozens of U.S. states in 2025, according to the CDC, and it has reportedly been identified in at least 60 countries.

Potential takeaway

This fungus has infected many people.

Question to ask

Is this a lot? How does it compare to other infections?

Severity3I am uncertain whether to believe the original takeaway.

Example 3

Article link: Andrew Tate, brother arrested

Relevant span of text from the article

Tate asks a person off-screen to hand him two boxes of pizza... The boxes had Romanian type on the sides, potentially letting officials know he was in the country.

Potential takeaway

Tate was arrested because his exchange with Greta Thunberg revealed his location in Romania.

Question to ask

What evidence does the article provide that authorities identified his location through this video, rather than already knowing he was in Romania?

Severity3I am uncertain whether to believe the original takeaway.

Example 4

Article link: A.I. Agents Are Here. But Who's Accountable For Their Actions?

Relevant span of text from the article

Entire article

Potential takeaway

"Human Passport" is a necessary solution for ensuring accountability in AI agents.

Question to ask

Why is "Human Passport" mentioned so prominently throughout the article, and does the article provide balanced coverage of alternative approaches?

Severity3I am uncertain whether to believe the original takeaway.

Final tips before starting

  • The “potential takeaway” should describe what a reader might conclude from the text after a surface-level or non-careful reading, not your evaluation of the text itself.

    Bad example: “This is only the company’s perspective.”

    Better: “The new products in Shake Shack’s menu are appropriate for people using GLP-1 drugs.”

  • The “question to ask” should help a reader critically reflect on and potentially adjust their confidence in the potential takeaway. It should not describe how the article could be improved.

    Bad example: “This needs to be explained to readers.”

    Better: “Is this claim supported by experts, or only by the company itself?”