⏱️ Overview

>> "The future is not set. There is no fate but what we make for ourselves" >> >> --Kyle Reese, The Terminator

Have you ever wished you could go back in time to get a do-over? How'd that work out for you? Unfortunately this lab probably won't give you the chance for a do-over, either. But the good news is that, using the powers of "make believe," you're going to have your shot at righting some scientific wrongs-- almost as good as fixing your own personal greatest mistakes, right? Let's just assume you agree and move on...

In this lab, you'll play the role of a time-traveling scientist. In the usual (i.e., non--time-traveling) way of doing science, you don't get to see what the "answer" is in advance. Instead, you try to do the best you can at designing and carrying out a good study, analyzing the data in reasonable ways, interpreting the results given what you know at the time, and so on. But the true test of a scientific finding is something that we don't typically have immediate access to: time. Specifically, future time. In other words, as the scientist writing up a scientific study, you never truly know whether your findings and interpretations will hold up. It's only after further studies are carried out, your ideas are poked and prodded by other scientists, etc., that we start to get a sense of whether a finding is "real." But what if it didn't have to be that way?

As a time-traveling scientist, this week you'll get the chance to skip ahead to years after your original study is carried out. You'll read papers written after your study that followed up on the original, explored similar questions, or used similar approaches. Then, unlike real-world you, the time-traveling scientist version of yourself will get to re-frame your original study, but with the benefit of foresight.

⏱️ Learning objectives

The major goal of this lab is to explore the "discussion" section of scientific papers. The primary goal of the discussion section is to situate the current study's findings within the context of the broader literature. Discussion sections also often suggest potential areas for future exploration or follow-up work. In this lab, you'll carry out a literature search, read several scientific articles organized around a chosen theme, and draft a simulated discussion section. In particular, you will:

Learn about and practice writing a discussion section
Practice carrying out an effective literature search
Practice reading, summarizing, discussing, and interpreting scientific articles
Practice synthesizing key findings across multiple papers and studies
Practice communicating clearly and directly

⏱️ Procedure

Each group will begin the lab exercise by finding, reading, and summarizing a 10--30 year old scientific paper. This paper will serve as a sort of "template," in that you'll pretend (for the purposes of this exercise) that your group was in the process of writing up the study described in that paper. In other words, even though that template study was actually carried out 10--30 years ago, you'll pretend it was just completed today. Like the magical sci-fi device featured in your favorite time travel story, your imagination can make you a mental time traveler, sending your consciousness back through the turbulent oceans of time-- but without all of that pesky temporal paradox stuff, like accidentally destroying the universe by meeting yourself in the past.

Next, you'll find several more modern papers (written within roughly the last 10 years) that cite the template paper. These will help to illustrate how the template paper has "aged" with respect to the broader literature. You'll also find an additional more modern paper (again, written within roughly the last 10 years) that does not cite the template paper, but focuses on a similar question, paradigm, or theme. This last paper will help to provide additional context regarding "alternative" approaches and interpretations for the template study.

After having found, read, and interpreted these paper's you'll draft a new discussion section for the template paper, as though you were writing up the paper for publication. However, whereas the template paper could only cite work published prior to when it was written, your revised version will have the benefit of knowing whether the original paper stood the test of time after its publication. This will provide an opportunity to potentially re-interpret the template study's key findings, discuss new pitfalls and limitations not raised in the original paper, and re-think future directions, all with the benefit of knowing the "future" relative to the original paper (i.e., hindsight relative to today, but foresight relative to the template paper's original time of writing-- isn't time travel fun?).

🧳 Part 1: Find an (old) paper to use as a "template"

The first thing you'll need (as a group) is a paper. To make this lab exercise work, you should find something with the following characteristics:

The paper should be published sometime in the 1990--2010 range
The paper should be published in a peer-reviewed journal
The paper should have at least 25 citations
The paper should not be retracted (i.e., if you go to the publisher's website and click on the paper, it should not have a retraction notice)
The paper should be about an interesting topic that you won't get bored reading about, writing about, and/or discussing
The paper should be about something psychology-related and/or neuroscience-related
The paper should be something you haven't read before (i.e., nobody in your group should have read the paper)
The paper should be less than 20 pages long (much shorter is fine)
It should be empirical (i.e., describing an experiment and reporting the results-- as opposed to a review paper, opinion paper, theory paper, etc.)
You should be able to understand the paper (e.g., don't pick something inscrutable or super complicated)

The easiest way to search for papers is using Google Scholar. If you use the search box while on Dartmouth's campus, the PDFs for the articles that come up should be free for you to read (paid for by the generosity of the Dartmouth library system!). If you're off campus, you'll need to use a Virtual Private Network to replicate that functionality. Google Scholar also lists citation and reference information in a convenient way within its search results, making it ideal for this lab exercise.

Semantic Scholar is another useful tool for carrying out literature searches. It is similar to Google Scholar in many respects, although the way keywords are applied to find articles works a bit differently. You'll likely find that you prefer either Google Scholar or Semantic Scholar, depending on which better matches the way you search and think.

A third interesting way to find papers is through the Elicit AI Research Assistant tool. This is a somewhat new option (and you'll need to sign up for a free account in order to use it). The idea is that it's based around "questions" rather than keywords. It's worth trying out!

Ultimately, each group should find a single template paper. Share a PDF on Slack in the #literature-review-lab channel. Think of this paper's time of writing as the moment your group is time-traveling back to.

🧳 Part 2: Find papers that cite your template paper

Your next job is to find 5 papers that cite your (original) template paper. If you find your template paper on Google Scholar, you can easily find other papers that cite it by clicking the "Cited by" link. (Feel free to use other approaches to finding citing papers too.) Try to find a set of 5 papers with the following properties:

All of the papers should explore similar or related questions to the original
All of the papers should be written in 2011 or later
All of the papers should be published in peer-reviewed journals
None of the papers should be retracted
None of the papers should have overlapping co-authors with the original
At least one paper should be cited more than the original
Try to find at least 1 paper that attempted to replicate the original study's experiment
Try to find at least 1 paper that carried out a follow-up study to the original
Try to find at least 1 "review" paper (i.e., that synthesizes across several related studies)

Share PDFs of your group's set of 5 papers on Slack in the #literature-review-lab channel.

The seventh paper in your collection should be about a similar topic as your template paper, but should not cite the original template paper. (It's OK if this seventh paper cites one or more of the 5 citing papers you found in Part 2 of the lab.) You can use any of the above literature search tools (Google Scholar, Semantic Scholar, Elicit) or others to track down this last paper. Your paper should have the following properties:

It should haven been written in 2011 or later
None of the co-authors should overlap with the original paper
The paper should either attempt to answer a similar question to the original paper, or should

use a similar experimental paradigm

The paper should be published in a peer-reviewed journal
The paper should not be retracted

Share a PDF of this last paper in the #literature-review-lab channel on Slack.

🧳 Part 4: Summarize everything

However your group deems appropriate, create a brief summary of each of the seven papers you found (original, 5 citing papers, 1 non-citing paper). Everyone should read the original paper, but it's not critical for everyone to read every other paper in detail. Share your summaries with your other group members so that everyone can have some insights into what each paper is about. (The summaries don't need to be shared outside of your group; they're just for your own internal use.) Some suggestions:

Describe what the main question of each paper was
Describe how the main question was explored
Describe what the "answer" to the paper's main question was
Describe or discuss any relevant overlap with the template paper-- e.g., similarities and differences

in the overall approach, findings, interpretation, etc.

🧳 Part 5: Discuss and brainstorm

As a group, think about the set of papers you've added to your collection. Using the post-2011 papers as a guide, consider the questions like the following:

Were there follow-up studies to the original?
Were there unanswered questions in the original study that have now been explored further?
Have the original conclusions or interpretations held up? Have they been criticized or interpreted differently in later studies?
Were the approaches taken across the different papers similar? If not, what do you see as

the main strengths and weaknesses of each approach?

Did all of the papers reach similar conclusions? Or if different conclusions were reached, are the conclusions logically compatible? Which do you "trust" most, and why?
What do you think it all means? For example, what do you think the true answer to the question posted in the original "template" paper is?
Also come up with some still-unanswered questions and discuss how they could be studied in future (e.g., post-today) work

⏱️ Writing your lab report

Your lab report (recommended length: roughly 2--5 pages) should take the form of a formal discussion section in a scientific paper. You should write as though you had carried out the original study (the one in described in your template paper) today, in 2024. Use the papers you found in parts 2 and 3 to situate "your" study within the broader literature. Describe strengths and weaknesses of your approach relative to other potential approaches. Propose an interpretation of your findings (this can match the original paper's interpretations, but it doesn't need to match the original). Also discuss potential alternative interpretations, and their relative merits and shortcomings. Your discussion section should take the following form:

First, summarize the question, approach, and key finding of the original study
Next discuss other related work, and describe how it compares or contrasts with the original study's approach, findings, interpretations, etc.
Draw your conclusions about what you think it all means (e.g., what do you think the "true" answer to the key question is?)
Describe some still unanswered questions and propose how they might be studied in future work.
Make sure to include a bibliography!
Include a brief paragraph reflecting on how you used generative AI during this lab. Did AI help you find papers or insights you would have missed? Were there cases where AI-generated summaries were inaccurate or misleading?

⏱️ Using GenAI in this lab

Generative AI has powerful research capabilities-- but how much can you actually trust it?

🧳 GenAI Challenge: Deep Research Accuracy Audit

After your group has completed the manual literature search (Parts 1--3), try this:

Use "deep research": Use the deep research function of your preferred GenAI model (suggested: ChatGPT's deep research mode or Claude) to research your template paper's topic. Ask it to find the original paper, summarize its findings, identify papers that cite it, and describe how the field has evolved.
Spot-check for accuracy: Carefully compare the AI's output against the papers you actually read. For each claim the AI makes:
- Did you find the paper it references? Does the paper actually exist?
- Is the AI's summary of the paper accurate? Did it get the key findings right?
- Did the AI miss any important nuances or caveats?
- Did it invent any "hallucinated" papers, authors, or findings?
Evaluate reliability at different stakes: Reflect on how much you would trust this AI-generated research summary for different purposes:
- A personal hobby project (e.g., learning about a topic for fun)?
- An assignment for a course (where accuracy matters for your grade)?
- A financial decision (e.g., researching a major purchase)?
- A life-or-death decision (e.g., evaluating medical treatments)?
Reflect on what you learned:
- How many errors or hallucinations did you find?
- Were the errors obvious or subtle?
- Would you have caught them if you hadn't already read the papers yourself?
- What does this tell you about the role of AI in scientific literature review?

This exercise is designed to help you develop a calibrated sense of when and how much to trust AI-generated research summaries-- a skill that will only become more important as these tools improve.

⏱️ Closing discussion points

This lab marks the end of our explorations of formal scientific papers:

The Introduction section motivates the paper, captures the reader's interest, defines the key question(s), and summarizes the overarching approach and key findings
The Methods section describes the approach in detail, reminding the reader of the study's motivation and/or key findings as needed
The Results section describes what the study found, reminding the reader of the study's motivations and methods as needed
The Discussion section re-summarizes the approach and key findings, with a focus on situating the study within the context of the broader literature

As we step back, we can see that a recurring theme across all of these sections is that writing them effectively essentially requires re-stating the same ideas, but with a different focus each time. In each section you need to summarize "what you did" and "what you found." The trick is to use the repetition to your advantage to improve clarity and define emphasis.

The discussion section is ultimately what shapes the impression people will have in their minds when they finish reading your paper. Your study can have fantastic motivation, clear methods, and strong results. Those are all necessary elements of a strong "core" for your paper. But the discussion section is when you make your case for why it matters. In other words, by the time you get to the point of writing up a study, you've probably invested a lot of time and resources. What did you learn? Why should people care? How does this study fit in with the rest of human knowledge? Consider your reader's perspective, too. What do they care about? They've invested time in reading your paper; now show them it was worth that effort!

As with most things in science, there is no one "right" way to write a discussion section. To help guide you, think about what is effective at convincing you. If you put yourself in your reader's place, what's the most important takeaway message? What's the most effective framing? What do your readers need to know in order to care about what you did, and how can you maximize that impact? What others take away from your paper will determine how your science stands up to the test of time.