+ - 0:00:00
Notes for current slide
Notes for next slide

Reproducibility Workshop

COMM 783 | University of Pennsylvania

Dani Cosme

1 / 12

The reproducibility crisis

🔗 1,500 scientists lift the lid on reproducibility | Monya Baker

2 / 12

The reproducibility crisis

"More than 70% of researchers have tried and failed to reproduce another scientist's experiments, and more than half have failed to reproduce their own experiments."

🔗 1,500 scientists lift the lid on reproducibility | Monya Baker

3 / 12

The reproducibility crisis

  • Attempted to replicate 100 published studies based on the original materials using high powered designs

  • The mean replication effect size was half the magnitude (M = 0.20, SD = 0.26)

  • 36% of replication results were statistically significant (versus 97% of the original studies)

  • 39% of the replications were subjectively rated as replicating the original result

🔗 Estimating the reproducibility of psychological science | Open Science Collaboration

4 / 12

Reproducibility versus replicability

5 / 12

Reproducibility versus replicability

Reproducibility

  • Can a research finding be verified from the same data and code?
5 / 12

Reproducibility versus replicability

Reproducibility

  • Can a research finding be verified from the same data and code?

  • Same research question, same data, hopefully the same results

5 / 12

Reproducibility versus replicability

Reproducibility

  • Can a research finding be verified from the same data and code?

  • Same research question, same data, hopefully the same results

Replicability

  • Can a research finding be confirmed using the same methods, protocols and equipment?
5 / 12

Reproducibility versus replicability

Reproducibility

  • Can a research finding be verified from the same data and code?

  • Same research question, same data, hopefully the same results

Replicability

  • Can a research finding be confirmed using the same methods, protocols and equipment?

  • Same research question, new data, hopefully the same results

Content adapted from Reproducible Research Introduction from Data Carpentry

5 / 12

Why does it matter?

For society

For ourselves

6 / 12

Reproducible workflows

7 / 12

Reproducible workflows

  • The whole project: from design to publication and dissemination
7 / 12

Reproducible workflows

  • The whole project: from design to publication and dissemination

Content adapted from Reproducible Research Introduction from Data Carpentry

  • Facets of reproducibility:

    • Documentation: it's clear what happened and why; e.g. code is available and commented
7 / 12

Reproducible workflows

  • The whole project: from design to publication and dissemination

Content adapted from Reproducible Research Introduction from Data Carpentry

  • Facets of reproducibility:

    • Documentation: it's clear what happened and why; e.g. code is available and commented

    • Organization: it's clear where everything can be found; e.g. consistent file structures and naming conventions

7 / 12

Reproducible workflows

  • The whole project: from design to publication and dissemination

Content adapted from Reproducible Research Introduction from Data Carpentry

  • Facets of reproducibility:

    • Documentation: it's clear what happened and why; e.g. code is available and commented

    • Organization: it's clear where everything can be found; e.g. consistent file structures and naming conventions

    • Automation: it's simple to reproduce processes; e.g. an analysis script can be executed to produce the same results

7 / 12

Reproducible workflows

  • The whole project: from design to publication and dissemination

Content adapted from Reproducible Research Introduction from Data Carpentry

  • Facets of reproducibility:

    • Documentation: it's clear what happened and why; e.g. code is available and commented

    • Organization: it's clear where everything can be found; e.g. consistent file structures and naming conventions

    • Automation: it's simple to reproduce processes; e.g. an analysis script can be executed to produce the same results

    • Dissemination: it's possible for others to reproduce your work; e.g. materials, code, and data (if possible) are publicly shared

7 / 12

Reproducible workflows

Reflect on your current workflow

8 / 12

Reproducible workflows

Reflect on your current workflow

Less reproducible

More reproducible

8 / 12

Reproducible reporting with R Markdown

Project example

🔗 Project repository on GitHub | OSF project page

9 / 12

Reproducible report walk through

🔗 Workshop repository on GitHub

10 / 12

Extensions

11 / 12

Extensions

11 / 12

Extensions

11 / 12

Extensions

11 / 12

Extensions

11 / 12

Thank you!


Workshop materials are available at: github.com/dcosme/reproducibility-workshop

12 / 12

The reproducibility crisis

🔗 1,500 scientists lift the lid on reproducibility | Monya Baker

2 / 12
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow