class: center, middle, inverse, title-slide # Specification Curve Analysis ## COMM 783 | University of Pennsylvania ### <a href="https://dcosme.github.io/">Dani Cosme</a> ### 2022-04-07 --- ## The reproducibility crisis .pull-left[ * Attempted to replicate 100 published studies based on the original materials using high powered designs * The mean replication effect size was **half the magnitude** (*M* = 0.20, *SD* = 0.26) * **36%** of replication results were statistically significant (versus 97% of the original studies) * **39%** of the replications were subjectively rated as replicating the original result ] .pull-right[ <img src="img/replication_OSC.jpeg" width="100%" /> ] .footnote[🔗 [Estimating the reproducibility of psychological science | Open Science Collaboration](https://www.science.org/doi/10.1126/science.aac4716)] --- ## Undisclosed analytic flexibility .center[ <img src="https://www.si.umich.edu/sites/default/files/styles/hero_image_medium/public/image1.jpg?itok=yIJOsKAv" width="60%" /> .caption[ Graphic from: [Ditch the stale PDF](https://www.si.umich.edu/about-umsi/news/ditch-stale-pdf-making-research-papers-interactive-and-more-transparent) ] ] <br> .pull-left[ #### Problems * The "garden of forking paths" * "Researcher degrees of freedom" ] -- .pull-right[ #### Solutions * Preregistration * Specification curve analysis ] .footnote[🔗 [Gelman & Loken, 2013](http://stat.columbia.edu/~gelman/research/unpublished/forking.pdf)] --- ## Specification curve analysis (SCA) -- According to [Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z), the solution is to specify all "reasonable" models to test an association and assess the joint distribution across model specifications. -- <br> .small[ > Some researchers object to blindly running alternative specifications that may make little sense for theoretical or statistical reasons just for the sake of ‘robustness’... We believe one should test specifications that vary in as many of the potentially ad hoc assumptions as possible without testing any specifications that are not theoretically grounded. ] -- <br> This can be thought of as an explicit framework for sensitivity analyses / robustness checks, that enables inferential statistics across model specifications. --- ## The value Increased transparency. -- A better understanding of how conceptual and analytic decisions alter the association of interest. -- A more robust scientific literature with **increased replicability, generalizability, and translational value**. -- .center[ <img src="https://media.giphy.com/media/WUq1cg9K7uzHa/giphy.gif" width="50%" /> ] --- ## SCA overview -- ### 1. Specify all reasonable models -- ### 2. Plot specification curve showing association/effect estimates as a function of decisions -- ### 3. Test how inconsistent the curve results are given the null hypothesis of no association/effect --- ## 1. Identify reasonable models For the relationship of interest, determine the set of reasonable model specifications to test. -- #### Reasonable specifications should be: + Consistent with theory + Expected to be statistically valid + Non-redundant .center[ <img src="img/table1_2020.png" width="70%" /> .caption[ Table from [Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z) ] ] --- ## 1. Identify reasonable models **Type E decisions** * Alternative specifications are expected to be **equivalent** and thus can be treated as arbitrary * Include in the SCA **Type N decisions** * Alternative specifications **non-equivalent** (e.g. due to lower reliability or validity) and therefore some specifications can be treated as objectively better justified * Don't include clearly worse specifications in the SCA **Type U decisions** * It is **uncertain** whether alternative specifications are non-equivalent * Treat SCA as exploratory * Potentially break into separate curves when estimating curve stats .footnote[🔗 [Del Giudice & Gangestad, 2021](https://journals.sagepub.com/doi/full/10.1177/2515245920954925)] --- ## 2. Descriptive specification curve The specification curve visualizes the strength of the association/effect between two constructs of interest across model specifications and the analytic decisions associated with each model specification. -- .pull-left[ **Key features** * Two panels depicting 1) the curve and 2) the decisions * A vertical slice = information about a single model specification ] .pull-right[ <img src="https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41562-020-0912-z/MediaObjects/41562_2020_912_Fig2_HTML.png" width="100%" /> .caption[ Figure from [Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z) ] ] --- ## 2. Descriptive specification curve -- #### The curve * Model specifications are ranked -- * Shows the model magnitude, sign (positive or negative), and statistical significance -- * Often visualizes uncertainty around individual point estimates in the model specifications -- * May highlight a single a priori or previously reported association/effect estimates -- #### The decisions -- * Each row denotes a specific decision and whether or not that decision applied to a given model specification -- * Decisions are often grouped into categories to ease interpretation --- ## SCA examples .panelset[ .panel[.panel-name[Simonsohn et al., 2020] <img src="https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41562-020-0912-z/MediaObjects/41562_2020_912_Fig2_HTML.png" width="70%" style="display: block; margin: auto;" /> .caption[Figure from [Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z) ] ] .panel[.panel-name[Orben & Przybylski, 2019] <img src="https://media.springernature.com/full/springer-static/image/art%3A10.1038%2Fs41562-018-0506-1/MediaObjects/41562_2018_506_Fig3_HTML.png" width="52%" style="display: block; margin: auto;" /> .caption[Example specification curve analysis (SCA) from [Orben & Przybylski, 2019](http://nature.com/articles/s41562-018-0506-1) ] ] .panel[.panel-name[Flournoy et al., 2020] <img src="https://ars.els-cdn.com/content/image/1-s2.0-S1878929320300554-gr1.jpg" width="52%" style="display: block; margin: auto;" /> .caption[Example specification curve analysis (SCA) from [Flournoy et al., 2020](https://www.sciencedirect.com/science/article/pii/S1878929320300554) ] ] .panel[.panel-name[Cosme & Lopez, 2020] <img src="https://oup.silverchair-cdn.com/oup/backfile/Content_public/Journal/scan/PAP/10.1093_scan_nsaa155/2/nsaa155f1.jpeg?Expires=1652231337&Signature=bF-qwPQM2uk9U4ax9HcovP1UD5hfZarY7pDGZDfmV5cDwPYq~YWLDiHvpfoQtN0FwieSOYIMk1n5mLuS57oG36-atTuf9653INkwoEjrNge86E3dEqdzor~D3yhurWGE09vsuqLlm8M3TDr8IrFuujWzjFIGTWoVhxRW3zwt3cM1yzizhBTHZDmYCGInJXbQPQUFdrSYakpuRjTj8kj8RkkMWlnW4w3zRJrr0854Zvm4TobGS-IYDBTIRy9uBo35A9vt3h6-t2qWCC7tiTUi3xvIPxDiUIvOFFuLMO09~OdLSyFaMSpSxfwaD8fdKpaNfTlUPelvjsgdWkficjbIyQ__&Key-Pair-Id=APKAIE5G5CRDK6RD3PGA" width="57%" style="display: block; margin: auto;" /> .caption[Example specification curve analysis (SCA) from [Cosme & Lopez](https://psyarxiv.com/23mu5) ] ] .panel[.panel-name[Cosme et al., preprint] <br> <img src="img/cosme_curve.png" width="35%" style="display: block; margin: auto;" /> .caption[Example specification curve analysis (SCA) from [Cosme et al., preprint](https://psyarxiv.com/9cxfj) ] ] ] --- ## 3. Inferential statistics <br> #### Metrics of interest in the observed curve * Median curve estimate -- * The share of positive or negative associations that are statistically significant -- <br> .center[ ###But are the observed effects surprising given the null hypothesis? ] --- ## 3. Inferential statistics #### Test inconsistency with the null -- + Use bootstrap resampling to create a distribution of curves under the null hypothesis -- + Experimental designs = shuffle the randomly assigned variable(s) -- + Observational designs = "force" the null by removing the effect of x on y for each observation and sample from this null dataset -- + Estimate the curve and extract the curve median, and share of positive/negative significant associations -- + Repeat many times to get a distribution -- + Compare observed curve metrics to null curves to generate p-values --- ## 3. Inferential statistics #### Potential questions to test versus null -- + Is the median effect size in the observed curve statistically different than in the null distribution? -- + Is the share of dominant signs (e.g., positive or negative effects) that are statistically significant different than the null? <img src="img/table2_2020.png" width="75%" style="display: block; margin: auto;" /> .caption[Table from [Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z) ] --- class: center, middle ## Tutorial --- ## 1. Define reasonable specifications <br><br><br> .center[ ## What is the relationship between mental health and satisfaction with life? ] --- ## 1. Define reasonable specifications -- Ways of operationalizing of the IV "mental health" * `CEDS10` = depression score on the CESD-10 * `GAD7` = anxiety score on the GAD-7 * `PANAS_negative_Affect` = negative affect score on the PANAS * `PSS` = perceived stress score on the PSS -- Control variables * `age` = age * `gender` = gender * `mother_edu` = maternal education -- Analytic decisions * Statistical modeling approach * Linear regression * Outliers * Use all data points * Winsorize to the mean +/- 3 SD --- ## 1. Define reasonable specifications ### Visualize decisions <!-- --> --- ## Prep data These data are generated based on an existing dataset from a study looking at health and well-being * Create winsorized independent variables (+/- 3 SD from the mean) * Mean center and standardize each variable .scroll-output[ .extrasmall[ ```r # load data df = read.csv("sca_tutorial_inferences_data.csv", stringsAsFactors = FALSE) # tidy for modeling model_df = df %>% gather(variable, value, -PID, -age, -gender, -mother_edu, -SWLS) %>% group_by(variable) %>% mutate(mean_value = mean(value, na.rm = TRUE), sd3 = 3*sd(value, na.rm = TRUE)) %>% ungroup() %>% mutate(value_winsorized = ifelse(value > mean_value + sd3, mean_value + sd3, ifelse(value < mean_value - sd3, mean_value - sd3, value)), variable_winsorized = sprintf("%s_winsorized", variable)) %>% select(-mean_value, -sd3) %>% spread(variable_winsorized, value_winsorized) %>% group_by(PID) %>% fill(contains("winsorized"), .direction = "downup") %>% spread(variable, value) %>% gather(variable, value, -PID, -age, -gender, -mother_edu) %>% group_by(variable) %>% mutate(value = scale(value, center = TRUE, scale = TRUE)) %>% spread(variable, value) %>% select(PID, age, gender, mother_edu, SWLS, sort(tidyselect::peek_vars())) ``` ] ] --- ## Prep data These data are generated based on an existing dataset from a study looking at health and well-being * Create winsorized independent variables (+/- 3 SD from the mean) * Mean center and standardize each variable <div id="htmlwidget-d79cf9bbcfa178184148" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-d79cf9bbcfa178184148">{"x":{"filter":"none","vertical":false,"extensions":["FixedColumns"],"data":[["p001","p002","p003","p004","p005","p006","p007","p008","p009","p010","p011","p012","p013","p014","p015","p016","p017","p018","p019","p020","p021","p022","p023","p024","p025","p026","p027","p028","p029","p030","p031","p032","p033","p034","p035","p036","p037","p038","p039","p040","p041","p042","p043","p044","p045","p046","p047","p048","p049","p050","p051","p052","p053","p054","p055","p056","p057","p058","p059","p060","p061","p062","p063","p064","p065","p066","p067","p068","p069","p070","p071","p072","p073","p074","p075","p076","p077","p078","p079","p080","p081","p082","p083","p084","p085","p086","p087","p088","p089","p090","p091","p092","p093","p094","p095","p096","p097","p098","p099","p100","p101","p102","p103","p104","p105","p106","p107","p108","p109","p110","p111","p112","p113","p114","p115","p116","p117","p118","p119","p120","p121","p122","p123","p124","p125","p126","p127","p128","p129","p130","p131","p132","p133","p134","p135","p136","p137","p138","p139","p140","p141","p142","p143","p144","p145","p146","p147","p148","p149","p150","p151","p152","p153","p154","p155","p156","p157","p158","p159","p160","p161","p162","p163","p164","p165","p166","p167","p168","p169","p170","p171","p172","p173","p174","p175","p176","p177","p178","p179","p180","p181","p182","p183","p184","p185","p186","p187","p188","p189","p190","p191","p192","p193","p194","p195","p196","p197","p198","p199","p200","p201","p202","p203","p204","p205","p206","p207","p208","p209","p210","p211","p212","p213","p214","p215","p216","p217","p218","p219","p220","p221","p222","p223","p224","p225","p226","p227","p228","p229","p230","p231","p232","p233","p234","p235","p236","p237","p238","p239","p240","p241","p242","p243","p244","p245","p246","p247","p248","p249","p250","p251","p252","p253","p254","p255","p256","p257","p258","p259","p260","p261","p262","p263","p264","p265","p266","p267","p268","p269","p270","p271","p272","p273","p274","p275","p276","p277","p278","p279","p280","p281","p282","p283","p284","p285","p286","p287","p288","p289","p290","p291","p292","p293","p294","p295","p296","p297","p298","p299","p300"],[19.49,23.07,21.22,21.19,19.59,17.66,16.97,19.53,24.13,20.45,20.26,20.62,19.36,17.5,25.34,19.07,19.41,16.35,19.53,19.46,19.36,16.09,23.67,24.02,21.34,17.77,21.63,19.7,19.34,22.79,17.46,19.11,18.5,20.08,17.1,18.54,21.49,21.03,18.02,21.31,15.06,19.31,21.53,18.74,20.31,17.01,21.6,16.21,19.39,19.41,20.03,18.33,23.61,17.19,20.4,13.85,20.17,21.23,17.91,21.64,20.74,22.03,18.4,18.15,16.56,19.33,17.43,21.35,20.31,24.33,17.62,22.46,18.89,17.26,21.33,17.03,17.94,18.54,20.49,17.96,22.36,17.51,20.3,20.34,17.74,19.19,18.76,21.97,23.26,17.94,21.85,19.75,15.81,18.36,20.6,20.38,15.15,21.27,21.59,19.39,18.58,23.86,20.08,20.14,17.76,17.18,16.97,19.23,22.76,23.5,23.42,22.6,19.36,17.67,19.46,15.17,19.39,23.44,20.49,22.13,21.04,18.35,23.66,20.14,18.01,20.05,17.04,21.95,21.32,19.48,19.26,22.04,18.01,18.24,20.87,19.29,22.02,19.66,22.62,21.03,15.84,17.11,22.91,21.43,18.09,17.52,22.16,18.48,23.9,24.11,17.08,18.33,17.6,21.57,17.76,16.04,24.13,22.4,21.91,18.55,20.13,21.27,17.33,19.75,20.37,23.33,21.52,22.89,20.29,17.73,19.18,16.82,21.86,22.89,19.78,18.16,22.11,20.17,19.04,22.28,18.7,22.51,18.41,21.1,21.63,17.18,20.76,20.05,21.55,18.53,20.05,22.7,20.54,21.64,19.45,18.15,19.91,17.66,14.71,22.5,17.72,16.18,22.02,20.16,19.02,19.39,19.22,22.18,21.57,16.59,20.95,21.57,21.59,15.21,19.55,23.95,21.74,16.62,16.54,20.91,19.86,22.01,20.7,20.13,20.46,20.36,17.61,24.96,17.15,20,19.19,23.89,22.04,18.85,20.33,22.51,24.47,20.69,17.68,18,24.14,15.93,26.3,19.59,19.01,20.69,22.02,20.88,16.23,20.68,21,21.39,18.02,15.26,19.22,18.14,18.88,17.9,19.18,19.43,21.72,22.09,18.38,19.45,20.78,22.71,20.08,16.67,20.63,19.93,23.87,20.75,18.94,19,18.89,20.29,20.5,21.73,21.06,19.33,18.22,21.41,19.68,20.39,22.27,23.64,18.13,23.3,18.27,20.55,20.53,19.49,22.8,18.88,21.33,19.99,18.16,22.41,15.84,19.14],["female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male","female","male"],[7.08,3.54,5.1,6.71,3.88,5.8,3.98,4.54,3.3,5.3,5.31,4.78,6.23,4.31,2.01,5.28,4.37,3.99,4.18,2.27,4.2,1.61,4.17,5.48,4.04,5.72,5.95,4.78,2.47,4.27,3.82,3.01,4.71,7.9,3.88,1.83,4.6,6.05,5.22,6.21,4.85,2.83,4.1,4.69,5.78,3.54,2.66,2.05,5.41,4.83,3.4,4.52,3.23,3.86,5.39,4.94,3,3.98,4.88,7.68,5.47,5.09,7.95,2.12,6.54,6.16,2.64,7.59,3.9,4.36,3.76,3.27,5.15,5.75,7.27,5.46,4.06,3.79,2.1,4.84,4.6,4.48,4.47,2.92,4.26,4.51,5.33,4.27,2.84,3.59,4.95,5.63,4.18,3.07,3.38,4.91,6.13,4.92,6.31,5.36,3.27,3.05,5.82,4.69,3.95,6.67,3.98,7.31,6.48,3.15,5.24,4.46,6.79,6.19,4.1,5.05,5.53,5.72,6.17,1.98,5.28,3.46,3.58,3.84,4.26,6.66,5.24,3.8,6.61,4.12,5.15,5.37,5.47,5,3.78,3.81,5.34,3.28,2.39,4.72,4.56,4.8,5.77,5.51,1.8,2.67,3.96,4.67,3.88,4.18,5.33,4.02,5.71,1.16,3.73,5.03,3.6,5.22,2.18,5.61,7.7,3.68,4.04,6,5.71,7.21,5.14,7.22,5.85,4.77,3.57,5.5,4.47,4.07,2.38,5.51,6.29,7.05,3.91,4.76,3.77,5,4.32,4.37,4.76,5.81,3.38,5.18,2.92,4.53,2.69,4.23,4.79,4.37,5.31,6.89,3.08,2.87,5.7,3.69,3.88,6.08,2.82,5.11,4.58,4.88,5.54,6.35,5.39,4.08,3.72,3.57,4.14,3.71,6.46,3.91,5.43,4.83,4.39,6.36,4.16,5.02,4.64,4.61,2.57,6.09,5.2,7.47,5.79,5.16,2.66,5.34,4.88,5.97,4.39,3.03,5.32,5.26,2.73,4.91,3.61,5.42,6.25,2.91,4.38,5.45,5.22,6.04,3.28,5.51,7.16,4.72,3.53,5.96,5.16,6.26,3.85,4.53,4.43,4.76,5.41,3.27,4.44,5.06,8.77,5.68,3.87,4.74,6.03,5.57,5.33,5.76,4.73,3.2,3.26,3.54,4.34,3.98,5.68,4.79,2.02,4.73,5.33,5.48,3.42,1.85,3.83,4.95,2.49,5.35,3.03,5.29,6.17,2.51,6.13,5.05,4.71,4.53,7.22,5.56],[-1.12,1.66,-1.87,-0.86,0.64,0.79,-0.36,0.13,0.67,1.4,-0.82,1.41,0.16,-1.48,1.22,1.03,1.04,-0.25,0.81,-0.05,-0.33,1.13,0.36,-0.63,0.67,0.47,-0.4,-0.64,1.61,0.17,-0.16,0.1,0.23,0.39,-1.97,-0.13,0.08,0.84,0.16,-0.88,-0.08,1.55,-1.19,-0.87,-1.5,-1.15,1.08,-0.73,-0.15,-0.78,-0.09,0.71,2.18,0.45,-0.06,0.05,-0.04,-0.62,0.23,-0.54,0.63,-0.71,0.08,-1.52,-0.5,0.34,-0.65,0.89,-0.98,2.43,0.72,0,1.82,-1.22,2.02,1.25,-0.16,0.21,-0.61,-0.29,-1.74,0.52,2.42,0.81,0.24,-0.02,0.46,1.01,1.17,0.04,0.96,0.65,0.62,0.93,3.73,0.13,-0.93,0.25,-0.7,0.58,-0.35,1.12,-1.08,-1.13,0.87,1.18,-1.01,-0.05,0.2,0.97,-0.48,1.04,1.31,-0.02,-0.32,-0.62,-0.16,1.32,-0.85,0.51,-0.28,0.79,-0.31,-1.67,0.06,0.15,0.9,-1.03,-1.65,-1.24,-0.57,-0.87,0.09,-1.33,-0.62,-0.59,-0.9,0.18,-0.75,-0.03,-3.02,0.15,-0.86,0.95,0.03,0.01,-0.44,-0.95,-0.54,-1.75,-1.36,0.4,-0.08,-0.77,-2.29,-0.89,-0.77,-0.96,-1,0.2,-0.09,-1.47,-1.31,-0.12,1.52,1,-1.2,-1.15,0.15,-0.29,0.75,-0.27,0.97,0.6,0.26,0.2,1.87,1.36,1.16,0.46,-1.55,-0.67,-0.86,-1.41,0.89,0.14,-0.45,-1.1,0.84,-0.67,-1.91,-0.17,-0.26,1.18,-0.28,0.01,-0.9,-0.43,0.32,0.29,1.34,0.88,-0.14,-0.69,-1.61,0.2,0.69,-0.78,-1.56,-0.93,0.66,-0.08,-0.74,-0.37,0.41,0.49,0.03,0.43,1.17,1.72,-0.11,0.47,1.53,1.8,0.32,0.67,-1.12,0.45,-0.37,0.53,-1.55,0.03,1.88,1.4,0.76,-0.36,2.88,-0.55,1.14,1.5,0.37,-2.69,0.44,-0.46,-0.9,-1.82,0.75,-2.31,-0.52,0.38,0.14,0.19,-0.57,0.12,0.51,0.22,-1.29,1.34,0.53,1.24,0.33,0.13,0.18,-0.38,0.91,2.29,-1.23,0.27,-0.18,-1.18,-1.39,-0.72,-2.31,-0.22,0.57,-0.33,0.64,0.14,-1.32,0.01,-1.33,-0.71,0.77,0.01,0.21,-0.43,0.08,0.71,-1.26,0.91,-0.33,-0.23,0.1,-0.17,2.26,0.31,-0.24,1.99,-0.74,-0.99],[0.35,0.23,0.19,0.58,-1.19,0.4,-0.91,-0.45,-0.25,-0.61,-0.17,-0.76,0.62,2.16,-0.75,-0.14,-0.15,-0.56,-0.26,-1.06,0.02,-0.89,-0.68,2.6,0.34,-0.13,0.34,1.63,0.08,-1.46,-0.88,-0.05,0.17,-0.52,0.88,0.79,0.97,-1.1,-0.39,-0.48,0.83,-0.49,0.75,1,2.69,1.71,-0.58,0.51,-0.77,-0.66,-0.58,0.95,-1.14,0.66,-2.38,-1.98,-0.31,0.58,0.28,0.63,-0.78,-0.13,0.18,-0.24,0.66,0.47,0.77,-1.61,0.04,-1.47,-0.6,0.9,-1.51,0.62,0.39,-0.81,-0.67,-0.86,0.35,0.41,1,-1.19,0.56,-0.06,-0.91,-0.22,0.29,-0.01,0.34,0.18,-0.83,-0.67,-1.37,-1.51,-1.61,0.56,-0.26,0.92,0.81,-0.03,-0.2,-0.8,1.77,0.47,0.18,-2.78,-0.43,-0.39,-1.48,-1,0.53,0.24,-2.23,0.25,0.41,-1.07,-0.41,-0.12,0.11,-1.05,0.22,-0.73,0.62,2.23,-0.17,-0.38,-1.15,0.9,1.44,0.39,1.31,1.05,-0.15,1.2,-0.58,0.99,0.29,1.4,1.31,0.66,1.64,-1.18,0.9,-0.53,-0.24,1.45,-1.06,0.83,0.34,2.21,-0.3,0.11,-0.31,1.75,1.35,0.57,0.74,1.93,2.72,-0.82,-0.44,0.34,-0.95,-0.34,-0.25,-0.74,0.52,1.45,-0.99,0.01,-0.91,-0.26,0.05,-0.12,-0.27,-0.16,0.28,-2.53,-1.46,-1.12,0.96,-0.76,-0.38,0.52,-2.02,-0.67,0.23,2.14,0.55,1.2,1.94,-0.09,1.11,-1.72,-0.12,0.88,0.82,1.35,-0.56,0.59,-1.66,-1.92,1.28,-0.08,1.74,-1.32,-0.24,-0.43,-0.13,0.88,0.09,0.01,0.77,0.06,-0.01,-0.09,0.64,-0.76,-0.4,-0.42,0.83,-0.87,-0.87,-0.55,1.7,-0.32,-0.07,-0.83,-0.7,-0.48,0.85,0.95,-0.22,-1.14,-0.32,1.27,-1.68,0.4,-1.29,-0.49,-0.82,0.22,-0.75,-0.35,-1.28,2.49,0.35,1.2,1.49,-0.4,0.2,-0.85,1.04,0.04,-1.66,-1.03,1.96,-1.49,0.82,-0.38,0.34,-0.14,-0.15,-0.34,-1.94,-1.25,0.06,0.43,0.8,0.82,2.38,-0.29,1.08,-1.24,-0.89,0.25,0.68,-0.66,2.23,0.74,1.24,-0.18,-1.05,-0.68,0.87,0.04,0.52,-1.18,0.15,-0.79,0.74,0.52,-0.32,0.61,-1.17,-1.36,1.18,-2.35,-0.52,0.46],[0.35,0.23,0.19,0.58,-1.19,0.4,-0.91,-0.45,-0.25,-0.61,-0.17,-0.76,0.62,2.16,-0.75,-0.14,-0.15,-0.56,-0.26,-1.06,0.02,-0.89,-0.68,2.6,0.34,-0.13,0.34,1.63,0.08,-1.46,-0.88,-0.05,0.17,-0.52,0.88,0.79,0.97,-1.1,-0.39,-0.48,0.83,-0.49,0.75,1,2.69,1.71,-0.58,0.51,-0.77,-0.66,-0.58,0.95,-1.14,0.66,-2.38,-1.98,-0.31,0.58,0.28,0.63,-0.78,-0.13,0.18,-0.24,0.66,0.47,0.77,-1.61,0.04,-1.47,-0.6,0.9,-1.51,0.62,0.39,-0.81,-0.67,-0.86,0.35,0.41,1,-1.19,0.56,-0.06,-0.91,-0.22,0.29,-0.01,0.34,0.18,-0.83,-0.67,-1.37,-1.51,-1.61,0.56,-0.26,0.92,0.81,-0.03,-0.2,-0.8,1.77,0.47,0.18,-2.78,-0.43,-0.39,-1.48,-1,0.53,0.24,-2.23,0.25,0.41,-1.07,-0.41,-0.12,0.11,-1.05,0.22,-0.73,0.62,2.23,-0.17,-0.38,-1.15,0.9,1.44,0.39,1.31,1.05,-0.15,1.2,-0.58,0.99,0.29,1.4,1.31,0.66,1.64,-1.18,0.9,-0.53,-0.24,1.45,-1.06,0.83,0.34,2.21,-0.3,0.11,-0.31,1.75,1.35,0.57,0.74,1.93,2.72,-0.82,-0.44,0.34,-0.95,-0.34,-0.25,-0.74,0.52,1.45,-0.99,0.01,-0.91,-0.26,0.05,-0.12,-0.27,-0.16,0.28,-2.53,-1.46,-1.12,0.96,-0.76,-0.38,0.52,-2.02,-0.67,0.23,2.14,0.55,1.2,1.94,-0.09,1.11,-1.72,-0.12,0.88,0.82,1.35,-0.56,0.59,-1.66,-1.92,1.28,-0.08,1.74,-1.32,-0.24,-0.43,-0.13,0.88,0.09,0.01,0.77,0.06,-0.01,-0.09,0.64,-0.76,-0.4,-0.42,0.83,-0.87,-0.87,-0.55,1.7,-0.32,-0.07,-0.83,-0.7,-0.48,0.85,0.95,-0.22,-1.14,-0.32,1.27,-1.68,0.4,-1.29,-0.49,-0.82,0.22,-0.75,-0.35,-1.28,2.49,0.35,1.2,1.49,-0.4,0.2,-0.85,1.04,0.04,-1.66,-1.03,1.96,-1.49,0.82,-0.38,0.34,-0.14,-0.15,-0.34,-1.94,-1.25,0.06,0.43,0.8,0.82,2.38,-0.29,1.08,-1.24,-0.89,0.25,0.68,-0.66,2.23,0.74,1.24,-0.18,-1.05,-0.68,0.87,0.04,0.52,-1.18,0.15,-0.79,0.74,0.52,-0.32,0.61,-1.17,-1.36,1.18,-2.35,-0.52,0.46],[-0.25,0.67,-0.95,-0.7,-1.43,-0.59,-0.35,-0.87,0.4,-0.03,-1.73,-0.3,0.75,1.9,-0.69,-0.75,0.39,-0.48,0.47,-0.73,-1.56,-1.6,-0.65,2.65,-0.75,-0.28,0.38,0.81,0.96,-0.76,-0.59,1.26,0.04,-0.95,1.13,-0.29,-0.46,0.06,-0.4,-0.06,-0.42,0.97,-0.62,-0.13,2.14,1.51,-0.92,-0.34,-0.51,-1.96,-1.35,-0.1,-0.49,0.88,-2.07,-1.66,1.13,0.78,1.1,0.25,0.46,-0.96,0.08,0.31,0.62,-0.56,0.64,-1.32,0.34,-0.34,0.01,0.22,0.26,0.66,1.69,-0.15,0.2,0.13,0.07,-0.93,-0.22,-0.72,-0.04,0.84,-1.22,-0.15,0.95,-0.06,1.2,0.45,-0.64,-1.2,-1.52,-0.65,-0.9,0.35,-0.35,0.33,1.31,-0.33,-0.41,-1.45,1.04,0.2,0.74,-0.78,-0.16,-0.32,-1.31,-0.53,0.7,-0.2,-1.71,1.01,0.52,-0.11,-0.55,0.37,0.78,-1.17,-0.75,-0.3,0.18,1.43,-0.35,-1.64,1.31,0.61,0.91,0.72,1.5,-0.01,0.85,0.98,-0.64,1.13,-0.47,0.36,0.89,-0.45,1.55,-2.05,1.07,-0.33,0.56,2.74,-0.28,0.32,0.64,2.79,-0.99,0.48,0.79,1.73,2.03,0.66,-1.02,1.97,1.61,-1.55,0.11,-1.48,-1.59,-0.98,0.4,-1.58,0.69,0.44,-1,-0.92,-0.83,1.18,-0.2,-0.28,-0.81,1.04,-0.76,-1.44,-1.52,-0.72,1.05,0.25,0.27,1.2,-2.08,-1.71,-0.8,1.27,1.76,0.21,0.92,-1.97,1.13,-0.58,-2.09,1.52,-1.15,1.47,-0.48,0.89,-0.29,-0.8,-0.13,0.22,0.7,-2.75,0.64,-0.14,-0.86,0.53,2.02,0.44,0.98,0.86,-0.3,-1.14,0.51,0.54,-0.11,-0.84,1.65,-0.2,1.09,0.08,0.76,0.15,0.34,0.14,-0.61,-0.25,1.52,0.94,0.35,0.02,-1.18,0.63,-0.55,-0.22,-1.13,0.27,-0.13,-0.89,-0.52,0.55,-1.71,1.31,0.88,-0.23,1.77,-0.92,-0.43,-0.08,0.88,0.49,-1.58,-0.13,3.26,-0.95,0.14,0.68,0.33,0.27,-0.33,-0.08,-0.58,-0.97,0.46,0.9,0.69,-0.07,0.9,-0.87,-0.06,-1.3,-0.72,-1.45,1.02,-1.98,2.31,0.31,0.78,0.13,-2.01,-0.8,-0.01,-0.04,0.54,-0.94,-0.55,-0.04,0.46,-0.13,-1.2,1.69,0.5,-0.63,0.14,-0.25,-0.6,0.78],[-0.25,0.67,-0.95,-0.7,-1.44,-0.59,-0.35,-0.87,0.41,-0.03,-1.74,-0.3,0.75,1.91,-0.69,-0.75,0.39,-0.48,0.47,-0.74,-1.56,-1.61,-0.65,2.66,-0.75,-0.28,0.38,0.81,0.96,-0.76,-0.59,1.26,0.04,-0.95,1.14,-0.29,-0.46,0.07,-0.4,-0.06,-0.42,0.97,-0.63,-0.13,2.14,1.52,-0.92,-0.34,-0.51,-1.96,-1.36,-0.1,-0.49,0.88,-2.07,-1.66,1.13,0.78,1.1,0.25,0.46,-0.96,0.08,0.31,0.63,-0.56,0.64,-1.32,0.35,-0.34,0.01,0.23,0.27,0.67,1.7,-0.15,0.21,0.13,0.07,-0.94,-0.22,-0.72,-0.04,0.84,-1.22,-0.15,0.95,-0.06,1.2,0.46,-0.64,-1.21,-1.53,-0.65,-0.91,0.35,-0.35,0.33,1.32,-0.33,-0.41,-1.46,1.04,0.21,0.75,-0.79,-0.16,-0.32,-1.31,-0.53,0.7,-0.2,-1.71,1.02,0.53,-0.11,-0.55,0.38,0.78,-1.17,-0.75,-0.3,0.18,1.44,-0.35,-1.65,1.32,0.61,0.92,0.73,1.51,-0.01,0.85,0.99,-0.64,1.13,-0.47,0.36,0.89,-0.45,1.55,-2.05,1.07,-0.33,0.56,2.75,-0.28,0.32,0.64,2.79,-0.99,0.48,0.79,1.74,2.04,0.66,-1.03,1.97,1.61,-1.56,0.11,-1.48,-1.59,-0.98,0.41,-1.59,0.69,0.44,-1,-0.92,-0.83,1.19,-0.2,-0.28,-0.81,1.04,-0.76,-1.45,-1.52,-0.73,1.05,0.26,0.27,1.21,-2.08,-1.71,-0.81,1.28,1.77,0.21,0.92,-1.98,1.14,-0.58,-2.09,1.53,-1.15,1.47,-0.48,0.9,-0.29,-0.8,-0.12,0.22,0.7,-2.76,0.64,-0.14,-0.86,0.53,2.02,0.44,0.99,0.86,-0.3,-1.14,0.52,0.55,-0.11,-0.84,1.65,-0.2,1.09,0.08,0.76,0.15,0.35,0.14,-0.61,-0.25,1.53,0.95,0.36,0.02,-1.18,0.63,-0.55,-0.22,-1.13,0.27,-0.13,-0.9,-0.52,0.55,-1.71,1.32,0.88,-0.23,1.78,-0.92,-0.43,-0.08,0.89,0.49,-1.58,-0.13,3.01,-0.95,0.15,0.68,0.33,0.27,-0.33,-0.08,-0.58,-0.98,0.46,0.9,0.69,-0.07,0.9,-0.87,-0.06,-1.31,-0.72,-1.45,1.02,-1.99,2.32,0.31,0.78,0.13,-2.01,-0.8,-0.01,-0.04,0.54,-0.94,-0.55,-0.04,0.46,-0.13,-1.2,1.69,0.5,-0.64,0.14,-0.25,-0.61,0.78],[-0.85,1.31,-1.17,-0.85,-0.96,0.03,-0.97,-1.61,0.42,0.11,-0.67,0.26,1.28,1.05,-0.21,-0.12,0.55,-1.38,0.2,-1.63,-0.2,-1.18,-1.42,2.55,-0.1,0.09,0.07,0.73,1.38,-1.41,-0.95,0.1,1.69,-1.2,-0.28,0.12,-0.13,-0.63,-0.75,-1.13,-0.99,1.21,-0.84,0.12,1.98,0.49,-0.43,-0.56,-0.58,-1.9,-0.62,0,-0.43,0.63,-1.86,-1.97,-0.19,0.64,0.91,0.19,-0.66,-0.44,-0.37,-0.51,0.72,0.68,-0.51,-1.22,-0.73,0.02,-0.07,1.74,0.28,-0.05,2.38,0.39,-0.52,-0.04,0.05,-0.79,0.49,-0.87,1.02,2.03,-1.26,-0.99,1.14,-0.03,0.63,0.07,0.68,-0.47,-1.24,-0.98,0.87,0.52,-1.09,1.08,2.67,0.05,-0.25,-0.3,0.89,-0.52,0.88,-1.16,-0.82,-1.1,-0.74,-0.74,0.07,1.79,-1.32,1.53,-0.02,-0.84,0.1,1.22,0.05,-1.15,-0.4,-0.59,0.5,0.42,-0.11,-0.23,0.29,0.26,1.44,0.22,0.53,1.03,-1.14,1.33,0.42,0.67,-1.03,0.28,0.36,0.33,-0.02,-1.77,0.97,-0.17,0.23,2.02,-1.15,0.36,-0.53,1.36,-1.11,0.59,0.65,1.73,0.97,-0.02,-0.86,1.37,1.91,-1.99,-0.37,-0.94,-1.19,-1.16,1.71,-0.56,1.12,0.56,-0.83,-1.28,-0.8,-0.44,0.65,0.44,0.3,0.47,0.64,-1.77,-1.6,-1.18,0.64,-0.92,-0.41,0.36,-0.89,-1.12,0.15,0.68,0.69,-0.03,0.62,-0.64,1.03,-1.4,-0.68,1.4,-0.3,2.08,-1.33,1.27,-0.84,-0.61,0.52,-1.29,1.33,-2.1,0.12,-0.29,0.22,0.54,0.51,0.59,-0.05,-0.27,-0.12,-0.61,1.6,-0.26,0.31,0.15,2.38,-0.27,0.37,0.51,0.94,0.52,-0.96,0.7,-0.5,-0.87,-0,2.01,1.74,-0.44,-0.9,0.5,-1.54,1.83,-0.67,-0.29,-0.91,-2.39,-0.07,0.21,-1.81,0.71,1.42,-1.58,0.77,-0.66,-0.57,0.03,1.41,0.43,-1.22,-1.24,3.05,0.11,1.36,0.22,-0.23,-0.03,0.28,-0.62,-1.23,0.24,-0.34,0.42,1.31,-0.31,1.53,-0.61,-0.07,-2.27,-0.39,-1.07,1.68,-1.37,1.8,-0.01,0.53,1.18,-1.09,-0.03,1.21,0.23,0.86,-0.66,-0.91,0.11,1.51,0.56,0.96,0.56,0.43,-1.35,0.27,-0.52,-1.33,0.7],[-0.85,1.31,-1.17,-0.85,-0.96,0.03,-0.97,-1.61,0.42,0.11,-0.67,0.26,1.28,1.05,-0.21,-0.12,0.55,-1.38,0.21,-1.63,-0.2,-1.18,-1.42,2.55,-0.1,0.09,0.07,0.73,1.39,-1.41,-0.95,0.1,1.69,-1.2,-0.28,0.12,-0.13,-0.63,-0.75,-1.13,-0.99,1.21,-0.84,0.12,1.98,0.49,-0.43,-0.56,-0.58,-1.9,-0.62,0,-0.43,0.63,-1.86,-1.97,-0.19,0.64,0.91,0.19,-0.66,-0.44,-0.37,-0.51,0.72,0.68,-0.51,-1.22,-0.74,0.02,-0.07,1.75,0.28,-0.05,2.38,0.39,-0.52,-0.04,0.05,-0.8,0.49,-0.87,1.02,2.03,-1.26,-0.99,1.14,-0.03,0.63,0.07,0.68,-0.47,-1.24,-0.98,0.87,0.52,-1.1,1.08,2.67,0.05,-0.25,-0.3,0.89,-0.52,0.88,-1.16,-0.82,-1.1,-0.74,-0.74,0.07,1.79,-1.32,1.53,-0.02,-0.84,0.1,1.22,0.05,-1.15,-0.4,-0.59,0.5,0.43,-0.11,-0.23,0.29,0.26,1.44,0.22,0.53,1.03,-1.14,1.33,0.42,0.67,-1.03,0.28,0.36,0.33,-0.02,-1.77,0.97,-0.17,0.23,2.02,-1.15,0.36,-0.53,1.36,-1.11,0.59,0.65,1.73,0.97,-0.02,-0.86,1.37,1.91,-1.99,-0.37,-0.94,-1.19,-1.16,1.71,-0.56,1.12,0.56,-0.83,-1.28,-0.8,-0.44,0.65,0.44,0.3,0.47,0.64,-1.77,-1.6,-1.18,0.64,-0.92,-0.41,0.37,-0.89,-1.12,0.15,0.68,0.69,-0.03,0.63,-0.64,1.03,-1.4,-0.68,1.4,-0.3,2.09,-1.33,1.27,-0.84,-0.61,0.52,-1.29,1.33,-2.11,0.12,-0.29,0.22,0.54,0.51,0.59,-0.05,-0.27,-0.12,-0.61,1.61,-0.26,0.31,0.15,2.39,-0.27,0.37,0.51,0.94,0.52,-0.96,0.7,-0.5,-0.87,-0,2.01,1.75,-0.44,-0.9,0.5,-1.54,1.83,-0.67,-0.29,-0.91,-2.4,-0.07,0.21,-1.81,0.71,1.42,-1.58,0.77,-0.66,-0.57,0.03,1.41,0.43,-1.22,-1.24,3,0.11,1.36,0.22,-0.23,-0.03,0.28,-0.62,-1.23,0.24,-0.34,0.42,1.31,-0.31,1.53,-0.61,-0.07,-2.27,-0.39,-1.07,1.68,-1.37,1.8,-0.01,0.53,1.18,-1.09,-0.03,1.21,0.23,0.87,-0.66,-0.91,0.11,1.51,0.56,0.96,0.56,0.43,-1.35,0.27,-0.52,-1.33,0.7],[0.73,-0.2,0.99,-1.64,-0.23,0.96,-0.47,-0.7,-0.61,0.37,-1.49,-0.36,0.66,2.3,-1.13,0.13,0.03,-0.95,-0.88,-0.56,-0.86,-0.7,-0,2.01,-0.59,0.12,0.18,1.57,0.49,-0.37,-1.39,0.07,-0.38,0.16,1.45,0.5,0.49,0.1,-1.12,-0.78,0.24,0.25,-0.68,-0.56,0.64,0.7,-0.3,-0.77,0.19,-0.79,-0.31,0.7,-0.3,1.2,-1.38,-1.99,0.36,1.11,-0.65,0.17,-2.58,0.14,-0.71,-0.01,0.8,-0.13,1.94,-1.46,-0.15,-2.28,-0.48,0.78,-0.63,1,1.28,-0.27,0.5,0.03,-0.05,0.73,0.66,0.23,-1.23,1.63,-0.53,-0.31,-1.17,0.37,-0.44,-0.28,0.66,-1.19,-1.12,-0.68,-1.61,-0.59,1.43,0.63,1.27,0.57,0.73,0.08,0.95,-0.15,0.7,-1.81,-0.73,-0.92,-0.3,-1.74,0.68,0.55,-1.4,0.67,0.23,0.47,0.41,0.31,-0.55,0.3,0.63,-1.39,0.25,0.14,0.17,-0.63,0.04,0.04,0.71,0.34,-0.12,0.67,-0.59,1.61,-0.26,1.62,0.39,0.58,0.68,0.12,2,-1.1,1.76,0.56,1.2,0.68,-2.2,-0,1.38,1.63,-0.39,0.03,0.33,0.93,1.51,1.15,-0.98,0.58,1.42,-1.88,-0.43,0.38,-1.01,-0.86,0.68,-2.24,0.27,1.54,0.25,-0.13,-2.4,-0.15,-0.44,0.14,0.45,0.01,-0.76,-2.11,-0.97,-1.22,1.3,-1.04,-0.22,1.54,-0.4,0.31,-1.49,0.78,0.39,1.02,0.17,-0.28,0.75,-0.82,1.47,1.07,-1.45,1.04,-0.37,0.39,-0.38,-1.73,-1.67,0.38,2.45,-2.48,-0.72,-0.54,0.1,-0.55,0.09,-0.45,0.41,1.46,-1.01,-1.53,0.34,1.36,-0.65,0.14,1.8,0.05,0.65,0.4,1.23,-1.91,0.37,1.76,-0.14,-0.23,1,3.09,0.71,-0.58,0.17,0.1,-0.53,1.33,-1.07,0.03,-0.72,-1.02,-0.17,0.95,-1.11,2.11,0.57,0.32,0.11,-1.01,-0.08,-0.4,0.05,0.27,-0.61,-1.19,2.42,-0.52,-0.38,0.45,-1.01,0.44,0.36,-0.88,-1.78,-0.19,0.46,1.26,0.35,1.05,0.25,-0.34,0.95,-2.74,-0.09,-1.09,2.44,-0.64,1.31,-0.97,-0.44,0.46,-1.26,0.84,0.66,1.12,0.72,-0.92,-0.4,-1.04,0.43,-0.36,-0.15,1.47,-1.91,-0.07,-0.8,-0.97,0.13,0.32],[0.73,-0.2,0.99,-1.64,-0.23,0.96,-0.47,-0.7,-0.61,0.37,-1.49,-0.36,0.66,2.3,-1.13,0.13,0.03,-0.95,-0.88,-0.56,-0.86,-0.71,-0,2.01,-0.59,0.12,0.18,1.57,0.49,-0.37,-1.39,0.07,-0.38,0.16,1.45,0.5,0.49,0.1,-1.12,-0.78,0.24,0.25,-0.68,-0.56,0.64,0.7,-0.3,-0.77,0.19,-0.79,-0.31,0.7,-0.3,1.2,-1.38,-1.99,0.36,1.11,-0.65,0.17,-2.58,0.14,-0.71,-0.01,0.8,-0.13,1.94,-1.46,-0.15,-2.28,-0.48,0.79,-0.63,1,1.29,-0.27,0.5,0.03,-0.05,0.73,0.66,0.23,-1.23,1.63,-0.53,-0.31,-1.17,0.37,-0.44,-0.28,0.66,-1.19,-1.12,-0.68,-1.61,-0.59,1.44,0.63,1.27,0.57,0.73,0.08,0.95,-0.15,0.7,-1.82,-0.73,-0.92,-0.3,-1.74,0.68,0.55,-1.4,0.68,0.23,0.47,0.42,0.31,-0.55,0.3,0.63,-1.39,0.25,0.14,0.17,-0.63,0.04,0.04,0.71,0.34,-0.12,0.67,-0.59,1.61,-0.26,1.62,0.39,0.58,0.68,0.12,2,-1.1,1.76,0.56,1.21,0.68,-2.2,-0,1.38,1.63,-0.39,0.03,0.33,0.93,1.51,1.15,-0.98,0.58,1.43,-1.88,-0.43,0.38,-1.01,-0.86,0.68,-2.24,0.27,1.54,0.25,-0.13,-2.4,-0.15,-0.44,0.14,0.45,0.01,-0.76,-2.11,-0.97,-1.22,1.3,-1.04,-0.22,1.54,-0.4,0.32,-1.49,0.78,0.39,1.02,0.17,-0.28,0.75,-0.82,1.47,1.07,-1.45,1.04,-0.37,0.39,-0.38,-1.73,-1.67,0.38,2.45,-2.49,-0.72,-0.54,0.1,-0.55,0.09,-0.45,0.41,1.46,-1.01,-1.53,0.34,1.36,-0.65,0.14,1.8,0.05,0.65,0.4,1.24,-1.92,0.37,1.76,-0.14,-0.23,1,3,0.71,-0.58,0.18,0.1,-0.53,1.33,-1.07,0.03,-0.72,-1.02,-0.17,0.95,-1.11,2.11,0.57,0.32,0.11,-1.01,-0.08,-0.4,0.06,0.28,-0.61,-1.19,2.42,-0.52,-0.38,0.46,-1.01,0.44,0.36,-0.88,-1.78,-0.19,0.46,1.26,0.35,1.05,0.26,-0.34,0.96,-2.74,-0.09,-1.09,2.44,-0.64,1.31,-0.97,-0.44,0.47,-1.27,0.84,0.66,1.12,0.72,-0.92,-0.4,-1.04,0.43,-0.36,-0.15,1.47,-1.91,-0.07,-0.8,-0.97,0.13,0.32]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th>PID<\/th>\n <th>age<\/th>\n <th>gender<\/th>\n <th>mother_edu<\/th>\n <th>SWLS<\/th>\n <th>CESD10<\/th>\n <th>CESD10_winsorized<\/th>\n <th>GAD7<\/th>\n <th>GAD7_winsorized<\/th>\n <th>PANAS_negative_affect<\/th>\n <th>PANAS_negative_affect_winsorized<\/th>\n <th>PSS<\/th>\n <th>PSS_winsorized<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"scrollX":true,"scrollY":true,"pageLength":5,"autoWidth":true,"fixedColumns":{"leftColumns":1},"dom":"tp","columnDefs":[{"className":"dt-right","targets":[1,3,4,5,6,7,8,9,10,11,12]}],"order":[],"orderClasses":false,"lengthMenu":[5,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> --- ## 2. Specify and estimate models There are various methods for running a large number of model specifications. <br> Here, we'll look at one method using `{specr}`, but there are additional examples using `{MuMIn}` and `{purrr}` in the tutorial `.Rmd file`. --- ## Estimate models using `{specr}` .panelset[ .panel[.panel-name[info] #### Advantages * Very simple to use! * You can easily run the models in specific subsets of your data #### Disadvantages / limitations * Need to adapt functions for deeper customization ] .panel[.panel-name[code] ```r dvs = "SWLS" ivs = c("CESD10", "GAD7", "PANAS_negative_affect", "PSS", "CESD10_winsorized", "GAD7_winsorized", "PANAS_negative_affect_winsorized", "PSS_winsorized") control_vars = c("age", "gender", "mother_edu") model = "lm" output_specr = run_specs(df = model_df, y = dvs, x = ivs, controls = control_vars, model = model, subsets = NULL, keep.results = TRUE) ``` ] .panel[.panel-name[output] <div id="htmlwidget-1bda88e41eda8d9ac1e7" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-1bda88e41eda8d9ac1e7">{"x":{"filter":"none","vertical":false,"extensions":["FixedColumns"],"data":[["CESD10","GAD7","PANAS_negative_affect","PSS","CESD10_winsorized","GAD7_winsorized","PANAS_negative_affect_winsorized","PSS_winsorized","CESD10","GAD7","PANAS_negative_affect","PSS","CESD10_winsorized","GAD7_winsorized","PANAS_negative_affect_winsorized","PSS_winsorized","CESD10","GAD7","PANAS_negative_affect","PSS","CESD10_winsorized","GAD7_winsorized","PANAS_negative_affect_winsorized","PSS_winsorized","CESD10","GAD7","PANAS_negative_affect","PSS","CESD10_winsorized","GAD7_winsorized","PANAS_negative_affect_winsorized","PSS_winsorized","CESD10","GAD7","PANAS_negative_affect","PSS","CESD10_winsorized","GAD7_winsorized","PANAS_negative_affect_winsorized","PSS_winsorized"],["SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS"],["lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm","lm"],["age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age + gender + mother_edu","age","age","age","age","age","age","age","age","gender","gender","gender","gender","gender","gender","gender","gender","mother_edu","mother_edu","mother_edu","mother_edu","mother_edu","mother_edu","mother_edu","mother_edu","no covariates","no covariates","no covariates","no covariates","no covariates","no covariates","no covariates","no covariates"],[-0.61,-0.26,-0.05,-0.36,-0.61,-0.26,-0.05,-0.36,-0.6,-0.26,-0.05,-0.36,-0.6,-0.26,-0.05,-0.36,-0.59,-0.26,-0.02,-0.36,-0.59,-0.26,-0.02,-0.36,-0.6,-0.26,-0.01,-0.36,-0.6,-0.26,-0.01,-0.36,-0.59,-0.26,-0.02,-0.36,-0.59,-0.26,-0.02,-0.36],[0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05,0.05,0.06,0.06,0.05],[-13.16,-4.59,-0.77,-6.68,-13.16,-4.59,-0.77,-6.69,-13.2,-4.71,-0.91,-6.75,-13.2,-4.7,-0.9,-6.75,-12.57,-4.66,-0.28,-6.66,-12.57,-4.65,-0.28,-6.67,-12.6,-4.61,-0.26,-6.6,-12.6,-4.6,-0.25,-6.61,-12.62,-4.69,-0.34,-6.67,-12.62,-4.68,-0.33,-6.67],[0,0,0.44,0,0,0,0.44,0,0,0,0.37,0,0,0,0.37,0,0,0,0.78,0,0,0,0.78,0,0,0,0.8,0,0,0,0.8,0,0,0,0.74,0,0,0,0.74,0],[-0.7,-0.37,-0.16,-0.46,-0.7,-0.37,-0.16,-0.46,-0.69,-0.37,-0.17,-0.46,-0.69,-0.37,-0.17,-0.47,-0.68,-0.37,-0.13,-0.47,-0.68,-0.37,-0.13,-0.47,-0.69,-0.37,-0.13,-0.47,-0.69,-0.37,-0.13,-0.47,-0.68,-0.37,-0.13,-0.47,-0.68,-0.37,-0.13,-0.47],[-0.52,-0.15,0.07,-0.25,-0.52,-0.15,0.07,-0.25,-0.51,-0.15,0.06,-0.25,-0.51,-0.15,0.06,-0.26,-0.5,-0.15,0.1,-0.25,-0.5,-0.15,0.1,-0.25,-0.5,-0.15,0.1,-0.25,-0.5,-0.15,0.1,-0.25,-0.5,-0.15,0.09,-0.25,-0.5,-0.15,0.09,-0.25],[0.39,0.09,0.03,0.16,0.39,0.09,0.03,0.16,0.39,0.09,0.03,0.16,0.39,0.09,0.03,0.16,0.35,0.07,0,0.13,0.35,0.07,0,0.13,0.35,0.07,0,0.13,0.35,0.07,0,0.13,0.35,0.07,0,0.13,0.35,0.07,0,0.13],[0.38,0.08,0.02,0.15,0.38,0.08,0.02,0.15,0.38,0.09,0.02,0.15,0.38,0.09,0.02,0.15,0.34,0.06,-0,0.13,0.34,0.06,-0,0.13,0.35,0.06,-0,0.12,0.35,0.06,-0,0.12,0.35,0.07,-0,0.13,0.35,0.07,-0,0.13],[0.79,0.96,0.99,0.92,0.79,0.96,0.99,0.92,0.79,0.96,0.99,0.92,0.79,0.96,0.99,0.92,0.81,0.97,1,0.94,0.81,0.97,1,0.93,0.81,0.97,1,0.94,0.81,0.97,1,0.94,0.81,0.97,1,0.93,0.81,0.97,1,0.93],[46.85,7.69,2.41,13.77,46.85,7.68,2.41,13.78,93.38,15.31,4.35,27.3,93.38,15.28,4.35,27.33,79.41,11.09,0.27,22.47,79.41,11.05,0.27,22.51,80,11.01,0.41,22.21,80,10.98,0.41,22.25,159.26,22.02,0.11,44.43,159.26,21.94,0.11,44.53],[0,0,0.05,0,0,0,0.05,0,0,0,0.01,0,0,0,0.01,0,0,0,0.76,0,0,0,0.76,0,0,0,0.66,0,0,0,0.66,0,0,0,0.74,0,0,0,0.74,0],[4,4,4,4,4,4,4,4,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,1,1,1,1,1,1,1,1],[-351.41,-410.3,-420.36,-399.51,-351.41,-410.33,-420.37,-399.48,-352,-410.46,-420.85,-399.87,-352,-410.49,-420.85,-399.84,-360.93,-414.38,-424.91,-404.05,-360.93,-414.41,-424.91,-404.01,-360.54,-414.45,-424.77,-404.28,-360.54,-414.48,-424.77,-404.24,-360.96,-414.49,-425.12,-404.33,-360.96,-414.52,-425.12,-404.29],[714.82,832.6,852.73,811.02,714.82,832.66,852.73,810.96,712.01,828.92,849.69,807.74,712.01,828.98,849.7,807.68,729.85,836.76,857.81,816.09,729.85,836.82,857.82,816.01,729.08,836.9,857.53,816.55,729.08,836.97,857.54,816.47,727.91,834.98,856.25,814.67,727.91,835.05,856.25,814.58],[737.04,854.83,874.95,833.24,737.04,854.88,874.96,833.18,726.82,843.73,864.51,822.56,726.82,843.79,864.51,822.49,744.67,851.57,872.63,830.91,744.67,851.64,872.63,830.83,743.9,851.71,872.35,831.37,743.9,851.78,872.35,831.29,739.02,846.09,867.36,825.78,739.02,846.16,867.36,825.7],[182.84,270.76,289.55,251.97,182.84,270.82,289.56,251.92,183.57,271.05,290.48,252.57,183.57,271.1,290.49,252.52,194.82,278.23,298.45,259.71,194.82,278.29,298.46,259.64,194.32,278.35,298.18,260.1,194.32,278.42,298.18,260.03,194.86,278.43,298.89,260.2,194.86,278.49,298.89,260.13],[295,295,295,295,295,295,295,295,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,297,298,298,298,298,298,298,298,298],[300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300,300],["all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all","all"]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th>x<\/th>\n <th>y<\/th>\n <th>model<\/th>\n <th>controls<\/th>\n <th>estimate<\/th>\n <th>std.error<\/th>\n <th>statistic<\/th>\n <th>p.value<\/th>\n <th>conf.low<\/th>\n <th>conf.high<\/th>\n <th>fit_r.squared<\/th>\n <th>fit_adj.r.squared<\/th>\n <th>fit_sigma<\/th>\n <th>fit_statistic<\/th>\n <th>fit_p.value<\/th>\n <th>fit_df<\/th>\n <th>fit_logLik<\/th>\n <th>fit_AIC<\/th>\n <th>fit_BIC<\/th>\n <th>fit_deviance<\/th>\n <th>fit_df.residual<\/th>\n <th>fit_nobs<\/th>\n <th>subsets<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"scrollX":true,"scrollY":true,"pageLength":5,"autoWidth":true,"dom":"tp","columnDefs":[{"className":"dt-right","targets":[4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21]}],"order":[],"orderClasses":false,"lengthMenu":[5,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> ] ] .footnote[🔗 [`{specr}` documentation](https://masurp.github.io/specr/)] --- ## 3. Plot specification curve ### SCA plot with `{specr}` .panelset[ .panel[.panel-name[code] ```r output_specr %>% mutate(winsorized = ifelse(grepl("winsorized", x), "yes", "no"), x = gsub("_winsorized", "", x)) %>% plot_specs(., choices = c("x", "winsorized", "controls")) ``` ] .panel[.panel-name[output] <img src="SCA_tutorial_2022_files/figure-html/unnamed-chunk-17-1.png" width="396" style="display: block; margin: auto;" /> ] ] --- ## 3. Plot specification curve ### Plot guide .pull-left[.extrasmall[ Panel A * X-axis = ordered model specifications * Y-axis = standardized regression coefficient for the IV-DV relationship * Points = the standardized regression coefficient for a specific models * Error bars = 95% confidence intervals around the point estimate Panel B * X-axis = ordered model specifications (the same as panel A) * Y-axis (right) = analytic decision categories * Y-axis (left) = specific analytic decisions within each category * Lines = denote that a specific analytic decision was true for that model specification Color key * Red = regression coefficient was statistically significant values at p < .05 * Black/grey = regression coefficient was p > .05 ] ] .pull-right[ <img src="SCA_tutorial_2022_files/figure-html/unnamed-chunk-18-1.png" width="432" style="display: block; margin: auto;" /> ] --- ## 3. Plot specification curve ### Homebrew SCA plot .panelset[ .panel[.panel-name[code] .scroll-output[ .extrasmall[ ```r aes = theme_minimal(base_size = 11) + theme(legend.title = element_text(size = 10), legend.text = element_text(size = 9), axis.text = element_text(color = "black"), axis.line = element_line(colour = "black"), legend.position = "none", panel.border = element_blank(), panel.background = element_blank()) colors = c("yes" = "#90BAC7", "no" = "black") plot_data = output_specr %>% arrange(estimate) %>% mutate(specification = row_number(), winsorized = ifelse(grepl("winsorized", x), "yes", "no"), significant.p = ifelse(p.value < .05, "yes", "no"), x = gsub("_winsorized", "", x)) top = plot_data %>% ggplot(aes(specification, estimate, color = significant.p)) + geom_pointrange(aes(ymin = conf.low, ymax = conf.high), size = .5, shape = "", alpha = 1) + geom_point(size = .5) + scale_color_manual(values = colors) + labs(x = "", y = "standardized\nregression coefficient\n") + aes bottom = plot_data %>% mutate(`mental health` = x) %>% gather(variable, value, `mental health`, controls, winsorized) %>% unique() %>% mutate(variable = factor(variable, levels = c("mental health", "winsorized", "controls"))) %>% ggplot(aes(x = specification, y = value, color = significant.p)) + geom_point(aes(x = specification, y = value), shape = 124, size = 5) + facet_grid(variable ~ 1, scales = "free_y", space = "free_y") + scale_color_manual(values = colors) + labs(x = "\nspecification number", y = "") + aes + theme(strip.text.x = element_blank()) ``` ] ] ] .panel[.panel-name[output] <img src="SCA_tutorial_2022_files/figure-html/unnamed-chunk-19-1.png" width="396" style="display: block; margin: auto;" /> ] ] --- ## 3. Plot specification curve ### Grouped by IV .panelset[ .panel[.panel-name[code] .scroll-output[ .extrasmall[ ```r colors = c("CESD10" = "#005F73", "GAD7" = "#EE9B00", "PANAS_negative_affect" = "#9B2226", "PSS" = "#0A9396") plot_data = output_specr %>% arrange(estimate) %>% mutate(specification = row_number(), winsorized = ifelse(grepl("winsorized", x), "yes", "no"), significant.p = ifelse(p.value < .05, "yes", "no"), x = gsub("_winsorized", "", x)) %>% group_by(x) %>% mutate(median = median(estimate, na.rm = TRUE)) top = plot_data %>% ggplot(aes(specification, estimate, color = x, alpha = significant.p)) + geom_pointrange(aes(ymin = conf.low, ymax = conf.high), size = .5, shape = "") + geom_point(size = .5) + geom_hline(aes(yintercept = median, color = x)) + scale_color_manual(values = colors) + scale_alpha_manual(values = c(.5, 1)) + labs(x = "", y = "standardized\nregression coefficient\n") + aes bottom = plot_data %>% mutate(`mental health` = x) %>% gather(variable, value, `mental health`, controls, winsorized) %>% unique() %>% mutate(variable = factor(variable, levels = c("mental health", "winsorized", "controls"))) %>% ggplot(aes(x = specification, y = value, color = x, alpha = significant.p)) + geom_point(shape = 124, size = 5) + facet_grid(variable ~ 1, scales = "free_y", space = "free_y") + scale_color_manual(values = colors) + scale_alpha_manual(values = c(.5, 1)) + labs(x = "\nspecification number", y = "") + aes + theme(strip.text.x = element_blank()) ``` ] ] ] .panel[.panel-name[output] <img src="SCA_tutorial_2022_files/figure-html/unnamed-chunk-20-1.png" width="396" style="display: block; margin: auto;" /> ] ] --- ## 4. Inferential statistics 1. Run the SCA and extract the median, and the number of positive and negative statistically significant models -- 2. Use bootstrap resampling to create a distribution of curves under the null hypothesis -- 3. Generate p-values indicating how surprising the observed results are under the null hypothesis --- ## Null boostrap resampling .panelset[ .panel[.panel-name[overview] * Run SCA to retrieve associations between the independent and dependent variable in each model specification * Extract the dataset for each model specification (which was saved as a model object `fit` in the data frame) * Force the null on each specification by subtracting the effect of the independent variable of interest (b estimate * x) from the dependent variable (y_value) for each observation in the dataset * For each bootstrap, sample with replacement from the null dataset and run all model specifications to generate a curve * Extract median estimate, N positive & significant at p < .05, and N negative & significant p < .05 * Repeat process many times (e.g. 500 or 1000) ] .panel[.panel-name[define functions] * `run_boot_null` = wrapper function to run the bootstrapping procedure or load an existing output file * `sca_boot_null` = function that runs the boostrapping procedure * `summarize_sca` = function to summarize the observed specification curve * `summarize_boot_null` = function to summarize the bootstrapped curves ] .panel[.panel-name[bootstrap `specr` output] <div id="htmlwidget-ff1b68835fda5ec30b66" style="width:100%;height:auto;" class="datatables html-widget"></div> <script type="application/json" data-for="htmlwidget-ff1b68835fda5ec30b66">{"x":{"filter":"none","vertical":false,"extensions":["FixedColumns"],"data":[["Bootstrap001","Bootstrap002","Bootstrap003","Bootstrap004","Bootstrap005","Bootstrap006","Bootstrap007","Bootstrap008","Bootstrap009","Bootstrap010","Bootstrap011","Bootstrap012","Bootstrap013","Bootstrap014","Bootstrap015","Bootstrap016","Bootstrap017","Bootstrap018","Bootstrap019","Bootstrap020","Bootstrap021","Bootstrap022","Bootstrap023","Bootstrap024","Bootstrap025","Bootstrap026","Bootstrap027","Bootstrap028","Bootstrap029","Bootstrap030","Bootstrap031","Bootstrap032","Bootstrap033","Bootstrap034","Bootstrap035","Bootstrap036","Bootstrap037","Bootstrap038","Bootstrap039","Bootstrap040","Bootstrap041","Bootstrap042","Bootstrap043","Bootstrap044","Bootstrap045","Bootstrap046","Bootstrap047","Bootstrap048","Bootstrap049","Bootstrap050","Bootstrap051","Bootstrap052","Bootstrap053","Bootstrap054","Bootstrap055","Bootstrap056","Bootstrap057","Bootstrap058","Bootstrap059","Bootstrap060","Bootstrap061","Bootstrap062","Bootstrap063","Bootstrap064","Bootstrap065","Bootstrap066","Bootstrap067","Bootstrap068","Bootstrap069","Bootstrap070","Bootstrap071","Bootstrap072","Bootstrap073","Bootstrap074","Bootstrap075","Bootstrap076","Bootstrap077","Bootstrap078","Bootstrap079","Bootstrap080","Bootstrap081","Bootstrap082","Bootstrap083","Bootstrap084","Bootstrap085","Bootstrap086","Bootstrap087","Bootstrap088","Bootstrap089","Bootstrap090","Bootstrap091","Bootstrap092","Bootstrap093","Bootstrap094","Bootstrap095","Bootstrap096","Bootstrap097","Bootstrap098","Bootstrap099","Bootstrap100","Bootstrap101","Bootstrap102","Bootstrap103","Bootstrap104","Bootstrap105","Bootstrap106","Bootstrap107","Bootstrap108","Bootstrap109","Bootstrap110","Bootstrap111","Bootstrap112","Bootstrap113","Bootstrap114","Bootstrap115","Bootstrap116","Bootstrap117","Bootstrap118","Bootstrap119","Bootstrap120","Bootstrap121","Bootstrap122","Bootstrap123","Bootstrap124","Bootstrap125","Bootstrap126","Bootstrap127","Bootstrap128","Bootstrap129","Bootstrap130","Bootstrap131","Bootstrap132","Bootstrap133","Bootstrap134","Bootstrap135","Bootstrap136","Bootstrap137","Bootstrap138","Bootstrap139","Bootstrap140","Bootstrap141","Bootstrap142","Bootstrap143","Bootstrap144","Bootstrap145","Bootstrap146","Bootstrap147","Bootstrap148","Bootstrap149","Bootstrap150","Bootstrap151","Bootstrap152","Bootstrap153","Bootstrap154","Bootstrap155","Bootstrap156","Bootstrap157","Bootstrap158","Bootstrap159","Bootstrap160","Bootstrap161","Bootstrap162","Bootstrap163","Bootstrap164","Bootstrap165","Bootstrap166","Bootstrap167","Bootstrap168","Bootstrap169","Bootstrap170","Bootstrap171","Bootstrap172","Bootstrap173","Bootstrap174","Bootstrap175","Bootstrap176","Bootstrap177","Bootstrap178","Bootstrap179","Bootstrap180","Bootstrap181","Bootstrap182","Bootstrap183","Bootstrap184","Bootstrap185","Bootstrap186","Bootstrap187","Bootstrap188","Bootstrap189","Bootstrap190","Bootstrap191","Bootstrap192","Bootstrap193","Bootstrap194","Bootstrap195","Bootstrap196","Bootstrap197","Bootstrap198","Bootstrap199","Bootstrap200","Bootstrap201","Bootstrap202","Bootstrap203","Bootstrap204","Bootstrap205","Bootstrap206","Bootstrap207","Bootstrap208","Bootstrap209","Bootstrap210","Bootstrap211","Bootstrap212","Bootstrap213","Bootstrap214","Bootstrap215","Bootstrap216","Bootstrap217","Bootstrap218","Bootstrap219","Bootstrap220","Bootstrap221","Bootstrap222","Bootstrap223","Bootstrap224","Bootstrap225","Bootstrap226","Bootstrap227","Bootstrap228","Bootstrap229","Bootstrap230","Bootstrap231","Bootstrap232","Bootstrap233","Bootstrap234","Bootstrap235","Bootstrap236","Bootstrap237","Bootstrap238","Bootstrap239","Bootstrap240","Bootstrap241","Bootstrap242","Bootstrap243","Bootstrap244","Bootstrap245","Bootstrap246","Bootstrap247","Bootstrap248","Bootstrap249","Bootstrap250","Bootstrap251","Bootstrap252","Bootstrap253","Bootstrap254","Bootstrap255","Bootstrap256","Bootstrap257","Bootstrap258","Bootstrap259","Bootstrap260","Bootstrap261","Bootstrap262","Bootstrap263","Bootstrap264","Bootstrap265","Bootstrap266","Bootstrap267","Bootstrap268","Bootstrap269","Bootstrap270","Bootstrap271","Bootstrap272","Bootstrap273","Bootstrap274","Bootstrap275","Bootstrap276","Bootstrap277","Bootstrap278","Bootstrap279","Bootstrap280","Bootstrap281","Bootstrap282","Bootstrap283","Bootstrap284","Bootstrap285","Bootstrap286","Bootstrap287","Bootstrap288","Bootstrap289","Bootstrap290","Bootstrap291","Bootstrap292","Bootstrap293","Bootstrap294","Bootstrap295","Bootstrap296","Bootstrap297","Bootstrap298","Bootstrap299","Bootstrap300","Bootstrap301","Bootstrap302","Bootstrap303","Bootstrap304","Bootstrap305","Bootstrap306","Bootstrap307","Bootstrap308","Bootstrap309","Bootstrap310","Bootstrap311","Bootstrap312","Bootstrap313","Bootstrap314","Bootstrap315","Bootstrap316","Bootstrap317","Bootstrap318","Bootstrap319","Bootstrap320","Bootstrap321","Bootstrap322","Bootstrap323","Bootstrap324","Bootstrap325","Bootstrap326","Bootstrap327","Bootstrap328","Bootstrap329","Bootstrap330","Bootstrap331","Bootstrap332","Bootstrap333","Bootstrap334","Bootstrap335","Bootstrap336","Bootstrap337","Bootstrap338","Bootstrap339","Bootstrap340","Bootstrap341","Bootstrap342","Bootstrap343","Bootstrap344","Bootstrap345","Bootstrap346","Bootstrap347","Bootstrap348","Bootstrap349","Bootstrap350","Bootstrap351","Bootstrap352","Bootstrap353","Bootstrap354","Bootstrap355","Bootstrap356","Bootstrap357","Bootstrap358","Bootstrap359","Bootstrap360","Bootstrap361","Bootstrap362","Bootstrap363","Bootstrap364","Bootstrap365","Bootstrap366","Bootstrap367","Bootstrap368","Bootstrap369","Bootstrap370","Bootstrap371","Bootstrap372","Bootstrap373","Bootstrap374","Bootstrap375","Bootstrap376","Bootstrap377","Bootstrap378","Bootstrap379","Bootstrap380","Bootstrap381","Bootstrap382","Bootstrap383","Bootstrap384","Bootstrap385","Bootstrap386","Bootstrap387","Bootstrap388","Bootstrap389","Bootstrap390","Bootstrap391","Bootstrap392","Bootstrap393","Bootstrap394","Bootstrap395","Bootstrap396","Bootstrap397","Bootstrap398","Bootstrap399","Bootstrap400","Bootstrap401","Bootstrap402","Bootstrap403","Bootstrap404","Bootstrap405","Bootstrap406","Bootstrap407","Bootstrap408","Bootstrap409","Bootstrap410","Bootstrap411","Bootstrap412","Bootstrap413","Bootstrap414","Bootstrap415","Bootstrap416","Bootstrap417","Bootstrap418","Bootstrap419","Bootstrap420","Bootstrap421","Bootstrap422","Bootstrap423","Bootstrap424","Bootstrap425","Bootstrap426","Bootstrap427","Bootstrap428","Bootstrap429","Bootstrap430","Bootstrap431","Bootstrap432","Bootstrap433","Bootstrap434","Bootstrap435","Bootstrap436","Bootstrap437","Bootstrap438","Bootstrap439","Bootstrap440","Bootstrap441","Bootstrap442","Bootstrap443","Bootstrap444","Bootstrap445","Bootstrap446","Bootstrap447","Bootstrap448","Bootstrap449","Bootstrap450","Bootstrap451","Bootstrap452","Bootstrap453","Bootstrap454","Bootstrap455","Bootstrap456","Bootstrap457","Bootstrap458","Bootstrap459","Bootstrap460","Bootstrap461","Bootstrap462","Bootstrap463","Bootstrap464","Bootstrap465","Bootstrap466","Bootstrap467","Bootstrap468","Bootstrap469","Bootstrap470","Bootstrap471","Bootstrap472","Bootstrap473","Bootstrap474","Bootstrap475","Bootstrap476","Bootstrap477","Bootstrap478","Bootstrap479","Bootstrap480","Bootstrap481","Bootstrap482","Bootstrap483","Bootstrap484","Bootstrap485","Bootstrap486","Bootstrap487","Bootstrap488","Bootstrap489","Bootstrap490","Bootstrap491","Bootstrap492","Bootstrap493","Bootstrap494","Bootstrap495","Bootstrap496","Bootstrap497","Bootstrap498","Bootstrap499","Bootstrap500"],["SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS","SWLS"],[0,0.01,0.01,0.02,-0.01,0.01,0.01,0,-0.01,-0,-0,0.01,-0,0,-0,0.02,0,0.01,0.02,0,0.02,-0.02,0.02,0.02,0.01,-0.01,-0.01,0.01,-0,0,0.02,-0,-0.01,-0.01,0,0,0.02,0.03,0.02,0.02,-0,0.01,0.01,-0,-0.02,0.02,0,0.01,-0,-0.01,0,-0,0.01,0.01,0,-0.01,-0.02,0.01,0.01,-0,-0.01,0.01,0,0.02,0,0.02,-0.01,0.01,-0,-0,-0.01,-0.03,-0.01,0,0,0,-0.01,-0,-0.01,0.01,-0,-0.01,0.01,-0.03,0.01,0.01,-0.02,-0,-0.01,0.01,0,0,-0,-0,0.01,0,0,-0.01,0.01,0.02,0,0.02,0.01,0,-0.02,-0,-0,0.01,0.01,-0.01,0,-0,0.02,-0.02,-0.01,-0,0,0,-0.01,0.01,-0,-0.01,0.01,0.02,0.01,-0,-0,0,0,-0.02,0.01,-0.01,-0.01,-0.01,-0,0.01,-0,0.01,-0.01,-0.01,-0.01,0.01,0.01,-0.01,0.01,-0.02,0.01,0.01,0,-0.01,0,-0.01,0.01,-0,0.01,0.01,0.01,0.01,0.02,0.02,0,0.01,0,0.01,-0,0.01,-0,0,0.02,0.02,-0,-0,-0.02,0,-0,0.01,0.01,0.02,0.01,0,0,-0,0.01,0,-0.01,0.03,0.02,-0,-0.01,0.02,-0.02,-0.02,0,-0.02,0.01,-0,0.01,-0,0.01,0.02,-0.01,-0,0.01,-0.02,-0.01,-0.01,-0,-0,-0,-0,0.01,-0.02,0.01,0.01,0.01,0.01,0.01,-0,0,0.04,0.02,0.02,0.01,0.01,0.01,0.02,0.02,0.01,0.01,-0.01,0.02,0.01,-0.01,0.01,0.02,-0,0.01,0.01,0,-0,0.01,0.01,0.01,-0.01,0,0.01,0.01,0,0.01,0.01,-0.01,0.01,0.02,0.02,0,-0,0.01,0.01,0.01,0,0.01,0.01,0.03,-0.01,-0.01,0.01,0.02,-0.02,-0.01,-0,0.01,-0.01,-0.01,-0.02,-0.01,0.02,0.01,0,-0,0.01,-0.01,-0.01,0.01,-0.01,0.02,-0,0,0.01,-0.01,-0.01,-0,-0.01,0.01,-0.01,-0.01,0,0.01,0.01,0.02,0.02,0.01,0.01,0.02,0.01,-0,0.01,0,0.01,-0,0.02,0.01,-0.01,-0,-0,-0.01,-0.01,0.01,-0,0.01,0.01,-0.01,0.01,-0,0.01,-0.01,-0.01,-0.01,0.01,0.01,0,-0,-0,0.02,0,-0.01,0,0,0.01,-0.01,-0,0.04,0.02,-0.01,0.01,0.02,0.01,-0.01,0.01,0,-0.02,0,-0.01,0.02,0.01,0.02,0.02,-0.02,0.01,-0,-0.01,-0.01,-0,0,-0.01,0.01,0.01,0.01,-0.02,0,0,0,-0,-0.01,0.02,0.02,0.01,0.04,0,0,0.01,0.01,0.01,0.01,0.01,-0,0.01,0,0,0.02,0.01,0,0,0.03,-0.02,0.01,0.02,-0,-0.01,-0.01,0.01,0.02,-0.02,-0.01,-0.01,-0,-0,0.02,-0.01,0.01,-0,-0,-0.01,0.02,-0.01,0,0.01,0.02,0.01,0.01,-0.01,0.01,-0,0,-0.01,0.03,-0,0.01,0.01,-0,0.02,-0,0.01,0.02,-0.01,-0.01,-0.01,0,0,-0.01,-0.01,0.01,0,0,-0,0.01,-0,0,0,-0,-0.01,0.02,-0.01,-0,-0.01,0,-0.01,0.01,0.01,-0,-0.01,0.01,0.01,-0.02,0.01,0,-0.01,-0.01,-0.01,0,0.02,-0.02,-0.01,-0,-0.01,0.01,0.01,-0.01,0.01,0.01,0.01,0,0,-0,0.01,-0,-0,-0,-0.02,0.02,-0,-0,-0,0.01,-0.02,-0,-0.01,0.02,0.01,0.01,-0],[40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40,40],[0,0,1,0,3,1,2,0,0,0,1,4,1,1,0,1,0,5,2,0,0,1,7,4,1,3,1,0,0,3,3,4,0,2,2,0,3,3,6,0,0,2,0,1,0,3,1,5,0,1,1,0,1,1,1,0,0,2,2,0,0,0,0,1,1,3,0,2,0,0,0,0,0,1,0,0,0,0,0,1,3,1,0,0,0,3,0,0,1,0,3,0,0,3,1,0,3,0,1,1,1,2,1,0,1,2,1,2,0,0,0,3,3,0,0,0,2,0,0,0,0,1,2,1,0,1,1,4,0,0,6,2,0,1,3,0,1,2,0,0,0,1,0,0,5,2,1,3,1,0,2,0,0,1,0,0,0,4,2,2,0,5,2,3,0,0,1,1,2,3,0,0,0,1,1,4,2,0,1,0,1,0,3,2,0,5,1,0,0,2,1,0,1,0,0,0,1,2,2,1,0,1,1,0,1,1,1,1,0,1,1,0,1,0,3,1,1,0,0,7,0,1,1,2,1,1,4,0,4,0,4,0,0,3,4,0,0,1,2,0,1,1,1,4,0,1,1,0,0,2,3,3,1,2,3,1,0,4,1,1,3,1,8,2,1,1,7,2,0,0,2,0,0,1,0,2,2,2,0,1,3,4,3,1,1,1,2,0,1,0,1,0,4,3,0,3,1,2,0,3,2,1,1,2,2,2,0,2,0,1,2,0,2,0,1,0,1,0,0,1,2,0,0,2,0,1,0,0,0,0,0,3,1,0,0,0,3,0,2,0,8,4,0,3,2,0,0,2,0,0,0,0,5,0,4,5,0,5,2,0,2,1,0,1,2,0,2,1,0,0,1,0,0,3,1,4,5,1,1,1,3,4,2,0,3,0,1,0,1,2,1,0,1,0,0,4,0,4,0,0,3,0,0,0,2,4,0,0,1,0,0,0,3,3,0,3,1,1,2,0,1,0,1,1,5,1,2,2,0,0,0,5,2,1,0,3,0,0,1,1,2,4,0,2,4,1,0,2,1,0,5,1,0,3,2,0,0,0,1,0,0,2,0,0,2,1,0,0,1,3,0,3,1,2,2,1,2,0,1,0,0,0,1,3,0,0,0,0,1,1,2,1,0,0,0,1,1,1,4,0],[3,4,2,0,0,0,0,1,5,0,1,0,1,1,0,0,2,0,0,0,0,0,1,0,0,2,0,0,1,0,1,0,2,3,1,2,0,0,0,0,0,0,0,1,2,0,0,0,6,0,1,0,1,0,0,1,4,1,0,5,0,0,1,0,1,0,1,1,1,0,0,1,4,0,1,0,0,3,3,0,1,1,0,5,0,0,1,0,0,1,2,0,1,2,0,2,2,0,2,2,0,0,1,0,1,0,0,3,0,0,1,4,0,1,0,4,0,0,1,2,1,0,1,1,0,2,0,0,0,0,0,1,1,1,1,0,4,0,0,0,0,0,1,0,0,1,0,0,0,4,0,0,1,0,1,1,0,3,0,0,0,0,1,4,4,0,5,0,0,0,0,0,2,0,0,0,2,0,0,0,0,3,2,1,0,1,0,0,2,0,0,1,0,1,0,1,0,0,1,0,4,1,1,4,0,3,0,0,1,0,2,1,0,1,2,0,0,1,0,0,0,0,0,0,0,0,0,0,4,3,2,0,2,0,0,0,0,2,0,3,1,1,0,3,1,2,0,1,0,0,0,5,1,0,1,1,0,0,2,1,0,2,0,0,2,0,0,2,0,1,1,1,0,4,2,0,0,0,1,0,3,0,1,1,0,2,0,1,1,1,0,0,0,1,2,2,1,0,0,0,0,2,1,0,0,0,1,0,1,0,1,1,1,1,0,3,2,0,0,0,1,0,2,0,3,3,2,0,0,0,2,1,0,1,0,0,1,0,0,1,0,0,0,0,0,1,0,1,1,0,2,1,1,2,0,0,2,1,0,2,1,3,3,2,0,0,0,0,1,1,0,0,2,0,2,2,0,0,1,0,0,1,0,1,0,0,0,3,0,1,0,0,0,1,0,0,1,0,0,0,0,4,2,6,1,2,0,2,1,0,2,3,1,2,1,0,0,1,1,1,1,0,2,3,0,0,0,1,1,3,0,0,0,1,0,3,3,0,1,2,0,0,0,0,1,3,0,2,1,2,0,0,1,2,0,4,0,2,4,0,2,1,5,1,0,2,0,0,0,0,2,0,0,6,0,4,2,2,1,1,0,0,2,1,0,4,0,1,0,2,0,0,0,5,1,5,0,3,0,1]],"container":"<table class=\"display\">\n <thead>\n <tr>\n <th>id<\/th>\n <th>y<\/th>\n <th>median<\/th>\n <th>n<\/th>\n <th>n_positive_sig<\/th>\n <th>n_negative_sig<\/th>\n <\/tr>\n <\/thead>\n<\/table>","options":{"scrollX":true,"scrollY":true,"pageLength":5,"fixedColumns":{"leftColumns":1},"dom":"tp","columnDefs":[{"className":"dt-right","targets":[2,3,4,5]}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[5,10,25,50,100]}},"evals":[],"jsHooks":[]}</script> ] ] --- ## Generate inferential stats .panelset[ .panel[.panel-name[curve metrics] * Curve median * Share of positive statistically significant associations at p < .05 * Share of negative statistically significant associations at p < .05 ] .panel[.panel-name[p-values] These represent the number of times that an equally or more extreme value was observed in the bootstrapped null curve distribution (i.e. 1 / 500 = .002) <br> |y |Mdn |Mdn p |Positive share N |Positive share p |Negative share N |Negative share p | |:----|:-----|:------|:----------------|:----------------|:----------------|:----------------| |SWLS |-0.31 |< .001 |0 / 40 |1.000 |30 / 40 |< .001 | ] ] --- class:center, middle ## Resources --- .panelset[ .panel[.panel-name[Reading list] * [Specification curve analysis - Simonsohn, Simmons, & Nelson, 2020](https://www.nature.com/articles/s41562-020-0912-z) * [Increasing Transparency Through a Multiverse Analysis - Steegen et al., 2016](https://journals.sagepub.com/doi/10.1177/1745691616658637) * [A Traveler’s Guide to the Multiverse: Promises, Pitfalls, and a Framework for the Evaluation of Analytic Decisions - Del Giudice & Gangestad, 2021](https://journals.sagepub.com/doi/full/10.1177/2515245920954925) ] .panel[.panel-name[Example papers using SCA] * [Run All the Models! Dealing With Data Analytic Flexibility - Julia Rohrer](https://www.psychologicalscience.org/observer/run-all-the-models-dealing-with-data-analytic-flexibility) * [The association between adolescent well-being and digital technology use - Orben & Przybylski, 2019](http://nature.com/articles/s41562-018-0506-1) * [Screens, Teens, and Psychological Well-Being: Evidence From Three Time-Use-Diary Studies - Orben & Przybylski, 2019](https://journals.sagepub.com/doi/10.1177/0956797619830329) * [Age-related change in task-evoked amygdala-prefrontal circuitry: a multiverse approach with an accelerated longitudinal cohort aged 4-22 years - Bloom et al., preprint](https://www.biorxiv.org/content/10.1101/2021.10.08.463601v1) * [Neural indicators of food cue reactivity, regulation, and valuation and their associations with body composition and daily eating behavior - Cosme & Lopez, 2020](https://psyarxiv.com/23mu5/) * [Message self and social relevance increases intentions to share content: Correlational and causal evidence from six studies - Cosme et al., preprint](https://psyarxiv.com/9cxfj) ] .panel[.panel-name[Shiny apps] * [Amygdala multiverse explorer - Paul Bloom](https://pbloom.shinyapps.io/amygdala_mpfc_multiverse/) * [SCA explorer - in progress](https://dcosme.shinyapps.io/explore-sca/) ] .panel[.panel-name[Tutorials] * [Amygdala multiverse walkthrough - Paul Bloom](https://pab2163.github.io/amygdala_mpfc_multiverse/into_the_bayesian_multiverse.html) * [Specification curve analysis: A practice guide](https://dcosme.github.io/specification-curves/SCA_tutorial_inferential) ] ] --- class:center, middle # Thank you! The repository can be found at: [https://github.com/dcosme/specification-curves/](https://github.com/dcosme/specification-curves/)