Draw (Causal) Conclusions from Data – Some Evidence

To be data literate, one should be able to draw conclusions from multivariable observational data. But this is tricky. E.g., to investigate the gender pay gap, it must be decided whether the effect should be calculated adjusted or unadjusted for job. The correct conclusion depends on the qualitative assumptions about the data generating process. To investigate the conclusions drawn by students, a randomized experiment is conducted. The same data is presented in two different contexts with (possible) different structural causal models so once the adjusted and once the unadjusted effect might be appropriate. Also it is varied whether a directed acyclic graph is presented before or after the data table with the estimated effect. Results indicates that conclusions drawn from the same data differ by context but may also be inconsistent to the assumed data generating process.