Excellent heuristics and good explanation! I might add:
* Is the mechanism of the causal effect understood, and supported by studies at different levels of analysis, e.g. is a policy intervention also supported by studies of the psychological mechanisms. Or is the medical intervention supported by pharmacology, etc.
* How well do I think the outcome can be measured? As you know, anything that might be analogized to measurement error in the outcome that is truly random error, biases the effect sizes toward zero. Corrections are theoretically possible. Non-random measurement error in the outcome can also be analyzed but let's not worry about that now...
* How did I hear about the study? This heuristic loses value quickly as I learn more about the study, but in general studies I hear about in the mainstream media are less reliable, and the interpretation offered in the media is naturally less reliable than what is written in the research articles.
* Was the study pre-registered? This is a heuristic across all fields, and/but a much stronger heuristic within fields that have some pre-registration and flexbility in how analyses are conducted.
* is it consistent with my prior beliefs? This is sorta a cheat / catch-all... obviously we all use this heuristic, and sometimes over-rely on it. The most interesting and valuable research is that which modifies our prior beliefs, and so despite improving accuracy of beliefs about most research claims, it probably on average harms the accuracy of beliefs about the (far more rare) research claims that people really should be updating the most on.
* What do researchers I respect think of the study claim/evidence? Is their view surprising given what I perceive to be their priors and/or incentives?
Excellent heuristics and good explanation! I might add:
* Is the mechanism of the causal effect understood, and supported by studies at different levels of analysis, e.g. is a policy intervention also supported by studies of the psychological mechanisms. Or is the medical intervention supported by pharmacology, etc.
* How well do I think the outcome can be measured? As you know, anything that might be analogized to measurement error in the outcome that is truly random error, biases the effect sizes toward zero. Corrections are theoretically possible. Non-random measurement error in the outcome can also be analyzed but let's not worry about that now...
* How did I hear about the study? This heuristic loses value quickly as I learn more about the study, but in general studies I hear about in the mainstream media are less reliable, and the interpretation offered in the media is naturally less reliable than what is written in the research articles.
* Was the study pre-registered? This is a heuristic across all fields, and/but a much stronger heuristic within fields that have some pre-registration and flexbility in how analyses are conducted.
* is it consistent with my prior beliefs? This is sorta a cheat / catch-all... obviously we all use this heuristic, and sometimes over-rely on it. The most interesting and valuable research is that which modifies our prior beliefs, and so despite improving accuracy of beliefs about most research claims, it probably on average harms the accuracy of beliefs about the (far more rare) research claims that people really should be updating the most on.
* What do researchers I respect think of the study claim/evidence? Is their view surprising given what I perceive to be their priors and/or incentives?