Cognitive Modelling Challenge
Currently, there are at least twelve theories describing syllogistic reasoning (Khemlani & Johnson-Laird, 2012). These theories entail predictions which are partly mutually contradicting. Still, there is no established measurement to evaluate the theories in terms of criteria like explanatory power, response prediction and simplicity. Cognitive modelling is a recent field of computer science aiming at formally modelling human's responses to reasoning tasks. From these models, we expect to eventually gain insights into the actual cognitive mechanisms of human cognition.
That is why we have established a benchmark incorporating key results from psychological experiments such as the percentage of the given response. In this year's challenge, we want you to program a model which predicts the response data from human participants. To evaluate different cognitive models several accepted methods from mathematical psychology and artificial intelligence exist. Theory predictions can be modeled by multinomial process trees (e.g., Oberauer 2006; Klauer et al., 2007; Ragni et al., 2014). The model itself can be modeled by using AIC, BIC, and Fishers Information Approximation. These can be with small effort evaluated using packages in R like MPTinR. We will evaluate the models and determine a winner.
- Immediate inferences (Khemlani et al., 2015)
- 64 syllogism problems (Khemlani & Johnson-Laird, 2013)
- Generalized Quantifiers, (second source) (Oaksford & Chater, 2007; Ragni et al., 2014)
- Belief bias (Klauer, 2000)
- Preference effects in relational reasoning (Ragni & Knauff, 2013)
- Pseudo-transitive relations (Goodwin & Johnson-Laird, 2008)
- Complex relations (Goodwin & Johnson-Laird, 2005)
Anderson, J. R. (2007). How can the human mind occur in the physical universe?. Oxford University Press, USA.
Byrne, R. M. (1989). Suppressing valid inferences with conditionals. Cognition, 31(1), 61-83.
Goodwin, G. P., & Johnson-Laird, P. N. (2006). Reasoning about the relations between relations. The Quarterly journal of experimental psychology, 59(6), 1047-1069.
Goodwin, G. P., & Johnson-Laird, P. N. (2008). Transitive and pseudo-transitive inferences. Cognition, 108(2), 320-352.
Klauer, K. C., Stahl, C., & Erdfelder, E. (2007). The abstract selection task: new data and an almost comprehensive model. Journal of Experimental Psychology: Learning, Memory, and Cognition, 33(4), 680.
Oaksford, M., & Chater, N. (2007). Bayesian rationality: The probabilistic approach to human reasoning. Oxford University Press.
- Khemlani, S., & Johnson-Laird, P. N. (2012). Theories of the syllogism: A meta-analysis. Psychological bulletin, 138(3), 427.
- Khemlani, S., Lotstein, M., Trafton, J. G., & Johnson-Laird, P. N. (2015). Immediate inferences from quantified assertions. The Quarterly Journal of Experimental Psychology, 68(10), 2073-2096.
- Klauer, K. C., Musch, J., & Naumer, B. (2000). On belief bias in syllogistic reasoning. Psychological Review, (4), 852-884.
- Ragni, M., Khemlani, S., & Johnson-Laird, P. N. (2014). The evaluation of the consistency of quantified assertions. Memory & cognition, 42(1), 53-66.
Ragni, M., & Knauff, M. (2013). A theory and a computational model of spatial reasoning with preferred mental models. Psychological review, 120(3), 561.