You are here: Home Modeling Challenge

The PRECORE Challenge: Predicting Individual Syllogistic Reasoning

 

Send us an email and we will keep you notified about updates related to the challenge:

 

The goal of the proposed challenge is to extend the fundamental modeling problem underlying cognitive research: predicting human behavior. It relies on the Cognitive COmputation for Behavioral Reasoning Analysis (CCOBRA) Framework, a benchmarking tool implemented in Python that actively integrates the individual human into the prediction loop. Focusing on the domain of syllogistic reasoning, one of the cornerstones of human reasoning research, the challenge invites modelers from cognitive science and artificial intelligence alike to submit their attempts at capturing the processes underlying the human mind.

 

Modeling Task

  • Given: Syllogistic reasoning problems represented as lists of quantifiers and terms, e.g.,  [[Some, models, managers], [All, models, clerks]]. Possible quantifiers are [All, Some, No, Some ... not]. Premises are always constructed from a single quantifier linking two distinct terms.
  • Predict: Human response, e.g., [Some, managers, clerks].
  • Goal: Optimize for predictive accuracy.
  • Evaluation: Model implementations are evaluated using the CCOBRA framework. See the syllogistic benchmark for example models and training data.

 

Current Best Predictive Models

 

Evaluation Framework

 

Participation

  • Implement a model in Python using the CCOBRA framework
  • Test the model either locally or via the online evaluation platform
  • Send us your final model submission by May 15th, 2019
  • Submission guidelines:
    • Zip archive containing your model
    • We only accept zip archives which can be evaluated successfully on the online evaluation platform

 

CCOBRA Modeling Framework

  • Classical Cognitive Modeling:
    • Training Phase: Extract patterns from human behavioural data
    • Test Phase: Use patterns to generate predictions for unseen tasks
  • CCOBRA introduces a dynamic individual prediction loop to the test phase:
    • Present the reasoning task
    • Predict the human response (accuracy as performance metric)
    • Adapt own behaviour based on true response
  • Core Ideas:
    • The individual prediction loop simulates the experiment as run by a single individual reasoner
    • The model is supposed to generate its prediction under the same conditions as the human participant
      • Same textual representation of the task
      • Same sequence of tasks
      • Additional information about the human performance (true responses, response times, etc.)

 

Syllogistic Reasoning

  • Quantified categorical assertions ("All A are B; All B are C; what follows?")
  • Two premises
  • Four figures
  • 64 distinct syllogistic problems
  • 8 quantified response options + No Valid Conclusion (NVC)
  • See Khemlani & Johnson-Laird (2012), "Theories of the syllogism: A meta-analysis", for additional information

 

Contact

In case of questions or remarks, feel free to contact