You are here: Home Modeling Challenge

Bridging the Gap 2019

The goal of the proposed challenge is to extend the fundamental modeling problem underlying cognitive research: predicting human behavior. It relies on the Cognitive COmputation for Behavioral Reasoning Analysis (CCOBRA) Framework, a benchmarking tool implemented in Python that actively integrates the individual human into the prediction loop. Focusing on the domain of syllogistic reasoning, one of the cornerstones of human reasoning research, the challenge invites modelers from cognitive science and artificial intelligence alike to submit their attempts at capturing the processes underlying the human mind.

 

Modeling Task

  • Given: Formalized representation of the underlying experiment (task, response options)
  • Goal: Predict the human response

 

CCOBRA Modeling Framework

  • Classical Cognitive Modeling:
    • Training Phase: Extract patterns from human behavioural data
    • Test Phase: Use patterns to generate predictions for unseen tasks
  • CCOBRA introduces a dynamic individual prediction loop to the test phase:
    • Present the reasoning task
    • Predict the human response (accuracy as performance metric)
    • Adapt own behaviour based on true response
  • Core Ideas:
    • The individual prediction loop simulates the experiment as run by a single individual reasoner
    • The model is supposed to generate its prediction under the same conditions as the human participant
      • Same textual representation of the task
      • Same sequence of tasks
      • Additional information about the human performance (true responses, response times, etc.)

 

Syllogistic Reasoning

  • Quantified categorical assertions ("All A are B; All B are C; what follows?")
  • Two premises
  • Four figures
  • 64 distinct syllogistic problems
  • 8 quantified response options + No Valid Conclusion (NVC)
  • Khemlani, 2012 for additional information

 

Participation

  • Implement a model in Python using the CCOBRA framework
  • Test the model either locally or via the online evaluation platform
  • Send us your final model submission by May 15th, 2019
  • Submission guidelines:
    • Zip archive containing your model
    • We only accept zip archives which can be evaluated successfully on the online evaluation platform

 

Material

 

Contact

In case of questions or remarks, feel free to contact