NEJM Medical Board Residency Exams Question Answering Benchmark

Intro

This is the official repository for the benchmark dataset by Katz & Cohen et. al (NEJM-AI, 2023):

GPT versus Resident Physicians — A Benchmark Based on Official Board Scores.

The dataset is composed of 5 official board residency exams, one per core specialty:

  • internal medicine
  • general surgery
  • pediatrics
  • psychiatry
  • obstetrics & gynecology

Each exam contains over 100 multiple choice questions. Some questions admit more than one correct answer.

The official passing rate for these exams was 65%. For physician performance see the the paper.

Example:

Question:

Where is Ghrelin secreted from?
A. From the antrum.
B. From the body of the stomach.
C. From the fundus.
D. From the whole stomach.

Answer:

B,C