RedBrick AI has comprehensive features to help you record multiple annotations per image or conduct multi-reader studies.

There are two core use cases for multiple labeling on RedBrick AI:

  1. Consensus, or multi-reader, single output: Have multiple labelers annotate a single task and record their inter-annotator agreement scores, i.e., measure the overlap between their annotations. The output of consensus is a single set of ground truth.
  2. Task duplication, or multi-reader, multiple output: Have multiple annotators annotate a single image and generate N unique ground truth records for a single image.

Task duplication vs. consensus