Consensus
Consensus provides you with both a quantitative measure of annotation quality (by means of an Inter-Annotator Agreement Score) and the opportunity to create higher-quality annotations by combining the opinions of multiple annotators.
How Does Consensus Work?
With Consensus enabled, multiple annotators can be required to label each Task in the Label Stage. Each individual annotator will only see an empty Task and will not be able to view the annotations done by the other annotators.
Once all the annotators have completed the Task, RedBrick AI will calculate an Inter-Annotator Agreement Score between the annotations. Please reference the following documentation for more information on how we calculate these scores.
The Inter-Annotator Agreement Score is a quantitative measure of quality that can help you select the best set of annotations created by your annotators. It also gives reviewers the ability to arbitrate between the opinions of multiple annotators before generating a single, high-quality Ground Truth.
Enabling Consensus
As of v1.1.2, you must enable Multiple Labeling for your Project upon Project creation.
Once enabled, complete step (4) of Project creation by selecting a minimum number of labelers that will be required to annotate each Task. If your project has a Review Stage, you can enable auto-acceptance to automatically accept Tasks whose agreement scores are higher than the specified threshold.
Assigning Tasks to Multiple Users
RedBrick AI has an Automatic Assignment Protocol that will automatically assign multiple users to a Task. This protocol is enabled by default on Project creation and can be configured either when creating your Project or any time afterward in your Project's General Settings.
As annotators request Tasks by clicking on the Label button in the top right of the Dashboard, RedBrick AI will automatically assign available Tasks by prioritizing those that are already in progress/or assigned to other users.
Alternatively, you can manually override any Task assignment on the Data page. When Consensus is enabled, the Assign dropdown will allow you to select multiple users.
Optionally, you can manually assign any number of labelers to a Consensus Task, including a number that is greater than the minimum number of required labelers. However, the Automatic Assignment Protocol will only assign up to the minimum value.
Inter-Annotator Agreement
Once all assigned annotators have completed a Task, RedBrick AI will generate an Inter-Annotator Agreement Score, which is calculated by comparing each labeler's annotations with those of every other labeler and averaging the pairs of scores.
| User 1 | User 2 | User 3 | |
|---|---|---|---|
User 1 |
|
| |
User 2 |
|
| |
User 3 |
|
|
Agreement = Average(Score(U1,U2),Score(U1,U3),Score(U2,U3)).
The type of comparison function used to calculate the Scoredepends on the type of data and annotations you and your team are working with. Please reference the following documentation to read more about how RedBrick calculates Inter-Annotator Agreement.
Review Stage Absent
If there is no Review Stage after the Label Stage, the set of annotations with the highest Agreement Score (with respect to other annotations) will be selected and stored in Ground Truth. This is the set of annotations that will be exported by default, but you can also export all versions of the annotations.
Review Stage Present
When a Review Stage is present, the annotation sets of all labelers will be displayed in the Annotation Tool.
The users that annotated the Task will be listed in the lefthand toolbar. By default, annotations are color-coded by user, but the Consensus Reviewer can also control the visibility of each Object Label Entity individually.
Best Annotations and Super Truth
The reviewer’s primary task is to analyze the multiple sets of annotations generated by the labelers and produce a single set that will be saved and pushed to the next Stage. In RedBrick AI's Editor, this single set of annotations is referred to as the Best Annotations.
By default, RedBrick AI selects the set of annotations with the highest Inter-Annotator Score as the Best Annotations.
Reviewers can view, hide/show, etc. all sets of annotations in a Task. This allows the reviewer to analyze the work done by the labelers and select the set of annotations that they consider to be of the highest quality.
If a reviewer is satisfied with an existing annotation set, they can simply designate it as Best Annotations and accept the Task.
If a reviewer wishes to make changes to an existing set of annotations or start completely from scratch, they can either click on the Edit button under a user in the right hand panel or click on Create New under Super Truth.
Doing so will create a novel set of annotations known as a Super Truth and automatically designate the set as Best. The reviewer can then annotate the Task as they see fit.
Only Super Truth Annotations can be edited!
All other annotations in the Review Stage are View Only.
Once a reviewer is satisfied with the current Best Annotations, they can accept the Task. This saves the Best Annotations and ascribes only that set to the Task. All other annotations are also saved and are available on export. If the reviewer rejects the Task, all labelers will be required to re-annotate the task.
The video below contains a brief walkthrough of how you can use Consensus in both your Project and the Editor.
Exporting Consensus Annotations
If a task has gone through Consensus, you will get access to all versions of the annotations done by all users. You will also have access to additional metadata like the annotation similarity scores. You can export the data using the following CLI command inside your project directory:
Please view the format reference for an overview of the exported format.
If you want to export only a single version of the annotations (i.e. the labeler with the best annotations or the base annotations qualified in Review), you can run the following command:
Last updated
