This section of the documentation will cover how RedBrick AI calculates inter-annotator agreement between two users. For two sets of labels, annotation instances are matched up by category. For the same category, instances are matched up by selecting pairs that maximize the overall agreement score. For two instances of the same category, RedBrick AI uses the following similarity functions
RedBrick AI uses IOU for these annotation types. For two annotations A and B IOU is defined by:IOU=A∩BA∪BLandmarksFor landmarks/keypoints, RedBrick AI uses a normalized Root Mean Squared Error (RMSE) to compute similarity, where similarity is Similarity=1−RMSE. MSE=n1i∑n(Pi−P^i)2RMSE=MSEWhere n is the number of components of the point (2 for 2D, 3 for 3D), and Pi,Pi^ are normalized components (by width, height, depth of the image) of the two points.
For angle measurements, the vectors between each arm of the angle measurement are compared. The two angles comparing both sets of measurement arms are computed. The similarity score is then defined by:Similarity=1−2πθ1+θ2Where θ1,θ2 are the angles between the two sets of measurement arms.
For classification labels, the agreement is binary. If the chosen category and attributes match, the consensus score will be 100%, otherwise, it will be 0%.