> ## Documentation Index
> Fetch the complete documentation index at: https://docs.redbrickai.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Custom Label Validation
You can define a custom JavaScript script that continuously compares annotations to a schema/set of rules and informs you (in real-time) of any mistakes in your annotations. You can also prevent labelers from submitting tasks when your validation script finds errors.
For most annotation projects, there is a schema/rule-set/taxonomy that labelers must follow. A large portion of errors in annotation projects is due to oversights/slips during labeling in adhering to the schema. \
\
For example, the labeler must segment a tumor and fill out a few related attributes if a tumor is found. A common error can be forgetting to fill out all attributes when the tumor is present. Post-processing scripts usually reveal these errors.
Prevent simple, recurrent errors from occurring by writing a set of tests that will be run regularly, informing labelers of any mistakes they're making.
## Overview
By default, all projects have a custom check to warn labelers when they submit tasks without any annotations. You can enable/disable custom validation under Project Settings -> Label Validation.
### Prevent Submissions with Errors
By default, labelers will just receive the error messages as a warning, and they will still be able to submit the task anyway. To prevent the labelers from submitting with any errors present, toggle the *Prevent submission with errors* switch.
### Custom JavaScript Function
You will write the custom validation as a JavaScript function. This JavaScript function runs on each labeler's browser while they are annotating data. \
\
The Javascript function has the following definition:
```typescript theme={null}
function(task: Task, labels: Label[]): string[] {
// Your custom validation logic
assert(false, "This assertion was false");
}
```
#### `label: Label[]`
The validation function has a single input - a list of labels containing minimal metadata about the labels. Please see the definition of the `Label` object below:
```typescript theme={null}
interface Label {
category: string[];
attributes: LabelAttribute[];
labelType: TaskType;
numFramesLabeled?: number;
instanceTracks?: { [name: string]: FrameState[] };
seriesIndex?: number;
}
// Label attribute
interface LabelAttribute {
name: string;
value: boolean | number | string;
}
// Task Type
enum TaskType {
ITEMS = 'ITEMS',
CLASSIFY = 'CLASSIFY',
BBOX = 'BBOX',
POLYGON = 'POLYGON',
POLYLINE = 'POLYLINE',
POINT = 'POINT',
ELLIPSE = 'ELLIPSE',
SEGMENTATION = 'SEGMENTATION',
MULTI = 'MULTI',
MULTICLASSIFY = 'MULTICLASSIFY',
LENGTH = 'LENGTH',
ANGLE = 'ANGLE',
}
interface Task {
orgId: string;
projectId: string;
stageName: string; // i.e. "Label" or "Review_1"
taskId: string;
name: string; // Name given for the task at upload
metaData: Record ;
classification?: Classification;
series: Series[];
}
interface Series {
name: string;
metaData: Record ;
dimensions: [number, number, number];
classifications?: Classification[];
instanceClassifications?: InstanceClassification[];
landmarks?: Landmark[];
landmarks3d?: Landmark3D[];
measurements?: (MeasureLength | MeasureAngle)[];
boundingBoxes?: BoundingBox[];
cuboids?: Cuboid[];
ellipses?: Ellipse[];
polygons?: Polygon[];
polylines?: Polyline[];
segmentMap?: {
[instanceId: string]: {
category: Category;
attributes?: Attributes;
overlappingGroups?: number[];
};
};
}
interface VideoMetaData {
frameIndex: number;
trackId: string;
keyFrame: boolean;
endTrack: boolean;
}
interface Classification {
attributes: Attributes;
video?: VideoMetaData;
group?: string;
}
interface InstanceClassification {
fileIndex: number;
values: Attributes;
group?: string;
}
interface MeasurementStats {
average?: number;
area?: number;
volume?: number;
minimum?: number;
maximum?: number;
}
interface Landmark {
point: Point2D;
category: Category;
attributes?: Attributes;
video?: VideoMetaData;
group?: string;
}
interface Landmark3D {
point: VoxelPoint;
category: Category;
attributes?: Attributes;
group?: string;
}
interface MeasureLength {
type: 'length';
point1: VoxelPoint;
point2: VoxelPoint;
absolutePoint1: WorldPoint;
absolutePoint2: WorldPoint;
normal: [number, number, number];
length: number;
category: Category;
attributes?: Attributes;
group?: string;
}
interface MeasureAngle {
type: 'angle';
point1: VoxelPoint;
point2: VoxelPoint;
vertex: VoxelPoint;
absolutePoint1: WorldPoint;
absolutePoint2: WorldPoint;
absoluteVertex: WorldPoint;
normal: [number, number, number];
angle: number;
category: Category;
attributes?: Attributes;
group?: string;
}
interface BoundingBox {
pointTopLeft: Point2D;
wNorm: number;
hNorm: number;
category: Category;
attributes?: Attributes;
stats?: MeasurementStats;
video?: VideoMetaData;
group?: string;
}
interface Cuboid {
point1: VoxelPoint;
point2: VoxelPoint;
absolutePoint1: WorldPoint;
absolutePoint2: WorldPoint;
category: Category;
attributes?: Attributes;
stats?: MeasurementStats;
group?: string;
}
interface Ellipse {
pointCenter: Point2D;
xRadiusNorm: number;
yRadiusNorm: number;
rotationRad: number;
category: Category;
attributes?: Attributes;
stats?: MeasurementStats;
video?: VideoMetaData;
group?: string;
}
interface Polygon {
points: Point2D[];
category: Category;
attributes?: Attributes;
stats?: MeasurementStats;
video?: VideoMetaData;
group?: string;
}
interface Polyline {
points: Point2D[];
category: Category;
attributes?: Attributes;
video?: VideoMetaData;
group?: string;
}
// i is rows, j is columns, k is slice
interface VoxelPoint {
i: number;
j: number;
k: number;
}
// The position of VoxelPoint in physical space (world coordinate) computed using the Image Plane Module
interface WorldPoint {
x: number;
y: number;
z: number;
}
interface Point2D {
xNorm: number;
yNorm: number;
}
type Category = string;
type Attributes = { [attributeName: string]: string | boolean | string[] };
```
You can generate a sample `Label[]` object by going to the labeling tool -> opening command bar `cmd/ctrl + k` -> *Copy current label state to clipboard.*
Your custom validation script must return a list of warning/error messages describing the issues found. Return an empty array `[]` if no errors are found. These error message strings will be displayed to the labelers on the labeling tool.
To help you write a validation function with several checks, RedBrick AI has a custom-defined function `assert` that accepts a boolean statement and a corresponding error message. The two scripts below will produce the same result:
```typescript theme={null}
function(task: Task, labels: Label[]): string[] {
assert(labels.length >= 1, "You have not created any labels!");
assert(labels.length <= 5, "You have created too many labels!");
}
```
```typescript theme={null}
function(task: Task, labels: Label[]): string[] {
const errors = [];
if (labels.length < 1) {
errors.push("You have not created any labels!");
}
if (labels.length > 5) {
errors.push("You have created too many labels!");
}
return errors;
}
```
## Validate Your Code
Before saving your script, you should validate that your code executes as expected. Click on the validate button on the bottom right of the Settings page, and paste the JSON object copied from the labeling tool to see if your code executes as expected:
## Displaying the Validation on the Labeling Tool
Your custom validation script will be regularly run. If any warnings are found, an indicator will appear on the right side of the bottom bar. If you have enabled **Prevent submissions with errors**, the indicator will be red.
## Example Scripts
### Check if Exact Categories are Present
For this example, let's say we are expecting each task to contain the following segmentations - *necrosis, enhancing tumor, non-enhancing tumor and edema.*
```typescript theme={null}
function(task: Task, labels: Label[]): string[] {
const expectedCategories = [
'necrosis',
'enhancing tumor',
'non-enhancing tumor',
'edema',
];
// Iterate over all expected categories
for (const category of expectedCategories) {
// Check if the category is present in any label
const categoryPresent = labels.some(
(label) => label.category[0] === category
);
// assert with message
assert(categoryPresent, `The ${category} category is missing!`);
}
}
```
### Validate Only Single Instance of a Category Has Been Created
This script validates only a single instance of a particular category has been created. If you're expecting semantic segmentation labels, this check can ensure labelers don't accidentally create multiple instance segmentations.
```typescript theme={null}
function(task: Task, labels: Label[]): string[] {
const semanticCategory = 'edema';
const labelsFiltered = labels.filter((label) => label.category[0] === semanticCategory);
assert(
labelsFiltered.length === 1,
`Expected exactly 1 ${semanticCategory} annotation`
);
}
```
### Verify That Specific Segmentation Type Is Visible on Specific Series
The following script examines the Series Identifier and verifies whether a specific Segmentation type is present on it. In this example, you could use this script to be sure that labelers cannot finalize a Series that ends in "DWI" (a common naming convention for DWI images) without including an "Infarct" segmentation on the Series.
More broadly speaking, this script is an example of the extensive functionality available when combining the `label`, `task`, and `series` objects.
```javascript theme={null}
function (task: Task, labels: Label[]): string[] {
assert(labels.length > 0, "You haven't created any labels! Are you sure you want to submit?");
for (let label of labels) {
if (label.category[0] === "Infarct") {
assert(
task.series[label.seriesIndex].name.startsWith("DWI_"),
"Segmentation 'Infarct' is allowed only on 'DWI' images"
);
}
}
}
```