Two reviewers can watch the same robot demonstration and disagree about whether the intended action succeeded. A data dashboard may summarize their agreement, but the training team still needs to decide which annotation belongs in the prepared collection. The database selection question is whether the workflow preserves the evidence and decision behind that label instead of retaining only a convenient final value.
Agreement measures how judgments relate; an adjudication record explains what the project decided to use. Neither should be presented as automatic proof that the chosen annotation is correct.
Name the judgment that needs resolution
Consider a hypothetical demonstration in which an object briefly reaches the target area and then slips away. One annotator labels the attempt successful at the first visible contact. Another uses the final state and labels it unsuccessful. The immediate issue may be the success definition rather than careless labeling or a faulty database.
The project should identify what evidence the reviewers were allowed to inspect and which written rule applied. A disagreement about the endpoint of an episode is different from a disagreement about what is visible in the same frame. Keep those distinctions in the review request so the person resolving it knows which question actually needs an answer.
Keep the agreement measure identifiable
Label Studio Enterprise’s task-agreement documentation distinguishes agreement on individual control tags from an overall task score. It also describes different calculation methodologies, including consensus and pairwise comparison. A recorded score therefore needs its measurement context; the label “agreement” alone is not a complete definition.
For the proposed project record, retain which field was compared, the relevant metric or method, and the rule version used for the judgment. Do not interpret a higher number as a substitute for examining the disputed evidence. Equally, a low agreement score should trigger an appropriate investigation rather than automatically disqualifying a person or every annotation in the collection.
Separate submissions from the resolved label
Store the annotators’ submissions as inputs to the decision. The resolution should identify the selected interpretation, its reviewer, and the basis for applying it to the prepared data. This is a suggested workflow model, not a claim about a particular annotation tool’s default storage behavior.
| Information | Why keep it separately? | Mistake it helps avoid |
|---|---|---|
| Original judgments | Show the actual disagreement | Making alternatives disappear |
| Applicable rule | Define what reviewers were asked | Mixing different success definitions |
| Agreement context | Explain what the score measures | Treating unlike scores as equivalent |
| Resolution | Identify the annotation chosen for use | Assuming majority selection proves truth |
| Prepared collection | Show where that resolution was used | Believing every export changed automatically |
The record can be compact. Its value comes from preserving the relationship between the inputs and the chosen label, not from collecting an elaborate narrative for every straightforward example.
Revisit labels when the rule changes
Suppose the hypothetical project later requires the object to remain in the target area until the demonstration ends. An earlier resolution under a contact-based rule should not silently acquire the new meaning. Decide which existing annotations need review and which prepared collections still represent the previous definition.
This does not require treating every historical judgment as a mistake. It requires an explicit change in the question being answered. A researcher comparing two training runs should be able to discover that their success labels reflect different definitions before attributing the difference entirely to the model or database. Preserve unresolved cases as such until someone actually applies the revised rule.
Evaluate OceanBase on the review relationships
OceanBase Database is a relational database candidate for connecting episodes, annotation submissions, rule versions, resolutions, and dataset selections. PostgreSQL and MySQL can also serve as options for this metadata application. The comparison should examine how each proposed implementation represents these relationships and makes them usable to the review team.
An OceanBase deployment would still need the annotation interface, appropriate access controls, and a collection-building process that reads the intended resolution. A database does not choose the correct interpretation of a video simply by storing the associated rows. Ask the implementer to demonstrate a disputed episode through the complete review path rather than scoring the product on an annotation count alone.
Follow one resolved case into a collection
A useful proposed acceptance exercise starts with the two conflicting judgments, resolves them under an identified rule, and follows the result into a specific prepared collection. Then change the rule and show which decision remains historical and which request is open for review. No actual product or labeling test is reported here.
The result should let a project explain why it used a label, what evidence remained uncertain, and which experiment consumed that choice. This is a different task from calculating agreement. Keeping both tasks visible supports a more honest training-data workflow than presenting a single final label or a single quality score as the entire answer.
This article is based on documented sources and editorial analysis. Proposed evaluation examples are not independently measured benchmark results.