AI Data Annotation Work

Initial Setup Checklist

1. Read the full annotation guideline document before labelling a single item—not during labelling.
2. Complete the client's required sample batch or qualification test to confirm you understand the taxonomy before accessing the main dataset.
3. Label a small pilot batch of 10–20 items and review it for consistency before working through the full batch.
4. Maintain a disagreement log throughout: any item where you are uncertain about the correct label goes in the log with your reasoning.
5. Deliver in the specified output format with a quality summary noting disagreement rate and escalated items.

Action Prompts

Finished Example

Common Mistakes

• Guessing on borderline items instead of escalating

  Impact: A consistent guessing pattern on edge cases introduces systematic bias into the training data, which surfaces as model errors at scale—far downstream from your batch.

  Correction: Maintain a disagreement log. Any item where you cannot point to a specific guideline rule that resolves the label goes in the log and gets escalated, not guessed.

• Increasing labelling speed by skipping the full-read of each item

  Impact: Rushing through items produces a high inter-annotator disagreement rate, which triggers quality review and potential batch rejection on gated platforms.

  Correction: Set a sustainable pace that allows you to read each item fully. Throughput matters only if accuracy is maintained—rejected batches have zero value.

• Inventing productivity or income projections you have not personally measured

  Impact: Decisions about how much time to invest in annotation work are made on false data, leading to frustration when actual throughput and availability differ from projected figures.

  Correction: Track your actual labelled items per hour on a real batch before estimating how much work you can take on. Do not plan based on unverified productivity claims from forums or advertisements.

Practice Execution

1. Find a publicly available annotation guideline or create a simple one for a classification task you understand well (e.g., topic classification for news headlines).
2. Use Prompt 1 to confirm your understanding of the guideline before labelling anything.
3. Label 20 sample items using the guideline strictly. Do not deviate, even for cases where you personally disagree with the rule.
4. Identify two borderline cases in your batch and log them using Prompt 2.
5. Review your 20 labels and check: would you assign the same label if you saw the same item again in two hours?
6. Format your output as a CSV with item ID, text, label, and notes columns.