Pillar 01 · AI & Data Operations

I learned this work by doing it.

AI models are trained and judged by people: annotating, ranking and evaluating data to a standard. I've worked in that operation for six years: first doing the tasks, then checking them, then leading the teams and running the projects.

Workflow experience

The kinds of work I've run, and what makes each one hard.

I've worked across all of these over my career. Choose a family to see what changes when it runs at scale.

Model evaluation

  • Preference ranking
  • A/B evaluation
  • AI response evaluation

What it is

People judging model outputs side by side or against a rubric, so the model learns what "better" means.

What it takes to run well

Quality here is a matter of judgement, and there is often no answer key. Delivery depends on calibration: reviewers who agree for the right reasons, and a way to spot drift before it reaches the client.

Data modalities

  • Text
  • Image
  • Audio
  • Video
  • 3D
  • Multimodal

What it is

The same operation applied to very different raw material, sometimes several at once.

What it takes to run well

Each modality changes the numbers: handling time, tooling, reviewer fatigue, and how you sample for QA. A plan that works for text will overrun on video.

Annotation

  • Segmentation
  • Classification
  • Video classification
  • Data annotation

What it is

Labelling data precisely, at volume, to a written specification.

What it takes to run well

The gains come from tight guidelines, a living record of edge cases, and audits that feed back into training quickly, not from asking people to go faster.

Content & policy

  • Advertisement evaluation
  • Content moderation

What it is

Decisions made against a policy rather than a ground truth.

What it takes to run well

Consistency and wellbeing carry as much weight as speed: clear interpretation of the policy, a defined escalation path, and teams set up to sustain the work.

How an AI data operation runs

Six stages, and I've worked in every one.

A general picture of the lifecycle, not any client's process. Quality is built into each stage, not checked once at the end.

  1. 01

    Scope

    Volume, quality bar, turnaround and SLAs turned into a capacity plan.

  2. 02

    Train

    Onboarding built with Training and Quality, and kept as short as it can safely be.

  3. 03

    Calibrate

    Reviewers aligned on the hard cases before the work counts.

  4. 04

    Produce

    Daily throughput against plan, with team leads owning their numbers.

  5. 05

    Audit

    Sampling, root cause analysis, and feedback that reaches people within days.

  6. 06

    Report

    One version of the numbers for clients, managers and the team.

Building or scaling an AI data operation?

I'm open to project and delivery roles in AI and data operations, and to selected consulting. The quickest way to reach me is email.