Quick answer: data labeling is the process of attaching tags, categories, or other ground-truth answers to raw data so a supervised machine learning model has something correct to learn from. A photo gets tagged “cat,” a support ticket gets tagged “billing issue,” a pair of chatbot replies gets ranked better or worse. Someone or something has to do that tagging before the data is useful for training, and for most of the industry’s history that someone has been a person.
So what does data labeling actually involve, who does it, and where does it break down? This entry covers the mechanics of labeling, the human workforce behind it, how labeling looks different for RLHF, how labeling differs from training data more broadly, and the quality problems that come with the job.
What Is Data Labeling?
Data labeling is the step that turns raw, unstructured material into something a supervised model can learn from. A model does not know on its own that a particular photo shows a pedestrian or that a particular sentence is spam; a label tells it that, and the model adjusts itself to match as many labels as it can.
Labels take different shapes depending on the task. Image work uses bounding boxes, segmentation masks, or a simple category tag, and text work uses category tags, sentiment scores, or full transcriptions.
Audio gets transcribed and sometimes tagged for speaker and emotion. A self-driving system needs its camera and lidar recordings labeled frame by frame, so it can learn what a pedestrian or a stopped truck looks like from every sensor it carries.
Newer systems add a format that does not fit the old tagging model at all: rankings, where a person orders several model answers from best to worst instead of marking a single fixed answer as correct. That shift matters enough that it gets its own section below.
How Does Data Labeling Work?
In practice, a team collects raw examples, writes guidelines describing exactly what each label means, and hands both to annotators working inside a labeling tool built for the task, a bounding-box drawer for images or a ranking interface for text. The annotators apply labels example by example, and disagreements get reviewed against the guidelines.
ImageNet, the dataset that helped kick off the deep learning boom in computer vision, was built this way at enormous scale: researchers used Amazon Mechanical Turk to crowdsource the labeling of millions of images across tens of thousands of categories. Commercial platforms today, including Amazon SageMaker Ground Truth and Scale AI, package the same idea into managed workflows that combine software, a paid workforce, and increasingly a layer of automation that pre-labels easy examples and leaves the hard ones for people.
Programmatic labeling is the automated end of that spectrum. Instead of a person tagging every example, a script or a weaker existing model applies labels based on rules or patterns, trading some accuracy for a lot of speed on huge datasets.
Quality control is built into the process rather than bolted on afterward. A serious labeling pipeline routes the same example to more than one annotator and checks whether they agree, and disagreements get escalated to a reviewer rather than resolved by whichever annotator saw it first.
Who Actually Labels the Data?
Behind most labeled datasets is a paid workforce, and a lot of that work is outsourced to firms operating in lower-wage countries. A Time investigation published in January 2023 reported that OpenAI had contracted with Sama, an outsourcing firm, to have workers in Kenya label text describing violence, self-harm, and sexual abuse, so that a safety system could be trained to detect and screen out that kind of content.
The reported take-home pay ran between roughly $1.32 and $2 an hour depending on seniority and performance, and the workers interviewed described being mentally scarred by the material they read. Sama ended the work in February 2022, eight months earlier than the contract had planned.
That case became well known, but the underlying arrangement is common. Firms like Sama, Appen, and Scale AI employ or contract distributed teams around the world specifically to do the labeling work that large AI developers do not do in-house, and the pay and conditions on that work vary a great deal by client and country.
The arrangement exists because labeling at the scale modern AI needs is genuinely expensive to do any other way. Hiring specialists directly does not scale to millions of examples, and outsourcing lets a company turn labeling capacity up or down as a project needs it, which is also why quality between projects and vendors can look so different.
What Is RLHF Labeling?
Reinforcement learning from human feedback, or RLHF, uses a different kind of labeling: instead of tagging a single correct answer, people rank multiple model responses to the same prompt from best to worst. Those rankings train a separate reward model, which then guides further training of the main model toward answers people preferred.
OpenAI’s work on InstructGPT is a documented example. The paper describes a team of about 40 contractors hired on Upwork and through Scale AI, screened first for how they handled sensitive content, who wrote example answers to prompts and ranked competing model outputs against each other.
That comparison data became the training signal behind fine-tuning a base model into a chat assistant that follows instructions.
Data Labeling vs Training Data: What’s the Difference?
The two terms get used loosely, but they are not the same thing. Training data is the entire set of examples a model learns from, labeled or not; a large language model pretrains on raw text with no labels attached at all. Data labeling is specifically the work of adding ground-truth tags to examples so supervised learning is possible in the first place.
Put another way, labeling is one way training data gets made, not the only way. It sits alongside feature engineering, which shapes the inputs a model sees rather than the answers it is trying to match.
What Are the Limits of Data Labeling?
Labeling quality is uneven even on datasets the field treats as standard. A 2021 study of label errors across ten widely used benchmark test sets estimated that at least 6 percent of ImageNet’s validation set carries an incorrect label.
Human labelers also bring their own inconsistency into a dataset, and specialized domains make this worse; a labeler without medical training will mislabel scans that a radiologist would not.
Supervised learning has no other way to get its answer key, so none of that is a reason to skip labeling. It is a reason to treat a labeled dataset as an approximation of the truth, produced by particular people working to particular guidelines, rather than as the truth itself.