The Lineage of Image Semantic Learning and Data-Driven/Data-Centric AI

Artificial intelligence has repeatedly changed its answer to a deceptively simple question: Where does intelligence in a learning system come from?

For much of the history of pattern recognition, the answer emphasized algorithms and representations. Researchers designed better features, better classifiers, better probabilistic models, and better optimization procedures. Data were indispensable, of course, but they were often treated as a relatively fixed input to the learning process.

Over time, another idea became increasingly important:

Perhaps progress does not come only from designing a better learning algorithm. Perhaps it comes from obtaining more data, better data, better labels, broader coverage—and eventually from allowing the learning system to identify weaknesses in its own data.

Today we use terms such as data-driven AI and data-centric AI to describe parts of this transition. But those terms compress several historically distinct ideas. Large-scale data are not the same thing as data-driven learning. Data-driven learning is not the same thing as data-centric AI. And building a large labeled dataset is not the same thing as designing a system in which the training data themselves can be diagnosed and improved.

The history is therefore more interesting than a single invention or date.

One particularly revealing place to reconstruct that history is image semantic learning: the effort to teach machines not merely to compare pixels, colors, and textures, but to associate visual information with human concepts such as tiger, forest, building, sunset, or Eiffel Tower.

This article begins an investigation of that lineage.

ImageNet deserves its place in history. But the ideas that made large-scale visual learning possible were developed over decades by researchers working on different pieces of the problem. Before ImageNet, researchers were already asking how to represent texture, color, shape, and local structure; how to bridge the semantic gap; how to infer a human query concept from feedback; how to annotate images automatically; how to organize concepts with lexical ontologies; and how to collect and label visual data at Web scale.

Three contemporaneous surveys are especially useful maps of this pre-ImageNet landscape: Rui, Huang, and Shih-Fu Chang's 1999 survey covered more than one hundred papers in content-based retrieval; Smeulders et al. surveyed roughly two hundred references in 2000 and made the semantic gap a central organizing problem; and Datta et al. surveyed almost three hundred contributions in image retrieval and automatic annotation in 2008. These surveys make clear that ImageNet arrived into an already deep, diverse field rather than an empty one.

The point is not to replace one creation myth with another. It is to recover the converging streams—and the people behind them—that gradually connected perception, semantics, interaction, annotation, and scale.

From Pixels to Meaning

Early content-based image retrieval systems exposed a fundamental problem that would remain with AI for decades: machines and humans did not naturally describe images in the same language.

A computer could calculate color histograms, textures, shapes, edges, and regions. A person wanted to ask for “a tiger in the forest” or “a sunset over the ocean.”

This discrepancy became widely known as the semantic gap.

One response was to engineer increasingly sophisticated visual features. Another was more consequential: learn the relationship between perceptual features and semantic concepts from examples.

That change sounds obvious today. It was not obvious when image retrieval was dominated by carefully designed visual representations and similarity metrics.

The representation side itself had a long lineage. Haralick, Shanmugam, and Dinstein formalized statistical texture features in 1973; Tamura, Mori, and Yamawaki explicitly connected computable texture measures to visual perception in 1978; Mallat's 1989 wavelet theory provided a multiresolution language for image structure; Swain and Ballard's 1991 color indexing made color histograms a practical matching representation; and Lowe's 1999/2004 SIFT work established highly distinctive local features robust to scale and rotation. These methods did not solve semantics, but they made images progressively more learnable.

Once semantic mapping becomes a learned function, the training examples become part of the machinery that determines what the system knows. That seemingly modest change creates a path toward data-driven AI.

1. Active Learning: Learning a Human Query Concept Without Words

A separate thread, also in 2001, attacked the semantic gap from a different direction. Instead of asking the user to formulate a semantic query in words, Support Vector Machine Active Learning for Image Retrieval asked the user only for binary judgments: relevant or irrelevant. The system then learned the user's latent query concept from those examples.

The distinction from much traditional relevance feedback of the period is important. Query-point movement, query reweighting, and related similarity-based schemes generally presented images that were already close to the system's current estimate of the query. If the first relevant examples were red flowers, such a system was likely to show more images resembling red flowers. That is useful exploitation, but it can explore the semantic boundary slowly.

SVMActive deliberately queried a different kind of example: an unlabeled image close to the current SVM decision boundary—the image whose semantic membership was most uncertain under the current classifier. The user might therefore be shown a yellow rose, a red tulip, a flower against a forest, or a red object that was not a flower. The system did not literally ask, “Do you mean roses only?” or “Does the background matter?” But the boundary example turned the image itself into an implicit semantic question.

SVMActive changed relevance feedback from “show me more of what I already think you mean” toward “show me the example whose label would most reduce my uncertainty about what you mean.”

Why the kernel matters

The published system represented each image by a 144-dimensional vector of hand-designed perceptual measurements. It then used a radial-basis-function (RBF) kernel. The kernel trick implicitly maps the 144-dimensional descriptor into a much richer reproducing-kernel Hilbert space (RKHS). For the standard Gaussian RBF interpretation, that RKHS is infinite-dimensional; the important historical point here is that classification is performed through a nonlinear kernel representation rather than a simple linear boundary in the original descriptor coordinates. The system never needs to enumerate those coordinates explicitly.

At feedback round t, the kernel SVM learns a maximum-margin separator:

The active query is then the unlabeled image with the smallest absolute decision value:

After the user labels that boundary case, the labeled set changes, the SVM is retrained, support vectors and coefficients may change, and the maximum-margin hyperplane moves. When mapped back into the original 144-dimensional descriptor space, that moving hyperplane corresponds to a potentially complicated nonlinear semantic boundary.

This distinction matters. It would be inaccurate to say that active learning itself continually reshapes a fixed Gaussian RKHS. A more precise statement is:

The 2001 paper motivates its uncertainty rule through version-space geometry: each labeled example constrains the set of separators still consistent with the observed judgments, and an informative query should cut away as much of that uncertainty as possible. Its formal version-space lemma is stated for a finite-dimensional feature space; the operational RBF-SVM rule is the practical boundary-distance approximation. Thus, for the infinite-dimensional Gaussian-RBF interpretation, version-space contraction is best read as the geometric motivation for the query rule rather than as a new theorem about the entire infinite-dimensional RKHS.

This should also be distinguished from important earlier semantic-gap work. Rui et al. (1998) used relevance feedback to adapt feature weights to a user's high-level need over a database exceeding 70,000 images, while Chang, Chen, and Sundaram's Semantic Visual Templates (1998) represented personalized concepts with successful exemplar queries. SVMActive's distinctive move was to make uncertainty itself the query policy: instead of primarily presenting top-ranked or neighborhood examples, it selected boundary cases expected to be most informative about the user's latent concept.

Traditional relevance feedback versus kernel-space active learning

The scientific combination is therefore more specific than either kernel learning or active learning alone:

2. Learning Semantic Annotation from Examples

Our 2003 CBSA work—Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machines—provides one useful historical marker.

The problem was not simply how to retrieve visually similar images. The objective was to start with a relatively small collection of manually labeled examples, learn classifiers from them, and then propagate semantic labels to the larger unlabeled collection.

This is an important transition. Humans no longer need to write down the rule that distinguishes a forest from an animal or a building. They provide examples, and the statistical learner attempts to discover the decision boundary.

The paper tested the approach on 25,000 images across 116 semantic categories, substantial scale for an image-semantic experiment at the time. More revealing for the history of data-driven learning, the experiment explicitly varied the amount of training data.

Using 20%, 30%, and 50% of the images for training produced annotation accuracies of approximately 49%, 56%, and 61%, respectively.

At roughly the same historical moment, results in natural-language processing were making the importance of scale increasingly difficult to ignore. Banko and Brill's 2001 ACL paper showed several learning curves continuing to improve as the corpus grew from millions toward roughly a billion words.

These developments deserve to be studied together rather than retrospectively assigning one field a privileged starting point. Something was changing across machine learning: data quantity itself was becoming an experimental variable.

3. But More Data Are Not Necessarily Better Data

Increasing the size of L, the training set, immediately raises a harder question:

Which data are missing?

This leads to a second stage in the lineage.

The CDE program appeared first at ACM Multimedia 2003 in Confidence-Based Dynamic Ensemble for Image Annotation and Semantics Discovery and was developed more fully in the 2005 ACM TOMM paper Semantics and Feature Discovery via Confidence-Based Ensemble. It began with an unusual criticism of conventional classifiers. It argued that learning systems commonly assumed three components to be fixed:

That formulation is interesting when viewed from 2026. Instead of assuming that an error necessarily means that the classifier needs improvement, the system asks what produced the error.

A low-confidence prediction could indicate an inadequate semantic vocabulary, an inadequate feature representation, or inadequate training data. The remedies correspondingly modify C, P, or L. In particular, when training data are under-representative, the prescribed remedy is to add the problematic instance to the training set.

That distinction deserves attention in histories of data-centric AI.

4. From a Large Dataset to a Mutable Dataset

There is a fundamental conceptual difference between saying:

Train on more data.

and saying:

Use model failures to discover which training data are missing.

The first is primarily a scaling strategy. The second makes the dataset an adaptive component of the learning system.

The CDE work illustrates this with a lighthouse photograph. The system mistakes the image for a wave scene because the existing training set poorly represents lighthouses. The analysis considers whether the appropriate repair is to add examples to an existing semantic category or create a new semantic category, with the ontology helping determine which intervention is appropriate.

In modern language, we might describe aspects of this problem using terms such as dataset coverage, hard examples, underrepresented slices, data acquisition, label refinement, or dataset repair. The terminology was different in 2005.

This is considerably closer to what is now called data-centric AI than simply training an algorithm on a large dataset. The paper itself distinguishes the approach from previous annotation methods that held C, P, and L fixed.

Whether this represents an early instance, a precursor, or part of a broader contemporary movement in data-centric learning is a historical question that requires a much wider literature investigation. That investigation is worth doing.

5. Data Alone Were Not Enough

There was another problem: an image does not exist in isolation.

Consider a photograph containing a tower. Pixels provide one source of evidence. But so might the location where the photograph was taken, the time, camera parameters, information about the photographer, and relationships among semantic concepts.

Our 2005 EXTENT work therefore took a different direction.

EXTENT combined these sources using an influence-diagram framework. The paper describes contextual variables including location, time, and camera parameters; perceptual information from the photograph; and semantic relationships among labels.

Importantly, this was neither purely hand-coded knowledge nor purely statistical learning. The model relied on both prior knowledge and data, with relationships between context/content and semantic labels and their strengths learned from observations.

In the landmark experiment, approximately 14,530 images were assembled from Stanford photographs, additional locally collected images, and 13,500 landmark images downloaded from the Internet. For the three-landmark experiment, location alone yielded about 30% recognition and SIFT visual features about 75%. Combining location and SIFT increased recognition to approximately 95%.

The lesson here was not simply more data. It was that the meaning of an observation depends on multiple sources of evidence and their relationships.

This thread would eventually intersect with ideas now described as multimodal learning, structured knowledge, context-aware AI, causal reasoning, and grounding.

6. Then Came the Era of Dataset Scale

By the late 2000s, several complementary projects had made scale itself part of the visual-learning agenda. WordNet supplied a lexical ontology that could organize concepts; WebSEEk had demonstrated semantic organization of large Web image/video collections; LabelMe turned Web-based human annotation into shared research infrastructure; 80 Million Tiny Images demonstrated the power and messiness of Internet-scale visual collection; and distributed learning systems were beginning to make large probabilistic models practical. These were different answers to a common question: how much of the visual world can we turn into learnable, organized data?

The next historical transition requires careful investigation.

During the 2000s, computer vision researchers increasingly recognized that the diversity of the visual world could not be captured by small, carefully curated datasets. The Web changed what was possible. Images existed in unprecedented quantities. Search engines provided partial semantic organization. Online communities generated tags and metadata. Distributed human labeling became practical. Computational infrastructure made increasingly large learning experiments possible.

Eventually, ImageNet would become the defining symbol of this transition.

Its importance should not be understated. ImageNet did something different from the earlier systems discussed above: it constructed an extraordinarily large, systematically organized labeled visual resource, using WordNet's semantic hierarchy as its organizational foundation.

And once large datasets met sufficiently powerful learning algorithms and computation, the consequences became historic.

The 2012 ImageNet breakthrough of deep convolutional networks did not merely improve image classification. It helped change the dominant methodology of artificial intelligence.

But the familiar shorthand—

—can obscure the longer intellectual lineage. Long before 2012, researchers were already asking:

• Can semantic labels be learned from visual examples?
• How does performance change as training data increase?
• Can labels be propagated automatically?
• How should ambiguous or underrepresented training examples be detected?
• Can a learner identify missing semantic categories?
• Should failures trigger acquisition of new training data?
• How should visual evidence be combined with context and semantic knowledge?
• How can these methods scale to hundreds of semantic concepts and tens of thousands—or eventually millions—of images?

ImageNet provided a particularly successful answer to one critical part of this emerging agenda: scale the labeled visual world dramatically.

It did not begin the entire agenda.

7. “Data-Driven” and “Data-Centric” Should Not Be Synonyms

Historical clarity requires separating at least three ideas.

Under these definitions, SVMActive belongs to an early model-directed data-acquisition lineage: the learner uses uncertainty to decide which semantic label to acquire next. CBSA belongs naturally in the data-driven annotation lineage. Banko and Brill, large Web corpora, ImageNet, and later foundation-model training make data scaling increasingly visible. The CDE formulation is historically intriguing because it explicitly permits the training set L, feature set P, and semantic space C to change in response to uncertainty and failure.

These traditions overlap. But they are not identical.

8. The Historical Question Is Not “Who Invented Data-Centric AI?”

That question is probably too simplistic. Ideas rarely appear fully formed on one date.

A better set of questions is:

Those questions lead to a lineage rather than a creation myth.

A Preliminary Lineage: People, Papers, and Converging Streams

A single chain is too simple. A more faithful picture is several streams that repeatedly crossed. The milestones below are therefore anchors, not claims of sole origin.

The serious historical task is not to pick one winner from this list. It is to determine which ideas existed when, what problem each solved, which ideas were parallel, and where those streams crossed.

ImageNet deserves its place in history. But it should not erase the history that made ImageNet possible.

Why Revisit This History Now?

There is a certain irony in AI's current trajectory.

For years, enormous progress came from increasing data and computation:

Foundation models pushed this recipe farther than almost anyone imagined possible.

Yet the frontier is again forcing us to ask questions remarkably similar to those raised by early semantic-learning systems.

What is missing from the training data? Which examples should be trusted? When does a model's confidence indicate inadequate knowledge? When is the semantic vocabulary itself wrong? How do we distinguish an inadequate model from inadequate evidence? How should context change interpretation? And when should a machine recognize that the world contains something outside the categories it has learned?

The scale has changed by many orders of magnitude. The underlying questions have not disappeared.

Perhaps the history of data-driven AI is therefore not a simple story in which algorithms gave way to data. It may instead be a recurring interaction among three things:

Understanding that lineage matters not merely for assigning historical credit.

It may help us understand what comes next.