Information Retrieval Part 3: Vectorization And Transformers (Not The Film)
The Evolution of Search: From Keywords to Context The landscape of information retrieval (IR) has undergone a tectonic shift over the last decade. In the early days of the internet, search engines functioned much like a digital library card catalog. They relied on exact string matching, looking for the specific sequence of letters you typed into a search bar. If you searched for “running shoes,” the engine looked for documents containing those exact words. If a high-quality page used the term “jogging footwear” instead, you might never find it. This was the era of sparse retrieval, dominated by simple frequency counts and keyword density. Today, we find ourselves in the midst of a semantic revolution. Modern information retrieval is no longer about matching characters; it is about understanding concepts, intent, and relationships. This transition has been fueled by two major breakthroughs: Vectorization and Transformers. These technologies allow machines to “read” and “understand” text in a way that mimics human cognition, albeit through the lens of complex mathematics. For SEO professionals, digital marketers, and tech enthusiasts, understanding these concepts is no longer optional—it is the key to navigating the future of AI-driven search. What is Vectorization in Information Retrieval? To understand how a computer processes language, we must first accept that computers are inherently bad at understanding words but exceptionally good at processing numbers. Vectorization is the process of converting text—whether it is a single word, a sentence, or an entire document—into a numerical format that a machine can manipulate. These numerical representations are called “vectors.” In simple terms, a vector is a list of numbers that represents a point in a multi-dimensional space. In the context of NLP (Natural Language Processing), these dimensions represent different features or “meanings” of the text. While a human might describe a “cat” as a small, furry, carnivorous mammal, a vectorization model might represent “cat” as a series of coordinates like [0.12, -0.56, 0.89, …]. The Move from One-Hot Encoding to Embeddings Early attempts at vectorization used a method called “one-hot encoding.” In this system, every unique word in a vocabulary was given its own dimension. If your vocabulary had 10,000 words, each word was a vector of 10,000 numbers, where all were zero except for the one position assigned to that specific word. This was highly inefficient and, more importantly, it failed to capture any relationship between words. To a one-hot encoder, the word “dog” was just as different from “puppy” as it was from “refrigerator.” Modern information retrieval utilizes “word embeddings.” Unlike one-hot encoding, embeddings are “dense” vectors. They use a fixed number of dimensions (often 300, 768, or more) to represent words. Words that share similar meanings or appear in similar contexts are placed closer together in this multi-dimensional vector space. This allows a search engine to mathematically determine that “king” and “queen” are related, or that “walking” and “ran” are different forms of the same concept. The Geometry of Meaning: Vector Space Models When we represent documents and queries as vectors, we create what is known as a Vector Space Model (VSM). In this model, the “relevance” of a document to a search query is determined by its physical proximity in the vector space. This is a fundamental departure from traditional Boolean search. In a VSM, search is essentially a geometry problem. When a user enters a query, the search engine converts that query into a vector. It then looks for document vectors that are located near the query vector. This approach allows search engines to identify relevant content even if the document doesn’t contain the exact words used in the query. This is the foundation of semantic search. Understanding Cosine Similarity How do we actually measure the “closeness” of two vectors? While there are several methods, the most common in information retrieval is Cosine Similarity. Instead of measuring the straight-line distance (Euclidean distance) between two points, Cosine Similarity measures the cosine of the angle between two vectors. Why use the angle? In text analysis, the length of a document can skew Euclidean distance. A very long article about “SEO” and a short tweet about “SEO” might be far apart in space simply because the long article has more words (larger magnitude). However, the direction of their vectors—representing their topic—will be very similar. Cosine Similarity produces a score between -1 and 1: 1: The vectors are identical in direction (highly relevant). 0: The vectors are orthogonal (no relationship). -1: The vectors are diametrically opposed (opposite meanings). For SEOs, this means that the “topical authority” of a page is mathematically calculated based on how closely its content vector aligns with the intent vector of a user’s search query. Transformers: The Engine of Modern NLP While vectorization provided the “map” for search engines, Transformers provided the “intelligence” to read it. Introduced by Google researchers in the 2017 paper “Attention Is All You Need,” the Transformer architecture revolutionized how machines process sequences of data, particularly text. Before Transformers, models like RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory) processed text word-by-word, from left to right. This was slow and often resulted in the model “forgetting” the beginning of a sentence by the time it reached the end. Transformers changed this by using a mechanism called “Self-Attention.” The Power of Self-Attention Self-attention allows a model to look at every word in a sentence simultaneously and determine which other words are most important for understanding its meaning. It essentially weights the relationships between words regardless of their distance from each other. Consider the sentence: “The bank was closed because of the river flooding.” When a Transformer processes the word “bank,” the attention mechanism links it heavily to “river” and “flooding,” allowing the model to understand that we are talking about a geographical feature, not a financial institution. In a different sentence—”The bank was closed because it was Sunday”—the model would link “bank” to “Sunday,” correctly identifying it as a business. This contextual awareness is what makes modern search feel so much more intuitive. BERT