Searching by Meaning

Overview

This lesson explains semantic search: searching by meaning instead of exact words. Semantic search can help find important information to bring into a conversation so that an agent can reason effectively. The key problem it solves is when the information that an agent needs is written with different words from what the agent is searching for or is present in the user's task description.

Where this fits

The information an agent needs often already exists in a document somewhere — it just is not in the conversation yet. Semantic search is a way to find it when the document says the same thing in different words than the question uses. It answers "where is it" by matching meaning instead of matching strings.

A review of text search

Let's assume, the user asks an agent to schedule a "night delivery". So, the agent searches for “night delivery” in a policy document. However, the document uses the phrase “deliveries outside ordinary working hours” as the heading for the section that contains the relevant information. Ordinary text search will not find the needed information because it is worded differently.

The information is present. The wording is different.

search_document(doc = "doc-77", pattern = "night deliver")

0 matches.

search_document(doc = "doc-77", pattern = "night")

2 matches: p. 12 "overnight mooring"; p. 98 "night watch staffing".

The rule was in the document under this heading:

The query and the heading mean the same thing, but they share almost no words.

That is the problem:

Problem: Text search looks for matching strings.

It does not understand that two different strings
can express the same meaning.

Embeddings enable search by meaning

Consider these three statements from a university catalog:

Statements A and B use different words, but they communicate almost the same rule.

Statement C is about something completely different.

We want a computer to recognize that A and B are related.

To do this, we use a technique called embeddings.

An embedding model takes a piece of text and produces an embedding: a list of numbers.

You can think of this list of numbers as the text’s location on a map of meaning.

The map follows one basic rule:

Similar meaning = nearby embeddings
Different meaning = distant embeddings

The two course-requirement statements receive nearby locations because they mean similar things.

The meal-plan statement receives a more distant location because it is about something else.

The real map has thousands of dimensions, so we cannot draw it directly. A simplified two-dimensional version might look like this:

The academic requirements appear near one another. The campus-life statements form a different group.

The two calculus statements are especially close because they express nearly the same rule.

This works even when the statements:

• Use different words.
• Use different sentence structures.
• Are written in different languages.

The model learns these relationships by processing large amounts of text during training.

Meaning has many parts

Two statements can be similar in one way and different in another.

For example, we could group catalog statements by subject:

Here, the academic statements are grouped together because they concern degree requirements.

We could also group the same statements by whether they describe a rule or a service:

Now move-in day is grouped with course requirements because both describe something a student must follow.

Neither grouping is wrong. They compare different parts of meaning.

Real embeddings use thousands of dimensions to represent many kinds of relationships at the same time.

What the numbers mean

Imagine a very simple embedding with only three dimensions:

The first two embeddings are close because their numbers are similar.

The third is farther away.

This is only a teaching example. Real embedding dimensions do not have simple labels such as “about food.” The model learns the dimensions during training, and individual dimensions are usually difficult to interpret.

The important point is:

Why we do not embed an entire document

An embedding model can accept a sentence, a page, or a whole document.

But every piece of text produces only one embedding.

That creates a problem.

The more information one embedding must represent, the less precise it becomes.

This is especially dangerous when a document contains conflicting statements:

Both statements are about STAT 3011 and degree requirements. Their embeddings may therefore be close.

But the rules are different:

• One allows the course.
• The other does not.
• They apply to different majors.

If both statements are embedded together, those differences may become difficult to retrieve.

One embedding must represent everything in the text.

As the text becomes larger and more varied,
important details become harder to find.

To create more precise embeddings, we divide the document into smaller pieces.

Dividing a document into chunks

Dividing a document into pieces is called chunking.

Each chunk receives its own embedding.

For example, instead of creating one embedding for an entire catalog, we might create separate embeddings for:

• Each major’s requirements.
• Each academic policy.
• Each course description.
• Each housing rule.

This gives each piece a more specific location on the map of meaning.

But chunking introduces another problem: a chunk can lose important context when it is separated from the surrounding text.

Consider this page break:

Now suppose ordinary code divides the pages every 12 lines:

Chunk 43 contains the answer to this question:

“Is MATH 3320 required for the computer science major?”

But the words “computer science major” are in Chunk 41.

Chunk 43 is now just a list of math and statistics courses. Its embedding may place it near mathematics requirements instead of computer science requirements.

The information still exists, but its context has been separated from it.

search_by_meaning(
  query = "math courses required for the computer science major",
  top = 3
)

1. Chunk 118 — Mathematics major: core requirements
2. Chunk 41 — CS department history and the heading
   “REQUIRED COURSES — COMPUTER SCIENCE MAJOR”
3. Chunk 87 — Mathematics minor: course list

The key lesson is:

A chunk’s embedding is created from the text inside that chunk.

If the heading, subject, rule, or exception is placed in another
chunk, that context is missing from the embedding.

Choosing the right chunk size

Chunks can be too large or too small.

There is no perfect number of words or lines for every document.

A useful chunk should contain:

• One complete idea.
• The heading or subject that explains the idea.
• Any important conditions or exceptions.
• Enough context to answer the expected questions.

Chunk size should be based on the questions users will ask, not just a fixed line or character count.

Smarter chunking

Simple chunking code can split text every few hundred words or at every page break.

That is fast, but it does not understand the document.

A language model can help create better chunks.

Let a model choose the boundaries

Before storing a document, ask a model to identify where each complete idea begins and ends.

For example:

The model can keep this heading:

“REQUIRED COURSES — COMPUTER SCIENCE MAJOR”

together with the course list that follows it, even if the list continues onto the next page.

This requires extra work when the document is first processed, but every later search can benefit from the better chunks.

Store the questions each chunk can answer

A second technique is to generate likely questions for each chunk.

Consider this chunk:

A model could generate these questions:

We can create embeddings for those questions and connect them to the original chunk.

Then, when a user asks a similar question, the system compares a question with other questions instead of comparing a question with a course list.

Original approach:

User’s question → compare with document chunks


Question-generation approach:

User’s question → compare with generated questions
                → return the connected document chunk

This can improve retrieval because the stored text is written in the same form as the user’s query.

Breaking a complex task into smaller searches

Users often ask questions that require several pieces of information.

For example:

This is not one simple search. It contains several questions:

1. What courses are required for the computer science major?
2. Which completed mathematics courses count toward those requirements?
3. What prerequisites do the remaining courses have?
4. How many courses can the student take each term?
5. Are there residency or minimum-term requirements?

If we embed the entire request as one query, its meaning may be too broad. It might match general advising information instead of the specific rules needed to answer each part.

A better approach is to search for each sub-question separately.

think(thought = "I need to answer five smaller questions:
1. Which courses are required for the CS major?
2. Which completed math courses count toward those requirements?
3. What prerequisite chains do the remaining courses have?
4. What is the maximum course load per term?
5. Is there a residency or minimum-term rule?")

OK.

search_by_meaning(
  query = "courses required for the computer science major",
  top = 2
)

[CS core requirements; CS prerequisite map]

search_by_meaning(
  query = "mathematics courses that count toward the computer
  science major",
  top = 2
)

[STAT 3011 counting rule; policy for courses shared across majors]

search_by_meaning(
  query = "maximum course load per semester",
  top = 2
)

[course-load policy; overload policy]

search_by_meaning(
  query = "residency or minimum-term requirement after changing majors",
  top = 2
)

[residency rule for a new major]

The agent turned one broad request into several precise searches. It then combined the results into one answer.

Broad question → divide into specific questions
Specific questions → retrieve focused evidence
Focused evidence → combine into one answer

Two matching techniques

We have now improved both sides of semantic search.

During document processing

Generate questions that each chunk can answer.

This moves the stored chunk closer to the kinds of questions users will ask.

During search

Divide a complex request into smaller questions.

This gives each part of the request a more precise embedding.

These are two sides of the same idea:

Prepare the document around likely questions.

Prepare the search by identifying the exact questions
that must be answered.

What embeddings do not know

Embeddings are useful, but they do not determine whether a statement is correct.

They mainly help identify what a passage is about.

Three limitations are especially important.

1. Truth

Consider these statements:

They make opposite claims, but they use nearly the same words and discuss the same topic.

Their embeddings may be very close.

Semantic search can find both statements. It cannot decide which one is current or correct.

2. Negation

Consider:

The word “not” completely changes the claim.

But most of the text is identical, so the embeddings may still be close.

A system must read and verify the retrieved text instead of assuming that the nearest result agrees with the query.

3. Exact identifiers

Sometimes exact spelling is the meaning.

Examples include:

• MATH 3320
• Invoice #2214
• Gene BRCA2
• Contract A-1097
• Product model XR-15

For these searches, ordinary text matching is often more reliable than semantic search.

Choosing the right search method

Semantic search does not replace every other method.

A useful system has several options:

The right method depends on the question.

Strong systems often combine methods.

For example, they might use semantic search to find the relevant section, exact search to locate a course number, and a language model to interpret the rule.

The main idea

An embedding is a numerical representation of a piece of text.

You can think of it as the text’s location on a map of meaning.

Similar meanings usually receive nearby locations.

For useful search:

1. Divide documents into complete, meaningful chunks.
2. Keep headings, conditions, and exceptions with the text they explain.
3. Generate likely questions for important chunks.
4. Divide complex requests into focused sub-questions.
5. Use exact text search for names, numbers, and identifiers.
6. Read and verify the results, especially when truth or negation matters.