What makes Nepali-language document retrieval difficult?

A practical test plan for Nepali, English, and mixed-language document questions, from text extraction to evidence-backed answers.

Test the language combinations your team uses

A document assistant should be tested against the way people actually ask questions. Build separate examples for Nepali questions over Nepali documents, English questions over English documents, and cross-language questions. Include Romanized Nepali or mixed-language requests only when they reflect the intended users. These are proposed evaluation categories, not benchmark results.

Have a reviewer identify the passage that should support each answer. Compare retrieval on the same questions before and after a model or preprocessing change. Do not assume that a model described as multilingual will handle your organization’s terminology, abbreviations, or document collection equally well.

Inspect extracted text before choosing embeddings

Check whether a PDF contains usable text or needs optical character recognition. Compare the extracted text with the visible page, including names, headings, and tables. If the source uses a legacy font encoding, first confirm how it maps to Unicode; normalization alone does not perform that conversion.

Unicode normalization gives equivalent character sequences a consistent representation. It does not correct spelling, translate between languages, or resolve every OCR error. Preserve the source and test any normalization on representative text before indexing it. Keep identifiers and document versions attached so a reviewer can trace an answer back to the original.

Follow the evidence through the system

Consider a document assistant answering a question about an internal policy. Before it can compose a useful response, it needs to find the relevant passage, preserve its meaning, and distinguish the current policy from an older version. A fluent answer cannot repair evidence that never reached the model.

Inspect the retrieved passages alongside the final answer. This separates two questions: did the system find enough information, and did it use that information correctly? Keeping those questions separate makes a failure easier to diagnose.

Make retrieval inspectable

For each test question, retain the source identifiers, relevant metadata, retrieved text, and ranking order. Avoid logging sensitive material unnecessarily; the trace itself needs appropriate access and retention rules.

Ask a reviewer whether the evidence could support a correct answer. If not, inspect ingestion, extraction, chunk boundaries, document versions, and access filters before changing the answer prompt. An extraction problem and a ranking problem need different fixes.

Treat citations as something to check

A citation is useful when a reader can open the referenced passage and verify the associated claim. A document title alone may be too coarse, especially when the document is long or internally inconsistent.

Include test cases where the documents do not contain an answer. Define an acceptable response for those cases: explain the missing evidence, ask for clarification, or route to a person. A confident guess should not count as success simply because it reads well.

Change one part, then compare

Keep a fixed set of questions while trying a retrieval change. Compare evidence quality and answer quality separately, and record what became worse as well as what improved. Add newly discovered failures to a regression set.

The aim is a traceable relationship between a technical change and a useful result. That relationship is more valuable than a collection of impressive answers selected after the fact.