Notes on Building Software Over Indian Court Data

I have spent a while now building systems that read Indian court records at scale. Most of what cost me time was not infrastructure. It was a set of assumptions I brought from working with cleaner datasets, each of which turned out to be wrong in a way that only shows up after you have indexed a few million documents. This is a writeup of those assumptions, in the hope that it saves someone else the same weeks.

None of this is specific to any one vendor or stack. If you are scraping eCourts yourself, if you are parsing a bulk dump, or if you are consuming somebody's API, the same shape of problem shows up.

A Case Is Not a Document

The first mental model to throw away is that a case is a thing with a judgment attached. In practice a matter is a stream of documents produced over years: interim applications, adjournment orders, stay orders, orders on interlocutory applications, and eventually, sometimes, a final judgment. A single case number in a High Court can carry thirty or forty documents. Some of those documents are two lines long and say the matter is adjourned to a date.

This matters for three reasons.

Your primary key cannot be the document. If you key on the PDF, you will show a user forty near duplicate cases for what is one dispute. You need a case level entity and a document level entity, and a join between them.

Your relevance model degrades badly if you index adjournment orders alongside judgments with equal weight. A two line adjournment order that happens to contain a party name will outrank a substantive judgment on a party search, because term frequency normalisation rewards short documents.

Your latest status is a derived value, not a stored one. The most recent document under a case number tells you where the matter stands. That derivation has to handle documents arriving out of order, because they usually do.

Case Numbers Are Not Identifiers

A case number in India generally looks like WP/13424/2018: a type code, a serial number, and the year of registration. It is unique within a court, within a case type, within a year. It is nowhere near globally unique. Writ Petition 13424 of 2018 exists in more than one High Court.

Worse, the same matter accumulates different numbers as it moves: a diary number at filing, a registration number once it is admitted, an appeal number in the court above. People cite all of them interchangeably.

The 16 character CNR is the closest thing to a stable identifier, and it is worth extracting and indexing wherever it appears, because it encodes the state, the establishment and the year. It is not universal, though. Older records predate it and some tribunal records never carry one.

Practical consequence: any search UI that lets a user paste a case number has to treat that string as a fuzzy hint, not a lookup key. Users paste 123-2024, 123/2024, WP 123 of 2024 and W.P.(C) No. 123/2024 to mean the same thing. Normalise aggressively before you match: strip separators, collapse whitespace, and keep both the normalised and the original form so that either can hit.

Judge and Party Names Will Break Your Analyzer

This is where I lost the most time, and the failure mode is silent.

Judge names arrive in wildly inconsistent forms. Honorifics get concatenated. Initials run together with the surname, so you see T.KRISHNAKUMAR as one token. The same judge appears as JUSTICE K M BASHEER, K.M.Basheer, J. and Hon'ble Mr. Justice K.M. Basheer. Transliteration varies between courts and across decades.

If you feed those through a standard English text analyzer, two things go wrong. First, the default word break rules keep t.krishnakumar as a single token, so a search for Krishnakumar misses it entirely. The fix is a character filter that rewrites a period sitting between two letters into a space, restricted with a lookbehind and lookahead so that decimals like 1.5 crore and abbreviations like etc. are left alone.

Second, and this one is nastier: English stemming mangles short uppercase tokens. An acronym like KJS stems to kj, which silently merges it with every KJ in the corpus. Names must not go through a stemmer. Give them their own field with a lowercase and ASCII folding chain, no stemming, no synonyms, and keep the stemmed analyzer for body text where it actually helps.

Party names have a different problem: frequency. State of Maharashtra is a respondent in millions of records. A naive party search on a common government entity returns everything and ranks nothing. You need either a separate exact field for party matching, or a way to detect and down weight ubiquitous institutional parties.

Dates Lie, and Some of Them Are in the Future

Court metadata is typed by humans under time pressure. You will find decision dates in the future, decision dates before the registration date, and dates in at least three formats. Some of the future dates are typos with the year transposed, some are the next hearing date entered in the wrong field.

Do not silently drop those records, they are usually real documents with one bad field. Do exclude them from anything sorted by recency, or they permanently occupy the top of every latest judgments view. A cheap fix that works: keep them, but demote any record whose decision date is after today in the ranking stage.

Also separate the two years you care about. The year in the case number is the year of filing. The decision date year is the year the matter ended. Users searching 2019 cases mean one or the other depending on who they are, and a filter that silently conflates them produces results that look arbitrary.

Keyword Search and Vector Search Answer Different Questions

The most useful architectural decision I made was to stop treating these as competitors.

Keyword search is what you want when the user knows what they are looking for: a citation, a party, a section, a case number. It scales cheaply to hundreds of millions of documents, it is explainable, and it degrades gracefully. It is also the only thing that works on a corpus where most documents have never been enriched.

Vector search is what you want when the user knows the situation but not the vocabulary: an employer dismissed a workman without a domestic enquiry. No keyword query expresses that reliably, because the judgment may say termination, discharge simpliciter or removal from service.

The cost asymmetry is the point. Indexing text for keyword search is cheap. Embedding a document costs an API call and storage for a vector, per document, forever. At a few hundred million documents that is not a rounding error, it is the budget. So the honest architecture is a large keyword index over everything and a smaller vector index over an enriched subset, with the product being explicit about which one a given query hit.

Storage: Index It, Do Not Store It

A judgment PDF's extracted text can run to hundreds of kilobytes. If you put that in your search engine's stored source, your index size explodes and every query pays for it.

The pattern that worked: index the body fields so they are searchable, exclude them from the stored source, and keep the canonical text in your primary datastore. Search returns identifiers and a small display payload, then you hydrate from the primary store for the documents actually being shown. Recall is unchanged, index size drops by a large multiple, and query latency improves because the engine moves less data.

The corollary is that whatever you exclude from stored source cannot be returned by a search response. That is a contract you have to document, or every consumer will file the same bug.

Deep Pagination Needs Cursors

Offset pagination has a hard ceiling in every search engine, and long before that ceiling it gets slow, because computing offset 40,000 means the engine sorts 40,000 results and throws away 39,980 of them.

If your users export result sets, and legal users always eventually export result sets, expose a cursor. Return the sort key of the last hit and accept it back as a resume token. Two consequences to document: a cursor paginated response has no meaningful page number, and results are only stable if nothing was indexed between calls.

The Boring Things That Actually Matter

Citations are their own category of boring and important. Parse them into a canonical form at index time and store the canonical form as a keyword. Do not try to parse them at query time.

If You Want to Skip the Ingestion Problem

Happy to talk about any of it in the comments. If you have found a better way to handle party name frequency, in particular, I would like to hear it. Litigation checks by party name sit alongside this same corpus and are available to Enterprise accounts today and roll out to self serve tiers next; this post is about the ingestion problem underneath, not a pitch for it.