Indexing and Querying Academic Hub

Aug 30, 2026·
Aaron Scherf
Aaron Scherf
· 2 min read
Three stages show course and document narrowing, passage retrieval, and a sourced answer. Illustration: Course and document narrowing leads to passage retrieval and a sourced answer.
blog

Once books and notes have been converted to text, the next challenge is finding the right material and showing where an answer came from. The Source Indexer organizes the documents; RAG Analysis searches their passages to help answer questions.

The indexer creates a record for each document and groups related material by course. Search first narrows the collection to likely courses and documents, then ranks passages within them. A book can be relevant to a topic overall while only a few paragraphs contain the explanation a question needs.

The answering tool also limits how many passages each file can contribute. That helps keep one highly relevant textbook from crowding lecture notes and other sources out of the material used to write the answer.

In a recorded trial, it combined passages from two sources to answer a question about the spectral theorem, with references back to the material it used. When the collection does not support an answer, the system is intended to make that gap visible instead of dressing up a guess as a sourced result.

An illustrative source reference is Section 7.4, pages 264–266. That tells me which part of a source to open when I want to check an explanation or read further. A reference makes the reasoning easier to inspect; it does not guarantee that the answer interpreted the passage correctly.

This was an early milestone in the project. Since then, the work has expanded to include tutoring hints, answer checking, and more links between related notes. The project pages describe the current behavior, testing, and limitations; this update records the initial step from converted documents to searchable, referenced answers.

Aaron Scherf
Authors
Aaron Scherf (they/them)
PhD Student
I am a PhD student in Sustainable Development at Columbia University and a former USAID Foreign Service Officer. In Ukraine, I led the monitoring and evaluation team for over 50 development projects during the war. Previously, I worked on program design and social inclusion in Colombia, as well as budget analysis and knowledge management in Washington. Those experiences taught me how to ensure evidence is trustworthy and actionable, especially in conditions of rapidly changing and incomplete information. I now bring that perspective to research, data analysis, and practical software tools. I am available part-time for consulting and applied research collaborations.