Video-Distiller

Python · MIT

Why I built it

Content in this field now accumulates faster than watch time exists. Courses, talks, and papers pile up quicker than anyone can consume them, and video, the format most of it arrives in, offers exactly one interface: watching, at one hour per hour.

My version of the problem is concrete. In 2019 I took three courses, C, C++, and Advanced C++, around 160 hours each. I worked in C++ afterwards and everything held. Then projects moved on, the details faded, and the only way back the format offers is rewatching nearly 480 hours. That is not review; that is taking the courses again. And it misprices the problem: the knowledge isn't gone, the access is slow. What faded needs recall, not re-learning, and recall should cost minutes, not the hours I already paid once.

So I built the thing I wanted: point it at a video, get back structured, timestamped notes, and, once you've distilled a few, a queryable knowledge base over the whole collection. Watching becomes optional. Asking becomes the interface.

What it does

Three models do the reading, because a video carries two channels of information and neither one alone is enough. A vision-language model reads what's on screen: slides, code, diagrams, the whiteboard. ASR transcribes what's said. An LLM fuses the two, aligned by timestamp, into grounded, concept-level notes.

The repo is two layers, deliberately:

Layer 1, distill. A seven-stage pipeline:

ingest → transcribe → detect_scenes → extract_visuals → align → distill → render

Every stage reads and writes typed Pydantic artifacts to a job directory, so any stage can be re-run in isolation.

Layer 2, knowledge base. The notes from many videos get embedded into a local, on-disk Qdrant index, no Docker, no server, and searched with hybrid retrieval: dense vectors for meaning, BM25 for exact tokens, fused with reciprocal rank fusion. The BM25 half matters in technical content, where a literal identifier is often exactly what you're looking for and dense vectors alone will paraphrase it away.

The output of this project is the vector database itself. What consumes it comes next (more on that below).

The decisions I'd make again

Domain knowledge lives only in profiles. All of it, vocabulary to seed the ASR prompt, which code languages to expect, how to verify a snippet, sits in profiles/*.yaml. The pipeline code contains none of it. Point the same pipeline at a different profile and it distills a different subject. This came directly from production work: the systems that survive are the ones where configuration changes don't require code changes.

Provider abstraction, actually enforced. Pipeline code never imports a model SDK. All model access goes through one small layer, so the backend can be swapped, OpenAI to Gemini to whatever comes next, without touching stage logic. That single layer is what makes the cost routing below possible.

Typed artifacts between stages. Each stage's output is a JSON artifact with a Pydantic schema. When stage five misbehaves, you re-run stage five, not the pipeline. Anyone who has debugged a monolithic script that re-transcribes forty minutes of audio to test a prompt change knows why this matters.

The vector store is a protocol, not a dependency. The knowledge base talks to storage through a four-method protocol. Qdrant ships as the default because it runs embedded with zero infrastructure; pgvector or Chroma is an implementation away.

What it costs, and why that's a design axis

Cost is one of the constraints the architecture answers. Distilling a long course means thousands of frames through a vision model, and that's where the money goes. Three decisions keep it sane:

• The vision call is routable. The extract_visuals stage can be served by OpenAI or Gemini with one environment variable, and nothing else changes. Switching from GPT-4o at full resolution to Gemini 3.1 Flash-Lite with thinking off took vision from the dominant multi-dollar line item to about $0.30 per lecture. That one route accounts for roughly 80% of the total savings.
• Frames are downscaled to 1024px before the vision call. That lands them in a cheaper pricing tier, with no visible quality loss on slides and code.
• Every model response is cached by content hash. Re-running a stage costs nothing, whether that's iterating on the distill prompt or recovering from a crash.

The numbers: the first two courses, roughly 120 lectures of about 160 minutes each (117 processed as I write this), came to around $145 in API spend, with $160–180 the honest ceiling once the longest lectures are counted. The same set projected onto the original full-resolution GPT-4o path was $450–480, so call it a two-thirds cut, from about $4 per lecture to $1.20–1.50. And one line item kept for honesty: about $12 of that spend was wasted on a duplicate-processing bug before I caught it.

With vision handled, ASR becomes the floor. Whisper on a 160-minute lecture costs about a dollar by itself, which is what holds the per-lecture figure where it is. Optimising the biggest lever doesn't end the cost conversation; it moves the floor to the next one.

It grades its own homework

The repo ships an eval command that grades a finished job: code-compile rate for extracted snippets, vocabulary hit-rate against the profile, an LLM grounding spot-check, and cost, compression, and timing. Anyone can re-run it on their own jobs.

A note on the corpus, copyright, and privacy

The courses I distill are commercially licensed material I paid for, so the boundary is explicit: everything runs on my own machine, the notes and the vector database exist for my personal study, and nothing derived from the corpus, no transcript, no frame, no note, ships with the repo or appears in its examples. The repo is the tool. You bring your own content, and the same responsibility.

One design consequence deserves the same transparency: transcription and vision extraction call external model APIs, which means audio and frames from your content pass through OpenAI or Google. Both publish their data terms. OpenAI states that data sent to the API is not used to train its models unless you explicitly opt in, though it is retained for up to 30 days for abuse monitoring. Google's Gemini API terms say the same for paid-tier use; on the unpaid tier, content can be used to improve Google's products, with one exception worth knowing: for users in the UK, EEA, and Switzerland, the paid-service data terms apply to every tier. Read the current versions yourself before processing anything licensed or private. Terms change, and this paragraph is a pointer, not a guarantee.

What's next

The vector database is deliberately the last artifact this repo produces, because retrieval is its own problem with its own failure modes. The next project consumes this one: cross-video summarisation and Q&A over the collection, built on the hybrid search this repo provides. Same pattern as before, the project first, then a post on what evaluating it actually took.

If you build a profile for your own domain, or swap in a different vector store, I'd genuinely like to hear how it went. The repo is MIT, and issues are open.