CurriculumGenerator

• A generation takes long enough that a normal web request would time out before it finished.
• The answer has to come from the uploaded material, so retrieval has to happen before generation.
• The app needs a predictable shape back, so the model must return curriculum JSON in a fixed schema.
• The background worker has to run on Windows, macOS and Linux.

From PDF to curriculum

Every generation runs the same pipeline:

1. PyMuPDF extracts the text from the uploaded PDFs.
2. The text is split into chunks, either overlapping or paragraph-aware.
3. SentenceTransformers turns each chunk into an embedding.
4. The embeddings are stored in ChromaDB.
5. A strict prompt, with the retrieved material, makes the model return the curriculum as JSON in a fixed schema.

The API and the job queue

The backend is a Flask API with endpoints to list all curricula, fetch one by id, and export one as a PDF. Generation does not run inside the request. It runs as a job on a Redis queue using RQ, and a worker picks it up. MongoDB keeps the curriculum history and its metadata.

My part and the team's

The repository I link is my fork of a teammate's original. It has 54 commits, and 36 of them are mine.

Generations that outlast a request

A long generation and a web request do not fit together: the request times out while the model is still working. The job queue solves this by separating the two. The API accepts the request and puts a job on the queue, and the worker does the slow part. I also tuned the timeouts on both the frontend and the backend so neither side gave up early.

A worker that runs on every OS

RQ's standard Worker forks a process for each job, and Windows cannot fork. SimpleWorker runs jobs inside the worker process instead. The worker code in core/worker.py picks SimpleWorker on Windows and macOS and Worker on Linux, so the same code starts anywhere.

Bugs in the data path

The rest of my work was in how data moved around: a bug where the database was being overwritten, a bug with the PDF flag, and curriculum data that could come back as either a string or an object and needed parsing for both.

Generation works end to end: upload PDFs, get a structured curriculum, see it in the history later, and export it as a PDF. Of the 54 commits in the repository, 36 are mine.

Add tests around curriculum parsing. It has to handle two formats, which is exactly where regressions hide.

Then compare the two chunking modes, overlapping and paragraph-aware, on the same PDFs, and keep the one that gives better curricula.