Pipelines

This project now supports reusable pipelines for: - indexing all docs into JSON - building per-training corpora from those docs - creating separate model bundles - training new bundles from scratch - fine-tuning existing bundles with new/updated docs - asking questions against a specific bundle with

SmallLanguageModel Pipelines

This project now supports reusable pipelines for:

0) Environment

Use .venv/bin/python in the commands below if your system Python does not have the required packages.

2) Build JSON Index + JSONL Content From Scan

This index now includes rich metadata per file: path, hashes, bytes, chars, lines, words, timestamps, and token counts (tokenswhitespace, tokensestimate4chars). If you want tokenizer-accurate token counts, add --tokenizerfile artifacts/models/<model/tokenizer/tokenizer.json.

3.5) Build Retrieval Chunks (Recommended)

buildchunkindex.py does not train the model; it prepares retrieval chunks used by RAG.

5) Initialize a New Model Bundle

Creates an isolated bundle under artifacts/models/<modelname.

8) Clone Existing Model Into New Version (Recommended)

This keeps the old model frozen and trains a new child version.

9) Fine-Tune Existing Bundle In Place (Optional)

Rebuild index first, then changed-only corpus: