Fine-tuning an LLM on 5 Years of Telegram Chats
Why & the plan
The idea, an old failure, and why fine-tune at all
1LoRA
The one idea that makes this possible on a rented GPU
2The build
Model, hardware & the messy data pipeline
3What broke
Three failures that taught me more than any tutorial
4Results
The payoff, the cost, the takeaways
5
PyCon DE 2025
HELLO 👋
Alessandro Romano
The human behind the bot.
CO-HOST
My Data Guest — on Substack
Interviews with top AI & data experts, deep dives, and courses.
Follow along mydataguest.substack.com →A chatbot that talks like me — without me.
Telegram has always been my playground for testing new tools. The one idea that stuck: could I build a digital version of myself — one that handles simple conversations in my tone, my humor, my nonsense?
Not because I needed it. Because the concept fascinated me — and because fine-tuning finally got cheap enough to try.
💬 The dataset, in one line
5 years of a group chat with 5 friends. Dark humor, inside jokes, gaming logistics, and absurd threads. The chat I use the most — so, the truest sample of "me."
YEARS AGO
- Tried it with LSTM + Seq2Seq models
- Training was painful, the loss curve erratic
- Gave up quickly — the results never felt like me
Then I realised…
attention is all I needed.
The Transformer changed what's possible for a hobbyist. The same idea that killed my old approach is what makes today's version work.
There's no accuracy metric for "sounds like Alessandro." So I defined the goal loosely — and that turned out to be the right call.
🎭 Tone, not facts
It should sound like me. It does not need to know true things about my life.
💬 Conversational
It should hold a short back-and-forth, not answer a single prompt in isolation.
😅 Fun > correct
If the output is absurd but feels like the chat, that's a win.
⚠️ Why this matters
Chasing "accuracy" would have been meaningless — the point was vibe. Knowing what you're optimizing for changes every later decision: model size, tolerance for hallucination, no need for eval harnesses.
If you've touched data science before, the shape is familiar. Four steps — the trick is in the details of each.
Prepare data
Export & parse chats into prompt / response pairs
Choose a model
Small enough to be cheap, big enough to "speak"
Fine-tune
LoRA on a rented GPU
Test
Run inference, read the (absurd) results
🔁 In reality this was a loop, not a line
I went back to "prepare data" more than once — after seeing what the model learned. The arrows should really point both ways.
The honest first question. Prompting and RAG are cheaper — so why move the weights?
| Approach | What it changes | Good for | For "sound like me"? |
|---|---|---|---|
| Prompting | Nothing — instructions at inference | Quick tasks, style hints | Style leaks; can't capture 5 yrs of voice in a prompt |
| RAG | Adds retrieved facts to context | Knowledge, citations, freshness | Wrong tool — I want a manner, not facts |
| Fine-tune | The model's weights — it internalizes patterns | Tone, format, behavior, style | ✓ Exactly this — bake the voice into the model |
✅ The rule of thumb
RAG changes what the model knows. Fine-tuning changes how it behaves. I wanted the how.
Updating all 7 billion parameters means storing gradients and optimizer states for every one. That's data-center hardware. There's a spectrum from "retrain everything" to "touch almost nothing."
Full fine-tune
All params trainable. Best fit, brutal cost — many × the model in GPU memory.
PEFT · LoRA
Freeze the base, train tiny injected matrices. <1% of params, most of the benefit.
Prompt / prefix tuning
Learn a few soft "virtual tokens." Cheapest, but limited expressive power.
PEFT = Parameter-Efficient Fine-Tuning — the family of methods LoRA belongs to.
Start simple. One layer of a Transformer just multiplies its input by a big weight matrix: h = W·x. Every line below is a weight — a 7B model stacks hundreds of these.
⚠️ The problem with full fine-tuning
To adapt the model you'd retrain every one of these weights — billions of them — and store gradients + optimizer state for each. That needs data-center GPUs. There has to be a cheaper way.
Freeze W — touch none of it. Instead add a parallel branch that squeezes the signal through a tiny rank-r bottleneck (down with A, back up with B), and train only that.
After training, the branch is just two small matrices. You can fold them back into W — one matrix, same shape as before. At inference, LoRA disappears.
LoRA is like giving a trained actor a script — not teaching them to act.
- The actor = the pretrained model. Already knows language.
- The script = my LoRA adapter. Learns the character.
- Swap scripts → same actor plays a different role.
I'm not building intelligence. I'm directing intelligence that already exists.
# freeze base, prep for k-bit (4-bit) training model = prepare_model_for_kbit_training(model) lora_config = LoraConfig( r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05, bias="none", task_type=TaskType.CAUSAL_LM, ) model = get_peft_model(model, lora_config)
- r = 16 — rank of the injected matrices. The main dial: higher = more capacity, more memory.
- lora_alpha = 32 — scales the update (α/r = 2×). How loudly the adapter speaks.
- target_modules — inject into attention's query & value projections only.
- lora_dropout = 0.05 — regularizes; matters a lot on small, noisy chat data.
WHY LoRA IS COMPELLING
- Tiny to store — save only the adapter (MBs), not a 14 GB model
- Cheap to train — <1% of params get gradients & optimizer state
- Swappable — many personalities, one frozen base (the "scripts")
- Faster — smaller update = quicker iterations
WHAT YOU ACTUALLY NEED
- A single GPU — even a modest one, thanks to 4-bit quantization
- QLoRA trick — load the base in 4-bit, train adapters in 16-bit on top
- The PEFT + bitsandbytes stack — a few lines of Python
- A clean dataset — the real bottleneck (up next!)
WHAT "4-BIT QUANTIZATION" ACTUALLY MEANS
Every weight is just a number. 4-bit snaps each one to 16 possible values — like a photo saved in 16 colours instead of millions. Far smaller, still the same picture.
Two independent ways to shrink the job. Rank cut how many numbers we train. Precision cuts how many bits each number costs. Stack both and you get QLoRA.
Rank r
Fewer numbers to train — a rank-16 bottleneck, <1% of params
Precision
Fewer bits per number — store each weight in 4 bits, not 16
Both at once
4-bit frozen base + 16-bit adapters → 7B on one GPU
WHAT A "BIT" BUYS YOU
Each bit doubles the options — more detail, more memory.
PRECISION vs. LOSS
Fewer buckets = coarser rounding. Each weight snaps to the nearest of 16 — that gap is quantization loss.
The bet: across billions of weights, tiny rounding barely changes behaviour — but memory drops 4× (14 GB → 3.5 GB).
Hugging Face has everything from tiny to enormous. I only needed a model that already speaks the language — so I could teach it a style. Bigger models = bigger GPUs = more money.
TinyLlama
~1.1BThe lightweight contender. Cheap and fast to train.
- Runs on almost anything
- Foreshadow: couldn't capture my tone
Mistral-7B
~7BThe workhorse. Enough capacity for subtlety and sarcasm.
- Picked up jokes, nicknames, patterns
- Needs a real GPU — but affordable to rent
Spoiler: the base model (Mistral-7B-v0.1) — not the Instruct version — is what finally worked. More on that in "What broke."
| GPU | Good for | Rent cost | Reality check |
|---|---|---|---|
| NVIDIA L4 | TinyLlama; struggles on 7B | ~$0.40 / hr | Fine for experiments, tight on memory |
| NVIDIA A100 | Mistral-7B, comfortably | ~$1.70 / hr | My pick for the real runs |
⚠️ Lesson learned the hard way
3-hour training runs + a flaky connection = wasted money. Checkpointing (I saved every 500 steps) is not optional.
SUPERVISED FINE-TUNING (SFT)
You teach by example. The dataset is thousands of (prompt → response) pairs, and the model learns to produce the response given the prompt. Almost every format is a flavour of this.
PROMPT · the input / context
What the model is given — an instruction, a question, or the conversation so far.
🧊 shown to the model — but not learned
generate
RESPONSE · the target
The desired output. This is the only part the loss is computed on.
🎯 what the model is trained to imitate
📝 Instruction
"Summarize this…" → answer. Task-following (Alpaca-style).
💬 Chat / conversational MY CHOICE
Turns of user ↔ assistant. Teaches tone & flow — exactly what I need.
📄 Completion
Raw prompt → continuation. The most free-form shape.
GETTING IT
- Downloaded the HTML export from Telegram Lite — easy
- Parsing it into something usable — the annoying part
ONE JSONL ROW = ONE CONVERSATION
Each row is the whole thread up to one of my replies:
- prompt — every line before my reply (User + Assistant)
- response — the reply I actually sent
- Rows overlap: the next prompt = this row + my reply
# row 1 — I reply "Hahaha" { "prompt": "User: Feel like playing COD tonight?\n", "response": "Hahaha" } # row 2 — same thread, one turn later { "prompt": "User: Feel like playing COD tonight?\n Assistant: Hahaha\n", "response": "Skydiving over a landfill…" }
THE RAW THREAD
Every time I (②, ③) reply → one training example.
UNROLLED INTO PAIRS
grey = context given
teal = reply to learn
— notice the prompt grows by one turn each time.
🧵 Why grow the context instead of one-off Q&A?
Because I'm cloning a conversation, not a quiz. Feeding the whole history teaches the model to reply in the flow of a thread — reacting to what was just said — instead of answering each message in isolation.
The prompt and my reply become one row of tokens. I don't want the model graded on re-typing the prompt — only on producing the reply. So in the label row, every prompt token becomes -100 — PyTorch's "skip this one."
# -100 for each prompt token, real ids for the reply labels = [-100] * len(prompt_ids) + response_ids loss = only computed where label ≠ -100
⚠️ I got this wrong twice
Mask the wrong side and the model learns to echo the question instead of answering — training falls apart.
and what it taught me
Loss cruised near ~2, then exploded to ~22 — enough to invalidate the run.
So I went digging in the data.
I bisected to the exact batch that triggered the spike and read every row — until one thread jumped out: a run of dark-humor jokes about a Call of Duty match.
{"prompt": "…what's for dinner?", "response": "pasta 🍝"}
{"prompt": "…you coming out tonight?", "response": "maybe later"}
{"prompt": "…", "response": "[dark-humor thread]"} ⚠
{"prompt": "…gg that was close", "response": "rematch?"}
The model was fighting the data — something in it refused to learn this pattern. Why?
I was fine-tuning Mistral-7B-Instruct. Instruct models are post-trained (RLHF) to be polite and helpful — and they actively resist learning patterns that violate that, including offensive or dark humor. That fight showed up as an exploding loss.
1 · Base model
Pretrained on raw text. A blank, willing actor. No agenda.
2 · + Instruct / RLHF
Tuned to be a helpful assistant. Learns to refuse certain content.
3 · My fine-tune
Pushes toward exactly what RLHF pushes away → gradients tug in opposite directions → instability.
✅ The fix
Switch to the base model: mistralai/Mistral-7B-v0.1. No alignment layer to fight. Loss behaved, and it finally learned my voice.
…a confident, precise, completely irrelevant time — every time.
Back to the data — same trick.
So much of the chat was coordinating gaming sessions — "what time tonight?" → "9:30." Those time-replies were overrepresented, so the model learned they're a great answer to almost anything.
{"prompt": "…what time tonight?", "response": "9:30"}
{"prompt": "…you online later?", "response": "9 PM"}
{"prompt": "…when's the raid?", "response": "around 10"}
{"prompt": "…ready to play?", "response": "9:30"} ⚠ ×1,400+
Preserve your voice — but you decide which behaviors are worth cloning. Do I really want a bot that answers everything with a time?
The loss spike scared me. Three settings did the real work of keeping training steady — here's each one in plain terms.
learning_rate
2e-4How big a step it takes each time it learns.
Too big → it overshoots and the loss explodes.
batch × accum
4 × 4Learn from 16 examples at once, even on a small GPU.
More at once → smoother, steadier learning.
warmup_steps
50Start slow, then speed up — don't floor it cold.
No warmup → a nasty spike in the first steps.
TinyLlama · ~1.1B
disappointing- Couldn't capture my tone
- Hallucinated near-gibberish
- Too small for sarcasm + absurd context
✕ Too small to hold the thread
Mistral-7B · base · ~7B
it worked- Picked up recurring jokes & nicknames
- Caught subtle patterns in how we talk
- Felt like a funhouse-mirror version of me
✓ Enough capacity for nuance
📐 The capacity lesson
Subtlety and context need parameters. For a nuanced persona, ~7B was the floor — 1B just couldn't hold the thread. (The other failure mode, alignment, was independent of size.)
This is real output.
Recurring jokes, our cadence, even the oddly specific times — turned into a punchline ("9:47. Precision is a mindset.").
🪞 The eerie part
"It cost me 14 jumps." Real inside joke, or hallucination? I honestly can't always tell. Maybe that's the point.
It defends the group's "rules."
Selling the console = exile. That's an actual running bit in our chat — the model picked up the logic of the joke, not just the words.
🎭 In character
"Read-only access to the banter" is a line I never wrote — but it's exactly how I'd have said it.
It knows the legend.
B. never connecting is a decade-old bit. The model didn't just recall it — it escalated it ("a decade compressed into one sec").
📈 It builds on the bit
The exaggeration is new, but the rhythm is 100% ours. This is what "captured my tone" actually feels like.
It combined both jokes.
Here it weaves D.'s exile and B.'s legend into one bit — a "party that never happened." Nobody wrote this. The model synthesized our lore.
🪞 The eerie part, again
A funhouse-mirror of the group: 100% fake, 100% us. That's the whole project in six messages.
Nothing fancy. The model lives on a RunPod GPU, a small Python script bridges it to Telegram, and it chimes in on the group chat every so often — not on every message.
RunPod GPU
Fine-tuned model provisioned & served on a rented GPU
Python bridge
A small script connects the model to the chat
Telegram
Reads the group's incoming messages
Replies now & then
Answers some messages, not all — so it feels human
🎲 Why not reply to everything?
A bot that answers every message reads like a bot. Chiming in occasionally — at random — is what makes it blend into the group and feel like one of us.
🤯 The counterintuitive win
Shortening threads and using one year instead of five gave the best results. Shorter, focused conversations = less noise = attention can lock onto what actually signals my reply. More data isn't always better — cleaner, tighter data is.
Next step: one more run on the full, properly cleaned dataset — assuming RunPod doesn't crash mid-training. 🙂
🗂️ Data > everything
The model is a few lines of Python; the dataset is the whole game.
🧊 Start from a base model
For persona/style — Instruct/RLHF will fight you.
💸 LoRA + QLoRA
Makes 7B fine-tuning genuinely affordable — ~$70, one GPU.
🔍 Inspect what you feed it
Overrepresented replies become the model's default.
✨ Cleaner beats bigger
Less, focused data outperformed 5× more.
💾 Checkpoint everything
Long runs on rented GPUs will get interrupted.
🎯 The meta-lesson
Fine-tuning has become accessible. The hard, interesting problems have moved upstream — into what data you choose, and what behavior you actually want to replicate.
THANK YOU
Maybe I sound like a
hallucinated LLM in real life.
And maybe that's exactly why it works.
Alessandro Romano · Questions?