Fixing EOS Token Errors: Why Your Fine-Tuned Model Won't Stop Generating

The "Infinite Conversation" Bug

You just spent 8 hours running Supervised Fine-Tuning (SFT) on an 8B parameter model. You load it up for inference and ask it a question. The model answers perfectly, but instead of stopping, it outputs a newline, writes a brand new question from a fabricated user, and proceeds to answer that too.

This happens because language models are fundamentally next-token predictors. When the model reaches the end of its intended answer, it needs a mathematical stop sign (the EOS token) to trigger the inference engine to halt. If your training dataset was improperly formatted, the model never learned to associate the end of its reasoning trace with that stop sign.

The Fix: Aligning Tokenizer Templates

Base models have different vocabularies. You cannot apply a generic ChatML format to a Llama 3 architecture without causing severe tokenization mismatches. Before passing your dataset into the SFTTrainer, you must enforce the correct special tokens.

from transformers import AutoTokenizer

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# 1. Verify what the model actually expects
print("Model EOS Token:", tokenizer.eos_token) 
# Llama 3 outputs: <|eot_id|> (NOT <|end_of_text|>)

# 2. Fix the padding index (Crucial for batched SFT)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
    
# Enforce right padding so the EOS token stays at the absolute end
tokenizer.padding_side = "right"

Applying the Chat Template

Never manually concatenate strings like "Assistant: " + text + "\n". Always map your JSONL data through the tokenizer's native apply_chat_template method. This guarantees the correct EOS token is baked into the tensor arrays during the forward pass.

def format_dataset(examples):
    # This automatically appends the correct EOS token per the model's config
    texts = [
        tokenizer.apply_chat_template(
            messages, 
            tokenize=False, 
            add_generation_prompt=False
        ) 
        for messages in examples["messages"]
    ]
    return {"text": texts}