How to fine-tune an LLM on Databricks

For many enterprises, the data worth fine-tuning on already lives in Databricks: support transcripts in Delta tables, documents in Unity Catalog volumes, labels from domain experts in a governed schema. Moving it somewhere else to train is slow, expensive and often not allowed. This guide shows the end-to-end pattern for post-training LLMs inside a Databricks workspace — from governed data to a served endpoint — and where a post-training platform fits.

The reference architecture

A production-grade fine-tuning pipeline on Databricks has five layers, all governed by Unity Catalog:

1. Data — source tables and volumes in Unity Catalog; curated training and eval sets written back as versioned Delta tables.
2. Compute — GPU clusters or serverless GPU compute for training and batch evaluation.
3. Training — a training framework (TRL, torchtune, Composer, or a platform like POFE) running on that compute.
4. Tracking — MLflow for parameters, metrics, artefacts and lineage; models registered in Unity Catalog.
5. Serving — Databricks Model Serving endpoints, or export to another runtime.

The governing principle: data, checkpoints and models never leave Unity Catalog's permission model. Every artefact has an owner, an access policy and lineage back to its sources.

Step 1: Prepare training data in Unity Catalog

Start from a governed source table and produce two outputs: a training table and a frozen eval table. Do the work in SQL or PySpark so it's reproducible and scheduled.

-- curated SFT examples from resolved support tickets
CREATE OR REPLACE TABLE ml.support.sft_v4 AS
SELECT
  ticket_id,
  redact_pii(customer_message)  AS prompt,
  redact_pii(agent_resolution)  AS response,
  intent, language
FROM prod.support.tickets
WHERE status = 'resolved' AND csat >= 4
  AND created_at < '2026-07-01';   -- temporal split: eval uses later data

• Redact PII before training, not after. A model can memorise rare strings such as names and account numbers.
• De-duplicate templated and near-identical examples; support and email data is full of them.
• Split by time where possible, so the eval reflects data the model hasn't seen.
• Record the table version (Delta time travel) used for each run, so results are reproducible.

For preference training, generate candidate responses from your SFT model in a batch job, then collect labels from domain experts or a calibrated judge into a pairs table with prompt, chosen and rejected columns.

Step 2: Choose GPU compute

Size compute from the model and method; the LoRA vs full fine-tuning guide has the memory maths. Typical starting points:

Two practical notes. First, GPU availability varies by cloud region; check quota before planning a large run. Second, if your Databricks GPU capacity is constrained, a post-training platform can burst training to external GPU capacity while keeping data access governed — reading from Unity Catalog through scoped credentials and writing checkpoints back to a volume.

Step 3: Train and track with MLflow

Whatever framework you use, log every run to MLflow with the data version, base model, method, hyperparameters and evaluation results. That is what lets you answer "which data produced the model in production?" six months later.

import mlflow, pofe

mlflow.set_experiment("/Shared/support-agent")
ws = pofe.workspace("acme", backend="databricks")   # uses the cluster's UC identity

with mlflow.start_run(run_name="qwen3-32b-sft-v4"):
    run = ws.train(
        base="qwen3-32b", method="sft", adapter="lora:64",
        data="uc://ml.support.sft_v4@v12",
        eval="uc://ml.support.eval_golden@v3",
        output="uc://ml.support.models/support_agent",
    )
    run.wait()   # metrics, lineage and the registered model land in MLflow / UC

Register the resulting model in Unity Catalog with an alias such as @candidate. Promotion to @champion should require the eval gate to pass, not a manual judgement call.

Step 4: Evaluate on governed eval sets

Keep your golden eval set in its own Unity Catalog schema with read-only access for training jobs, so it can't leak into training data by accident. Run batch evaluation on every candidate and log results to the same MLflow run.

• Score deterministic metrics (exact match, F1, execution accuracy) in Spark at scale.
• For open-ended outputs, use an LLM judge calibrated against expert labels — see evaluating fine-tuned models.
• Report slices (intent, language, segment) and confidence intervals, and compare against your current production system on the same items.

Step 5: Serve and close the loop

Deploy the champion model to a Databricks Model Serving endpoint or export the weights to another runtime. LoRA adapters can be served on top of a shared base model, which keeps costs down when you run several fine-tuned variants.

Then close the loop. Inference tables capture requests and responses back into Delta; sample them for review, add failures to the eval set and use the best interactions as new training data. A pipeline that retrains monthly on fresh, governed data is worth more than a single brilliant run.

Governance checklist and common pitfalls

Most of the effort in an enterprise fine-tuning project goes into making it governable, not making it train. A checklist to agree with your platform and security teams before the first run:

• Identity. Training jobs run as a service principal with read access to specific training tables and write access to one model schema — not as an individual user.
• Data scope. Training and eval schemas are separate; eval data is not readable by training jobs.
• PII. Redaction runs upstream of the training table and is recorded in lineage; sampled outputs are reviewed for memorised identifiers before release.
• Lineage. Every registered model links to the exact Delta versions of its training and eval data and to the MLflow run that produced it.
• Approval. Promotion from candidate to champion requires a passing eval gate and a named approver.
• Licensing. The open base model's licence is recorded with the model and reviewed for your intended use.
• Retention. Intermediate checkpoints and generated datasets have an expiry policy; they can be large and may contain sensitive material.

Pitfalls we see repeatedly

• Notebook-only pipelines. A run that only exists in someone's notebook can't be reproduced. Move data preparation and training into scheduled jobs early.
• Eval sets assembled from training data. Random splits of a de-duplicated table still leak templates and near-duplicates; prefer temporal or source-based splits.
• Under-provisioned GPU quota. Requesting GPU capacity late is the most common cause of schedule slip. Confirm quota in the target region before committing to a timeline.
• No path back from production. Without inference tables and a review process, the model never improves after launch.

Where POFE fits

You can assemble all of the above yourself. POFE gives you the assembled version, deployed inside your Databricks workspace:

• Native Unity Catalog data access and permissions — no copies outside the lakehouse.
• Tuned recipes for SFT, DPO, KTO and GRPO across 200+ open models.
• Eval suites with judge calibration, contamination scans and release gates.
• MLflow tracking and UC model registration out of the box.
• Optional burst to external GPU capacity when your workspace quota runs out.

Talk to us about a Databricks deployment, or try the workflow in the interactive demo.

Frequently asked questions