Clinical text is some of the most valuable fine-tuning data that exists and some of the most constrained. The workable pattern is: de-identify before training, train inside a controlled environment, evaluate with clinicians in the loop, and pick tasks whose outputs can be checked.
De-identification under HIPAA#
The HIPAA Privacy Rule defines two routes to de-identified data in 45 CFR §164.514(b): Safe Harbor, which requires removing 18 categories of identifiers (names, geographic subdivisions smaller than a state, most date elements, phone and fax numbers, email addresses, record and account numbers, device identifiers, full-face photographs and others) and having no actual knowledge that the remainder could identify someone; and Expert Determination, where a qualified expert documents that re-identification risk is very small.
Free text is the hard part: identifiers appear in narrative notes in unpredictable forms. Automated de-identification should be measured like any model — recall on a labelled sample of notes — and followed by a memorisation check on the trained model's outputs. This is an engineering summary, not legal advice.
Use cases with checkable outputs#
- Medical coding (ICD-10-CM, CPT). SFT on coded encounters, DPO from coder corrections. Exact-match and hierarchical partial credit are deterministic metrics.
- Prior authorisation and documentation review. Extract the criteria evidence from notes; evaluate field-level recall.
- Clinical summarisation. Higher value, harder to evaluate; requires clinician-calibrated judges and conservative release gates.
- Trial matching. Classify eligibility criteria against patient records; recall matters most.
Evaluating clinical models#
Slice by specialty and by rare codes — the long tail is where coding models fail and where reimbursement errors cluster. Calibrate any LLM judge against clinician labels and report sensitivity and specificity; judges tend to be lenient on fluent but wrong clinical text (why κ isn't enough). Gate on safety-relevant errors separately from average accuracy.
Frequently asked questions#
Can I fine-tune an LLM on patient records?
Only under the applicable rules: de-identified data via Safe Harbor or Expert Determination, or protected health information within a compliant environment and agreements. Most teams de-identify first and train inside their own controlled infrastructure.
What are the 18 HIPAA identifiers?
They are the categories listed in 45 CFR 164.514(b)(2), including names, small geographic subdivisions, date elements except year, phone and fax numbers, email addresses, Social Security, medical record, health plan and account numbers, certificate and license numbers, vehicle and device identifiers, URLs and IP addresses, biometric identifiers, full-face photographs, and any other unique identifying number or code.
References#
- 45 CFR §164.514 — Other requirements relating to uses and disclosures of protected health information (de-identification). www.ecfr.gov/current/title-45/subtitle-A/subchapter-C/part-164/subpart-E/section-164.514