Course: Course 3 — LLM Fine-Tuning Masterclass Module: FT02 — The Open Spectrum: Weights, Data, and Trust Duration: 30–40 minutes (a research and classification lab — no GPU, no training) Environment: A web browser and a text editor. No Python, no GPU, no model download. This is a procurement-style audit.
By the end of this lab you will have:
This lab is deliberately compute-free. The skill it builds is reading a release for what it actually gives you — which is the precondition for every later module that loads, fine-tunes, or deploys a base.
Create a file ft02-openness-audit.md. Paste this table. You will fill one row per release across Phases 1–5.
| Field | Meaning |
|---|---|
| Release | Canonical name + the org. |
| Tier | One of: open-recipe / open-data / open-weights-only / closed. |
| License | The exact license name. Note field-of-use / MAU / named-product restrictions. |
| Weights public? | Yes / No. If Yes, note dtype/quantization options shipped. |
| Training data released? | One of: full corpus / reproducible pipeline / described only / none. Link it. |
| Recipe released? | One of: full (code + config + checkpoints) / partial / summarized / none. |
| OSAID-compliant? | Best-effort Yes/No with the one-line reason. (Not the same as auditable.) |
| Auditable? | Yes / Partial / No — the verdict a HIPAA/IL6 assessor would reach. |
| HIPAA verdict | One paragraph (3–5 sentences). See Phase 6 for the rubric. |
| Primary sources | URLs to the model card, license, and release blog/paper. |
Why this exact field set. Every field maps to a question a regulator can name. "Data released?" maps to auditability. "Recipe released?" maps to reproducibility (the no-silent-drift argument). "License restrictions?" maps to procurement legality. Do not drop fields because the release doesn't address them — an unanswered field is itself a finding.
Open the release. Use the model card and the OpenBMB project page.
https://huggingface.co/openbmb/MiniCPM5-1B and https://github.com/OpenBMB/MiniCPMRecord: the tier, the license (and whether it has field-of-use restrictions), the auditability verdict, and your source URLs.
https://huggingface.co/allenai/OLMo-2-1124-7B (or any OLMo-2 size), the paper at https://arxiv.org/abs/2501.00656, and https://allenai.org/olmo.Record: note specifically whether intermediate checkpoints are public (this is what distinguishes open-recipe from open-data). The OLMo 2 paper is the canonical "fully open" citation — capture the exact scope.
https://huggingface.co/meta-llama/Llama-3.1-405B and the Llama 3.1 model card / technical report.Record: this is the canonical weights-only release. Be precise about what is described (e.g., "~15T tokens," "multilingual, code, reasoning") versus what is reproducible (nothing in the data pipeline is). The gap between "described" and "reproducible" is the entire point of this row.
https://huggingface.co/blog/smollm3 and the SmolLM3 model card.Record: SmolLM3 is the cleanest small-model open-recipe example. Note the distinction between a blog describing the recipe and the actual code/data being downloadable — score on what is actually shipped, not what is described.
https://openai.com/index/gpt-4o-system-card/ (the system card) and the OpenAI terms of service.Record: this is the closed baseline. The point of including it is to make the contrast sharp: the same auditability column that reads "Yes" for OLMo reads "No" for GPT-4o, and the reason is structural (weights withheld), not a judgment about quality.
For each of the five releases, write a 3–5 sentence paragraph answering: "Can this release be the base model for a HIPAA-covered clinical deployment that must be auditable?" Use this rubric:
Write each verdict in the voice you would use in a procurement memo — factual, specific, citing the structural reason. Avoid "it's a great model" or "state of the art." Capability is FT03's concern; this lab is about trust.
Submit ft02-openness-audit.md containing:
The expected classifications (the deliverable is the table + verdicts, not just the tier labels):
| Release | Tier | License | Data released? | Recipe released? | OSAID-compliant? | Auditable? |
|---|---|---|---|---|---|---|
| MiniCPM5-1B | Open-recipe | Apache-2.0 | Yes (Ultra* datasets, Ultra-FineWeb) | Yes (code + config) | Yes | Yes |
| OLMo-2 | Open-recipe | Apache-2.0 | Yes | Yes (code + eval + checkpoints) | Yes | Yes |
| Llama-3.1-405B | Open-weights-only | Llama 3.1 Community License (700M-MAU + named-product) | Described only (aggregate) | Summarized | Borderline/compliant (per OSI's reading of the data clause) | No |
| SmolLM3 | Open-recipe | Apache-2.0 | Yes (named components) | Yes (pre/mid/post + synthetic) | Yes | Yes |
| GPT-4o | Closed | Proprietary (API-only) | None | None | No | No |
"Yes — auditable for HIPAA. OLMo-2 ships under Apache-2.0 with the full training data, training code, evaluation suite, and intermediate checkpoints released (Olson et al., arXiv:2501.00656), so the data lineage is inspectable end to end and the model can be rebuilt from a pinned commit, eliminating silent-drift risk. The weights are available, so the model can run on-premises / air-gapped — the security benefit the NTIA 2024 report attributes to open-weight releases. A covered entity can answer the assessor's question ('what did this model see?') from primary sources rather than the publisher's marketing. The remaining work is a data-level PII/licensing audit of the released corpus (FT04–FT07), not a provenance gap."
"No — not auditable for HIPAA as-is. Llama 3.1 ships weights under the Llama Community License (with its 700M-MAU and named-product restrictions, which must be cleared by procurement), but the training corpus is described only in aggregate ('~15T tokens') and the post-training recipe is summarized, not reproducible. A covered entity therefore cannot answer the assessor's question — 'was document X, or a regulated class of documents, in the training set?' — from primary sources; it can only cite Meta's description. The model can run on-prem (satisfying the NTIA deployment argument), but the auditability and reproducibility predicates are unmet. Choose an open-data/open-recipe base for the auditable deployment; reserve Llama for non-regulated workloads where the capability/auditability trade is acceptable."
The SmolLM3 and MiniCPM5-1B verdicts should mirror the OLMo shape (Yes — auditable, with a data-level vetting caveat). The GPT-4o verdict should be a clean "No" on structural grounds (weights withheld → cannot run on-prem → cannot satisfy the NTIA deployment pattern, regardless of data transparency).
https://crfm.stanford.edu/fmti/May-2024/index.html and score each release by hand. Compare your scores to the published FMTI numbers where available. (This is the bridge from "openness is a vibe" to "openness is a procurement criterion.")# Lab Specification — Module FT02: The Openness Audit
**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT02 — The Open Spectrum: Weights, Data, and Trust
**Duration**: 30–40 minutes (a research and classification lab — no GPU, no training)
**Environment**: A web browser and a text editor. No Python, no GPU, no model download. This is a procurement-style audit.
---
## Learning objectives
By the end of this lab you will have:
1. **Classified five real model releases** on the open spectrum (weights-only / open-data / open-recipe) using primary sources — model cards, license files, and release blogs — not marketing summaries.
2. **Produced the auditability verdict** for each release: can a HIPAA security officer prove what the model saw? (Yes / Partial / No, with the one-line reason.)
3. **Written the one-paragraph "can I audit this for a HIPAA deployment?" verdict** for each release, in the voice an assessor would accept.
4. **Internalized the audit template** as a reusable artifact — the same table you will fill in for every base-model selection decision in regulated work (sets up FT03 and FT21).
This lab is deliberately compute-free. The skill it builds is *reading a release for what it actually gives you* — which is the precondition for every later module that loads, fine-tunes, or deploys a base.
---
## Phase 0 — Set up the audit template (5 min)
Create a file `ft02-openness-audit.md`. Paste this table. You will fill one row per release across Phases 1–5.
### The Openness Audit template
| Field | Meaning |
| --- | --- |
| **Release** | Canonical name + the org. |
| **Tier** | One of: `open-recipe` / `open-data` / `open-weights-only` / `closed`. |
| **License** | The exact license name. Note field-of-use / MAU / named-product restrictions. |
| **Weights public?** | Yes / No. If Yes, note dtype/quantization options shipped. |
| **Training data released?** | One of: `full corpus` / `reproducible pipeline` / `described only` / `none`. Link it. |
| **Recipe released?** | One of: `full (code + config + checkpoints)` / `partial` / `summarized` / `none`. |
| **OSAID-compliant?** | Best-effort Yes/No with the one-line reason. (Not the same as auditable.) |
| **Auditable?** | `Yes` / `Partial` / `No` — the verdict a HIPAA/IL6 assessor would reach. |
| **HIPAA verdict** | One paragraph (3–5 sentences). See Phase 6 for the rubric. |
| **Primary sources** | URLs to the model card, license, and release blog/paper. |
> **Why this exact field set.** Every field maps to a question a regulator can name. "Data released?" maps to auditability. "Recipe released?" maps to reproducibility (the no-silent-drift argument). "License restrictions?" maps to procurement legality. Do not drop fields because the release doesn't address them — an unanswered field is itself a finding.
---
## Phase 1 — MiniCPM5-1B (OpenBMB) (5 min)
Open the release. Use the model card and the OpenBMB project page.
- **Where to look**: `https://huggingface.co/openbmb/MiniCPM5-1B` and `https://github.com/OpenBMB/MiniCPM`
- **Confirm**: the license; whether the pretraining/post-training data is named or shipped; whether the training code and config are public; the UltraChat / UltraFeedback / Ultra-FineWeb dataset lineage.
- **Fill the row.**
**Record**: the tier, the license (and whether it has field-of-use restrictions), the auditability verdict, and your source URLs.
---
## Phase 2 — OLMo-2 (Allen Institute for AI) (5 min)
- **Where to look**: `https://huggingface.co/allenai/OLMo-2-1124-7B` (or any OLMo-2 size), the paper at `https://arxiv.org/abs/2501.00656`, and `https://allenai.org/olmo`.
- **Confirm**: the license; whether data, code, eval, and intermediate checkpoints are released (the OLMo project calls itself "fully open" — verify the claim); what "fully open" means in their definition.
- **Fill the row.**
**Record**: note specifically whether intermediate checkpoints are public (this is what distinguishes open-recipe from open-data). The OLMo 2 paper is the canonical "fully open" citation — capture the exact scope.
---
## Phase 3 — Llama-3.1-405B (Meta) (5 min)
- **Where to look**: `https://huggingface.co/meta-llama/Llama-3.1-405B` and the Llama 3.1 model card / technical report.
- **Confirm**: the exact license (Llama 3.1 Community License) and its restrictions (the 700M-MAU clause, the named-products clause); whether the training corpus is shipped, described in aggregate, or withheld; whether the post-training recipe is reproducible or summarized.
- **Fill the row.**
**Record**: this is the canonical weights-only release. Be precise about *what is described* (e.g., "~15T tokens," "multilingual, code, reasoning") versus *what is reproducible* (nothing in the data pipeline is). The gap between "described" and "reproducible" is the entire point of this row.
---
## Phase 4 — SmolLM3 (Hugging Face) (5 min)
- **Where to look**: `https://huggingface.co/blog/smollm3` and the SmolLM3 model card.
- **Confirm**: the license; whether the release blog ships "the full training recipe including pre-training, mid-training, post-training, and synthetic data"; whether the data components are named and available.
- **Fill the row.**
**Record**: SmolLM3 is the cleanest small-model open-recipe example. Note the distinction between a *blog describing* the recipe and *the actual code/data* being downloadable — score on what is actually shipped, not what is described.
---
## Phase 5 — GPT-4o (OpenAI) (5 min)
- **Where to look**: `https://openai.com/index/gpt-4o-system-card/` (the system card) and the OpenAI terms of service.
- **Confirm**: whether weights are public (they are not); whether training data is described or shipped; whether the system card constitutes a recipe (it does not — it is a risk document, not a reproducibility document); the licensing posture (API-only, proprietary).
- **Fill the row.**
**Record**: this is the closed baseline. The point of including it is to make the contrast sharp: the same auditability column that reads "Yes" for OLMo reads "No" for GPT-4o, and the *reason* is structural (weights withheld), not a judgment about quality.
---
## Phase 6 — The HIPAA verdict (10 min)
For each of the five releases, write a 3–5 sentence paragraph answering: **"Can this release be the base model for a HIPAA-covered clinical deployment that must be auditable?"** Use this rubric:
- **A "Yes" verdict requires**: (a) weights available so the model can run on-prem/air-gapped (NTIA security-benefit argument); (b) training data released or reproducible so the data lineage is auditable; (c) recipe released so the model can be pinned and rebuilt (no silent drift); (d) a license that permits the deployment.
- **A "Partial" verdict**: weights available and on-prem deployable, but data or recipe gaps mean auditability is incomplete (e.g., a weights-only release that can run on-prem but cannot answer "what did it see?").
- **A "No" verdict**: weights withheld, OR data/recipe opaque to the point that the auditability question cannot be answered, OR a license that forbids the use.
Write each verdict in the voice you would use in a procurement memo — factual, specific, citing the structural reason. Avoid "it's a great model" or "state of the art." Capability is FT03's concern; this lab is about trust.
---
## Deliverables
Submit `ft02-openness-audit.md` containing:
- [ ] The completed audit table — one row per release (5 rows), every field filled or marked "not stated by publisher."
- [ ] Five HIPAA-verdict paragraphs (Phase 6), one per release.
- [ ] A 2–3 sentence closing reflection: which tier you would default to for a *regulated* deployment and why, in your own words.
---
## Solution key
The expected classifications (the deliverable is the table + verdicts, not just the tier labels):
| Release | Tier | License | Data released? | Recipe released? | OSAID-compliant? | Auditable? |
| --- | --- | --- | --- | --- | --- | --- |
| **MiniCPM5-1B** | Open-recipe | Apache-2.0 | Yes (Ultra* datasets, Ultra-FineWeb) | Yes (code + config) | Yes | **Yes** |
| **OLMo-2** | Open-recipe | Apache-2.0 | Yes | Yes (code + eval + checkpoints) | Yes | **Yes** |
| **Llama-3.1-405B** | Open-weights-only | Llama 3.1 Community License (700M-MAU + named-product) | Described only (aggregate) | Summarized | Borderline/compliant (per OSI's reading of the data clause) | **No** |
| **SmolLM3** | Open-recipe | Apache-2.0 | Yes (named components) | Yes (pre/mid/post + synthetic) | Yes | **Yes** |
| **GPT-4o** | Closed | Proprietary (API-only) | None | None | No | **No** |
### Expected HIPAA-verdict shape (one example, OLMo-2)
> "Yes — auditable for HIPAA. OLMo-2 ships under Apache-2.0 with the full training data, training code, evaluation suite, and intermediate checkpoints released (Olson et al., arXiv:2501.00656), so the data lineage is inspectable end to end and the model can be rebuilt from a pinned commit, eliminating silent-drift risk. The weights are available, so the model can run on-premises / air-gapped — the security benefit the NTIA 2024 report attributes to open-weight releases. A covered entity can answer the assessor's question ('what did this model see?') from primary sources rather than the publisher's marketing. The remaining work is a data-level PII/licensing audit of the released corpus (FT04–FT07), not a provenance gap."
### Expected HIPAA-verdict shape (one example, Llama-3.1-405B)
> "No — not auditable for HIPAA as-is. Llama 3.1 ships weights under the Llama Community License (with its 700M-MAU and named-product restrictions, which must be cleared by procurement), but the training corpus is described only in aggregate ('~15T tokens') and the post-training recipe is summarized, not reproducible. A covered entity therefore cannot answer the assessor's question — 'was document X, or a regulated class of documents, in the training set?' — from primary sources; it can only cite Meta's description. The model *can* run on-prem (satisfying the NTIA deployment argument), but the auditability and reproducibility predicates are unmet. Choose an open-data/open-recipe base for the auditable deployment; reserve Llama for non-regulated workloads where the capability/auditability trade is acceptable."
The SmolLM3 and MiniCPM5-1B verdicts should mirror the OLMo shape (Yes — auditable, with a data-level vetting caveat). The GPT-4o verdict should be a clean "No" on structural grounds (weights withheld → cannot run on-prem → cannot satisfy the NTIA deployment pattern, regardless of data transparency).
---
## Stretch goals
1. **Add a sixth release of your choice** — e.g., NVIDIA Nemotron, IBM Granite, Qwen, DeepSeek, or Phi. Run it through the same template. Note where the release is *partial* (e.g., post-training data documented but pretraining data not) and how that maps to the "Partial" auditability verdict.
2. **Score your five releases on the FMTI.** Pull 5–10 of the 100 FMTI indicators from `https://crfm.stanford.edu/fmti/May-2024/index.html` and score each release by hand. Compare your scores to the published FMTI numbers where available. (This is the bridge from "openness is a vibe" to "openness is a procurement criterion.")
3. **Write the OSAID-compliance check formally.** For each release, walk the three OSAID v1.0 requirements (code / "sufficiently detailed information about data" / use-modify-redistribute) and state Yes/No with evidence. Note specifically which releases are OSAID-compliant *without* being reproducible — that is the OSAID gap, demonstrated on real artifacts.