{ "module": "FT02 — The Open Spectrum: Weights, Data, and Trust", "course": "3 — LLM Fine-Tuning Masterclass", "version": "1.0.0", "duration_minutes": 40, "total_questions": 15, "bloom_distribution": { "target": "20% recall / 40% application / 40% analysis-design", "actual": { "recall": 3, "application": 6, "analysis": 6 } }, "passing_score_percent": 70, "questions": [ { "id": "Q01", "bloom": "recall", "type": "multiple_choice", "prompt": "What are the three tiers of model openness, from least to most open?", "options": [ "Closed → proprietary → commercial", "Open-weights-only → open-data → open-recipe", "Open-data → open-weights → open-source", "Permissive → copyleft → proprietary" ], "answer_index": 1, "rationale": "Tier 1 open-weights-only (weights, no data — Llama 3.x). Tier 2 open-data (weights + corpus/pipeline — MiniCPM, OLMo, Tulu, SmolLM3). Tier 3 open-recipe (weights + data + full code/config — OLMo, Tulu 3, SmolLM3). Each tier adds a trust property: use → audit → reproduce." }, { "id": "Q02", "bloom": "recall", "type": "multiple_choice", "prompt": "When and where was the OSI's Open Source AI Definition (OSAID) v1.0 released?", "options": [ "July 2024, by the NTIA in Washington DC.", "October 28, 2024, at the All Things Open (ATO) conference.", "December 2023, at NeurIPS.", "March 2025, by the EU AI Office." ], "answer_index": 1, "rationale": "OSAID v1.0 was released by the Open Source Initiative on 2024-10-28 at the All Things Open conference in Raleigh, NC. It is the first industry-standardized definition of 'open source' for AI systems. (The NTIA report is a separate July 2024 U.S. government document.)" }, { "id": "Q03", "bloom": "recall", "type": "multiple_choice", "prompt": "According to the NTIA 2024 Open-Model Weights Report, what security benefit do open-weight models provide?", "options": [ "They are immune to adversarial attacks because the weights are public.", "They allow firms, researchers, and users to use potentially sensitive data locally and on-premises.", "They guarantee no personally identifiable information is in the training set.", "They eliminate the need for any harness or policy gates." ], "answer_index": 1, "rationale": "The NTIA report states open-weight models 'provide security benefits by allowing firms, researchers, and users to use potentially sensitive data' locally / on-premises. This is the single best government citation for the on-prem/air-gapped argument — sensitive data stays inside your trust boundary. It does not claim immunity to attacks or PII guarantees." }, { "id": "Q04", "bloom": "application", "type": "multiple_choice", "prompt": "You are selecting a base model for a HIPAA-covered clinical deployment. Which tier MUST the release occupy to be auditable (so a security officer can prove what the model saw)?", "options": [ "Open-weights-only is sufficient — the weights let you run it on-prem.", "Open-data or open-recipe — you need the training corpus or a reproducible pipeline to audit data lineage.", "Any tier works, as long as the model card describes the data.", "Closed is fine if the vendor signs a BAA." ], "answer_index": 1, "rationale": "Auditability requires being able to prove what the model saw — which needs the training corpus or a reproducible data pipeline (open-data/open-recipe). Weights-only lets you run on-prem (the NTIA deployment argument) but cannot answer 'what did it see?'; a model-card description is not an audit; a BAA governs data handling, not model provenance." }, { "id": "Q05", "bloom": "application", "type": "multiple_choice", "prompt": "A vendor says their model is 'OSAID-compliant, therefore reproducible.' What is wrong with this statement?", "options": [ "Nothing — OSAID compliance guarantees reproducibility by definition.", "OSAID requires only 'sufficiently detailed information about' the data, not the data itself, so a compliant release can be non-reproducible. Compliance and reproducibility are different claims.", "OSAID compliance actually forbids reproducibility.", "The vendor means OLMo-compliant, which is a different standard." ], "answer_index": 1, "rationale": "This is the OSAID gap. OSAID's data clause requires 'sufficiently detailed information about the data' (sources, composition, processing) — not the data itself, and not an open license on the data. A Llama-style release (weights + described-in-aggregate data) can be compliant without being reproducible. 'OSAID-compliant' and 'reproducible/auditable' must never be conflated." }, { "id": "Q06", "bloom": "application", "type": "multiple_choice", "prompt": "Classify Meta's Llama 3.1-405B on the open spectrum.", "options": [ "Open-recipe — full data, code, and checkpoints released.", "Open-data — the training corpus is shipped.", "Open-weights-only — weights under the Llama Community License; data described in aggregate, recipe summarized, not reproducible.", "Closed — weights are withheld and only an API is offered." ], "answer_index": 2, "rationale": "Llama 3.1 ships weights under the Llama Community License (with 700M-MAU and named-product restrictions) but the corpus is described only in aggregate ('~15T tokens') and the post-training recipe is summarized, not reproducible. It is the canonical open-weights-only release. The weights are public (so not closed), but neither data nor full recipe is shipped." }, { "id": "Q07", "bloom": "application", "type": "multiple_choice", "prompt": "Classify OpenAI's GPT-4o on the open spectrum.", "options": [ "Open-weights-only — weights public under a permissive license.", "Open-data — training data documented in the system card.", "Closed — weights withheld, API-only, proprietary license; the system card is a risk document, not a reproducibility document.", "Open-recipe — full training code and data released." ], "answer_index": 2, "rationale": "GPT-4o is closed: weights are withheld, access is API-only under a proprietary license. The system card describes risks, not a reproducible training recipe. It cannot run on-prem (failing the NTIA deployment argument) and cannot answer provenance questions — the strict opposite of the open-recipe tier." }, { "id": "Q08", "bloom": "application", "type": "multiple_choice", "prompt": "A team picks a 'merged' model from the Hugging Face Hub (\"a merge of two bases, quantized, re-uploaded\") for a regulated deployment. What is the problem?", "options": [ "Merges are always lower quality than the original bases.", "It likely has no documented data lineage, no evals, and a license the uploader probably cannot grant. Without provenance back to original bases and data, it is unauditable.", "Merges are illegal under OSAID.", "There is no problem — community merges are implicitly trusted." ], "answer_index": 1, "rationale": "The 'trusting a community merge with no provenance' anti-pattern. A model card saying 'merge of X and Y, works great' is not an audit. In sensitive deployment you must trace provenance to the original bases and data; if you cannot, the release is unauditable regardless of how well it performs. Quality is FT03's concern; trust/provenance is FT02's." }, { "id": "Q09", "bloom": "application", "type": "multiple_choice", "prompt": "Which OpenBMB artifacts make MiniCPM an 'on-ramp hero' for a course that needs an auditable, consumer-hardware-friendly base?", "options": [ "It is closed-source but free to use via API.", "It is open-data/open-recipe under Apache-2.0, small enough to run and fine-tune on consumer hardware, and ships reusable datasets (UltraChat, UltraFeedback, Ultra-FineWeb).", "It is the largest open model available, beating GPT-4.", "It ships only weights, with no documentation." ], "answer_index": 1, "rationale": "MiniCPM (OpenBMB = Tsinghua + ModelBest) sits in a sweet spot: open-data/open-recipe under Apache-2.0, small enough (MiniCPM5-1B etc.) for consumer hardware, and accompanied by reusable open datasets (UltraChat, UltraFeedback, Ultra-FineWeb). You can point at every byte the model saw — the auditable base the early modules load." }, { "id": "Q10", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why is 'it's open, so it's safe' a false inference for a regulated deployment?", "options": [ "Because open models are always less capable than closed ones.", "Because openness is auditability, not safety. An open corpus can still contain PII, copyrighted text, poisoned examples, or liability-creating material. Openness gives the ability to vet; it does not do the vetting.", "Because open licenses forbid commercial use.", "Because OSAID explicitly forbids open models in healthcare." ], "answer_index": 1, "rationale": "Open data is auditable, not safe. An open corpus can still contain PII, copyrighted text, data-poisoning examples, or material that creates its own liability. Openness gives you the ability to inspect; it does not perform the inspection. Open bases still require a data audit (FT04–FT07), PII sweep, licensing review, and red-teaming." }, { "id": "Q11", "bloom": "analysis", "type": "multiple_choice", "prompt": "The Stanford FMTI (May 2024) found a mean score of 58/100 and a top score of 85/100 across 100 indicators. What is the most defensible reading of these numbers for a model-selection decision?", "options": [ "All foundation models are equally transparent, so openness does not matter.", "Transparency is measurable and tiered: fully-open developers cluster at the top, weights-only in the middle, closed at the bottom. The FMTI turns openness into a citable procurement criterion, and the tier you choose roughly predicts the score you can defend.", "The FMTI proves closed models are more transparent than open ones.", "The FMTI is obsolete and should never be cited." ], "answer_index": 1, "rationale": "The FMTI turns openness from a slogan into a number. The 58 mean and 85 top are less important than the structural finding: the index tracks the openness tier cleanly — fully-open developers (OLMo, IBM Granite) score highest; closed developers (OpenAI ~49, Anthropic ~51) below the mean. It is the bridge from 'openness is nice' to 'openness is a procurement criterion.'" }, { "id": "Q12", "bloom": "analysis", "type": "multiple_choice", "prompt": "A clinical system has run a closed vendor model for two years. The vendor silently updates the model and behavior changes under the validated pipeline. What property does this violate, and which openness tier prevents it?", "options": [ "It violates auditability; open-weights-only prevents it because the weights are public.", "It violates reproducibility (no 'silent drift'); open-recipe prevents it because you can pin the exact commit of data+code+weights and rebuild on demand, proving the validated model is the running model.", "It violates safety; abliteration prevents it.", "It violates the harness boundary; no openness tier addresses it." ], "answer_index": 1, "rationale": "This is silent drift. Reproducibility means being able to rebuild the model from a pinned commit and prove the model you validated is the one you are running. Open-recipe releases (data + code + weights + checkpoints) make this possible; open-weights-only does not (you can pin weights but not their provenance or a rebuild path), and closed releases make it impossible (the vendor controls the model)." }, { "id": "Q13", "bloom": "analysis", "type": "multiple_choice", "prompt": "An air-gapped government environment (IL6) is evaluating model bases. Why does the NTIA report's argument make a closed API model structurally unusable there, regardless of its capability?", "options": [ "Because closed models are always less capable than open ones.", "Because the closed model requires every inference to leave the trust boundary via a vendor API — which cannot operate inside a SCIF/air-gap. Open weights are the prerequisite for the model to run inside the boundary at all; open-data/recipe additionally answers 'what did it see?'.", "Because OSAID forbids closed models in government.", "Because closed models do not support quantization." ], "answer_index": 1, "rationale": "The NTIA argument is mechanical: if weights are not available, the model can only run via an external API, and every inference leaves the trust boundary. An air-gapped/SCIF environment physically cannot reach that API. Open weights are the deployment prerequisite (the model runs inside the boundary); open-data/recipe is the auditability prerequisite (you can prove the data lineage). Capability is irrelevant if the model cannot run there at all." }, { "id": "Q14", "bloom": "analysis", "type": "multiple_choice", "prompt": "Why did the OSI deliberately write OSAID's data clause as 'sufficiently detailed information about the data' rather than requiring the data itself, and what is the cost of that choice?", "options": [ "Because the OSI wanted to mislead users about openness.", "Because requiring full corpus release was impractical — much training data is private, licensed, or contains personal information that cannot legally be redistributed. The cost: a Llama-style release (weights + aggregate description, no corpus) can claim openness without being reproducible or auditable end to end.", "Because the data is never useful for reproducibility anyway.", "Because the OSI does not believe in data transparency." ], "answer_index": 1, "rationale": "The compromise was deliberate and pragmatic: requiring full corpus release would have excluded every frontier-scale release, since much training data is private, licensed, or legally non-redistributable. The cost is the OSAID gap — a release can be compliant (weights + code + 'sufficiently detailed information about' the data) without being reproducible or auditable. Critics (Software Freedom Conservancy, Hugging Face leaders) argue this erodes OSD's 'no hidden inputs' principle." }, { "id": "Q15", "bloom": "analysis", "type": "multiple_choice", "prompt": "Rank OLMo-2, Llama-3.1-405B, and GPT-4o by auditability for a regulated deployment, and state the structural reason for the ordering.", "options": [ "GPT-4o > Llama-3.1 > OLMo-2, because closed vendors have the most rigorous internal audits.", "OLMo-2 > Llama-3.1-405B > GPT-4o. OLMo-2 is open-recipe (data+code+checkpoints, auditable). Llama-3.1 is open-weights-only (runs on-prem but cannot answer provenance). GPT-4o is closed (cannot run on-prem at all). Each step down removes a trust property.", "They are equally auditable because all publish model cards.", "Llama-3.1 > OLMo-2 > GPT-4o, because Llama has the largest parameter count." ], "answer_index": 1, "rationale": "OLMo-2 (open-recipe: data + code + eval + checkpoints, Apache-2.0) is fully auditable — a skilled person can rebuild it and a security team can trace every input. Llama-3.1-405B (open-weights-only) can run on-prem (satisfying the NTIA deployment argument) but its aggregate-only data description means the provenance question is unanswerable from primary sources. GPT-4o (closed) fails even the deployment predicate — it cannot run inside a trust boundary at all. Each step down the tier ladder removes a distinct trust property." } ] }