Beyond the Model: AI Privacy, Safety, and Security for Students
It's late in the semester, and a professor has forty essays left to grade. She pastes an especially rough submission into a chatbot for help with feedback and gets something useful in seconds. What happens to that student's writing next depends on details she might not have considered: where the text goes, what gets stored, and whether it could train a future model.
She isn't the only one with questions. Many institutions are still working through those questions. In EDUCAUSE's 2025 AI Landscape Study, only 9% of higher education respondents said their cybersecurity and privacy policies were adequate for AI-related risks.
Most of what campus leaders read about AI privacy comes from policy and compliance. I see it from the other side of the table. I build AI tools for higher education, have advised companies on deploying AI, and teach graduate data science courses on how the models work.
From there, the risk looks different. Campus conversations about AI usually start with the model and which vendor seems safest. That comparison tells you less than people expect. Very few failures I've seen started in the model. Most started in the system around it: what data flows in, what the tool can access, and which guardrails sit on top.
This article follows one student essay through that system, from a US higher education perspective. It covers privacy (where student data goes), security (who can get in), and safety (what the tool does to the student using it).
What Goes Into the Model Determines What Can Come Out
The essay's first stop is the model, and what travels with it matters more than which model receives it. A course assistant needs the student's question and the relevant course material. It rarely needs a name, ID, grades, or history. Identifiers can be swapped for placeholder tokens before anything reaches the model and mapped back afterward, so the model never learns who it's talking to. Every field left out is one that can't leak.
The prompt is rarely the essay's last stop. Many tools store posts and submissions as embeddings in a vector database for later search, and those embeddings derive directly from the original writing. Application logs often capture full prompts and responses, and third-party monitoring services can receive copies too. Each copy must be secured, retained on a schedule, and eventually deleted.
Inputs shape fairness as much as privacy. A 2026 Stanford study found that four widely used LLMs shifted their writing feedback along stereotyped lines based on presumed student attributes, even when the essay was identical. Students described by race, language, or disability often got more praise and less substantive critique, and English language learners got feedback on grammar over ideas.
That extra praise came at the cost of the critique that would have improved their writing. The study used eighth-grade essays, but the mechanism doesn't depend on age. If a tool can give useful feedback without a student's demographic profile, it should never receive one. Data minimization protects privacy, and it also works as a practical safeguard against bias.
Context and Training Are Different Commitments
Once the essay reaches a tool, it can be used in two very different ways. Context, such as a syllabus, lecture notes, or a student's draft, may be supplied when someone asks a question and used for that response. Training changes the model itself, and there is no reliable way to undo it. That choice is made in the vendor's pipeline and contract, long before the model sees anything.
That permanence matters most for faculty intellectual property. Non-tenure-track faculty at Loyola Marymount University have proposed contract language barring the university from using their course materials to train AI. Grounding a course assistant in a professor's materials is sound design. Training on them is a separate agreement faculty should make explicitly.
Student work deserves the same line. A draft submitted for feedback should be used for that feedback. A vendor that uses customer data to "improve the product" should be able to say which of the two it means. Material used as context can be deleted when the course ends, while anything used for training can't.
The Model Is One Layer of a Larger System
When people say "AI models," they usually mean products like ChatGPT or Claude, which wrap the model in system instructions, moderation filters, retrieval, memory, logging, and escalation rules. Most of the protections live in those layers. That distinction often gets lost in higher ed conversations, yet it shapes nearly every privacy, security, and safety outcome.
Mental health shows what's at stake. In a study co-led by Mass General Brigham, 18% of surveyed college students reported using AI for mental health, and those with moderate or severe depression, severe anxiety, or suicidality were roughly twice as likely to do so. The students most likely to turn to AI are the ones a poor response could hurt most. The authors recommend that AI platforms embed crisis detection and referral. Those disclosures can surface anywhere, including a course assistant at 2 a.m. before an exam.
Major providers build crisis responses into their consumer products. ChatGPT directs users expressing suicidal intent to crisis hotlines. Claude.ai runs a separate classifier that watches for signs of suicide or self-harm risk and displays a banner linking to crisis helplines. That classifier sits outside the model, and it shows what a mature product brings: protections someone has already built, tested, and maintained. An institution building on a raw model inherits none of them. Someone on campus then has to decide what the tool does when a student discloses distress, who gets notified, and how that squares with the student's privacy.
Mature products carry these layers for a reason. Safety built into a model can be stripped away, sometimes by accident. Research on open-weight models shows guardrails can be removed with relatively little computing effort, often through fine-tuning. A Princeton-led team at ICLR 2024 found that even fine-tuning on benign, commonly used datasets degraded safety alignment, though to a lesser extent. A university that fine-tunes a model on its own course materials could end up with a tool that refuses harmful requests less reliably than the original.
So critical protections belong outside the model. A filter that strips personal information before a prompt is sent, a rule that flags possible distress for human follow-up, and a permission check that blocks certain agent actions work the same way however the model behaves. Prompts can be overridden and safety training can erode. Code around the model holds steady as long as someone maintains it.
Open-weight models offer real advantages, including keeping data on infrastructure the institution controls. They also hand every protective layer to whoever deploys them. The question is whether the institution can carry that for the life of the tool.
Agents Turn Exposed Data Into Actions
So far, the essay has only produced text. Agents can act on it. They can be programmed to read files, browse the web, call other tools, and act on a user's behalf, so a data exposure can become a sent message or a moved file before anyone notices.
In July, OpenAI disclosed that one of its pre-release models breached Hugging Face's systems during internal testing. Security experts who spoke with TechCrunch traced the attack to a sandbox OpenAI had failed to isolate from the internet. If containment can fail at a leading AI lab, no campus should assume it's safe by default.
On campus, the likelier threat is prompt injection. Researchers have shown that instructions hidden in a student's submission can sway an LLM grader, and a student could just as easily try to trick a homework tutor into giving away the answers. Either way, student text becomes the tool's instructions. Any tool that reads student work needs to be designed and tested for it.
The defenses sit outside the model: contained environments, tight permissions, reviewed logs, and alerts that notify a person when something goes wrong. Of these, permissions do the most to limit how far a single mistake can travel.
Permissions Determine What a Mistake Can Reach
Data minimization limits what a tool knows. Least privilege limits what it can do. Each tool, and each agent within it, should hold only the access its job requires. A course assistant can read course materials without seeing the gradebook, and a tool that drafts messages to students has no need to send them. Stanford now requires scoped, least-privilege credentials and sandboxing for AI agents that automate internal university workflows. When something goes wrong, the damage stops at whatever the tool could reach.
The safest designs separate sensitive data from the ability to act. The component that reads student records shouldn't be the one that sends emails or changes grades. Actions with real consequences, such as notifying a student, updating a record, or submitting a form, should wait for a person to confirm them, and any attempted restricted action should trigger an immediate alert. Stanford's guidance lists external communications, permission changes, deletions, and updates to systems of record among the actions that require human approval.
Good design only holds if someone keeps checking it. Before launch, a tool should be stress tested against the inputs that break things in practice: hidden instructions, students in distress, and requests outside its purpose. After launch, activity logs and regular audits should confirm that permissions haven't quietly expanded and safeguards still work after each model update. When something goes wrong, there should be a plan for who gets notified and how quickly, including affected students. A vendor that can describe its last incident and what changed afterward is usually further along than one that claims it has never had one.
The Best Governance Questions Come From How These Tools Are Built
None of this is a reason to avoid AI tools. Having no institutional option carries its own risk. In EDUCAUSE's 2026 report on AI and work, 94% of higher education employees had used AI tools for work in the past six months, and 56% had used tools their institution didn't provide. That second number is the real exposure, because a personal account is a system the institution can't see into at all. Student work ends up there anyway, where conversations may train models by default and the institution has no contract and no way to delete anything. Even a basic institutional tool should give leaders a system they can inspect and improve.
Most institutions already have standard protections, such as approved tool lists and contract terms prohibiting training on student data. California's new AB 1159 makes the latter law for college students in the state starting in 2027, and institutions elsewhere can ask vendors for the same commitment in writing. Those protections are the easy part. A vendor can meet every one and still make design choices that create the problems above. Having built one of these tools, I've found the most revealing questions follow the path the essay took.
Start with what goes in. When a tool connects to the LMS through LTI or roster sync, ask which student fields it pulls and why. Integrations often request more than they use, and every extra field can potentially leak or end up in a prompt. Ask whether demographic information ever reaches the model. Given the Stanford findings, the answer for most instructional tools should be no, and the vendor should be able to show where that data stops.
Then follow the data. Ask what the tool retrieves when it builds a prompt, and whether anything from a student's record can end up there. Ask whether the model provider keeps prompts or operates under a zero-retention agreement. Ask where else student text lives, including embeddings, logs, and monitoring services, and how long each copy is kept.
Safety questions should start with what happens when a student discloses distress in a tutoring session, a discussion post, or a late-night question about an assignment. Ask what the tool says, whether anyone is notified, and how that respects the student's privacy. If the model was fine-tuned, ask whether its safety was re-tested. Ask what happens after an incident: who is notified, how quickly, and whether affected students are told.
Finally, ask about permissions. What can the tool's credentials reach? What can it do without human approval? How does it handle instructions embedded in content it reads? Any tool that grades or summarizes submissions should be tested against hidden prompts before launch and after every model update.
A team that understands its system can answer all of this in one conversation. A vendor that can't has a gap to close before the contract is signed.
In my last article, I argued that AI initiatives deliver a measurable return when tied to a specific use case. The same specificity makes a tool reviewable. "We're deploying AI" gives a privacy review nothing to evaluate. A course assistant that reads the syllabus and lecture notes, never sees grades, and routes distress disclosures to the counseling center has clear boundaries to check. Privacy, security, and safety are properties of the system around the model, and that system only comes into focus once someone defines what the tool is for.
Sources
Robert, J., & McCormack, M. (2025). 2025 EDUCAUSE AI Landscape Study: Into the Digital AI Divide. EDUCAUSE. Summary via InfoDocket: https://www.infodocket.com/2025/02/15/report-2025-educause-ai-landscape-study-into-the-digital-ai-divide/
Robert, J. (2026, January 12). The Impact of AI on Work in Higher Education. EDUCAUSE. Conducted in partnership with AIR, NACUBO, and CUPA-HR. https://www.educause.edu/research/2026/the-impact-of-ai-on-work-in-higher-education
SEIU Local 721, LMU Non-Tenure-Track Faculty Bargaining Team. (2025, August 13). "Article Highlight: Artificial Intelligence and Intellectual Property." https://www.seiu721.org/2025/08/lmu-ntt-article-highlight-ai-ip.php
Tan, M., Phalen, L., & Demszky, D. (2026). "Marked Pedagogies: Examining Linguistic Biases in Personalized Automated Writing Feedback." LAK 2026. (600 eighth-grade essays from the PERSUADE dataset; K-12 data.) https://arxiv.org/abs/2603.12471
Mass General Brigham. (2026). "Study Finds Students with Highest Distress Use AI for Mental Health at Elevated Rates," reporting on "Clinical and sociodemographic predictors of AI use for mental health among college students," Journal of Affective Disorders, DOI: 10.1016/j.jad.2026.122058. https://www.massgeneralbrigham.org/en/about/newsroom/press-releases/study-finds-highly-distressed-students-use-ai-for-mental-health
Axios. (2025, September 2). "OpenAI to safeguard ChatGPT for teens and people in crisis." https://www.axios.com/2025/09/02/chatgpt-openai-mental-health-teens
Anthropic. (2025, December 18). "Protecting the wellbeing of our users." https://www.anthropic.com/news/protecting-well-being-of-users
Dombrowski, A.-K., Bowen, D., Gleave, A., & Cundy, C. (2025). "The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models." FAR.AI. arXiv:2507.11544. https://arxiv.org/abs/2507.11544
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R., Mittal, P., & Henderson, P. (2024). "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" ICLR 2024. https://arxiv.org/abs/2310.03693
Franceschi-Bicchierai, L. (2026, July 22). "How OpenAI's human mistake led to the AI-powered hack on Hugging Face." TechCrunch. https://techcrunch.com/2026/07/22/how-an-openais-human-mistake-led-to-the-ai-powered-hack-on-hugging-face/
Milani, A., Franzoni, V., Florindi, E., Omarbekova, A., Bekmanova, G., & Yergesh, B. (2025). "When AI Is Fooled: Hidden Risks in LLM-Assisted Grading." Education Sciences, 15(11), 1419. https://doi.org/10.3390/educsci15111419
Stanford University IT. (n.d.). "Know Your Risks: Agentic AI Risk Classifications." https://uit.stanford.edu/security/agenticai/riskclassifications
Echelman, A. (2026, September 11). "Tech companies are selling kids' data. A new California law aims to protect their privacy." CalMatters. https://calmatters.org/economy/technology/2026/09/students-data-california/