NUUBI
Back to Blog

How to Run an AI Pilot in One University Course

· Nuubi Team

A single-course pilot lets a university test whether an AI tool fits a real teaching need before committing budget, support staff and policy attention to a wider rollout. Privacy and security reviews set the conditions for using a tool, yet they leave the purchasing question open: what evidence would justify expanding the pilot, changing it or stopping it? A useful pilot defines that decision before the first student uses the tool and then gathers enough evidence to make it, whether the institution is a large research university or a small college.

The Teaching Problem, Owner and Decision Date

Start with one teaching problem tied to an existing learning objective, then record how the course handles it today. How do students receive feedback, how long does it take, and where does the process break down? That workflow is the baseline. Set the course, assignments and exclusions, name an owner and fix a decision date.

TLC Science at the University of Amsterdam publishes a GenAI pilot plan template covering scope, learning outcomes, evaluation, risks, privacy and accessibility. Nuubi's post on where AI adds real value in a course can help narrow the problem. Choose an instructor and course that reasonably resemble the intended rollout: an unusually skilled early adopter with extra support may produce results colleagues cannot reproduce.

A Hypothetical Graduate Research-Methods Pilot

Consider a hypothetical graduate research-methods course in which students write short explanations of methods, such as when a matched design is appropriate or what a confidence interval does and does not claim. Today they submit once and receive comments about a week later, after the assignment has closed. The objective is that students can explain a method's purpose, assumptions and limitations accurately in their own words, and the pilot covers four of these assignments in one section while excluding exams, the final project and any AI grading.

For each assignment, the student first writes an attempt without AI. A course-grounded assistant then responds by asking questions, identifying gaps and pointing to relevant readings, with instructions to avoid rewriting the explanation, supplying an answer or assigning a score. Those instructions need testing; they are not a guarantee of how the assistant will respond. The student revises and adds a short note on which feedback they used or rejected and why, and the instructor grades the revision under the existing rubric with the first attempt available for comparison.

Grading the revision replaces an existing task, but comparing drafts and revision notes can still add work. The instructor should log that time alongside spot-checks, student questions and class-wide corrections, against a sample of current feedback time recorded before launch. When a student reports incorrect feedback through a named course support channel, the instructor responds within a stated window, posts a correction if others could be affected and lets affected students fix their work before grading. A recurring error leads to a change in course materials or configuration, and a serious one pauses the activity.

Jisc's interim findings from AI marking and feedback pilots across 38 UK colleges and universities offer a useful caution: keeping humans in the loop was hard to sustain in practice. Parallel marking helped, as did clear communication with students. Formative assessment emerged as the most effective low-risk use case in those pilots. These setting-specific findings do not establish that all AI feedback is low risk or improves learning.

Institutional Review at Pilot Scale

Even a small pilot should address four areas with the relevant offices: privacy and security; contract terms, including an end date, cancellation rights and data handling at exit; accessibility and equivalent access; and department or program approval for the teaching change. EDUCAUSE describes AI procurement as shared work among IT and cybersecurity, procurement, general counsel and privacy staff, and the leaders it interviewed favored shorter commitments while the market keeps changing. The pilot team should bring its plan to these offices and let them set the requirements, and the student-data privacy review should describe the actual data flow and access rules rather than rely on a vendor's general assurances.

If the evaluation is intended as research, or its status is unclear, the team should consult the research office or IRB before collecting data. Plans to publish are worth raising early, although intent to publish does not by itself make a project research; the answer depends on the design and on institutional policy.

Student Expectations Before Launch

The syllabus and a first-week session should explain which AI help is allowed (feedback on the student's own draft) and which is prohibited (generating the explanation or submitting AI-written text), who can see AI interactions, what is stored and for how long, and how AI use is disclosed, which in this example happens through the revision note. Students should also know where to get technical and academic support, have an accessible alternative with comparable timing, such as instructor or peer feedback on the first attempt, and receive institution-funded access rather than being required to buy a personal AI subscription. The guidance should also distinguish required activities from optional ones and explain how the work is assessed. Because weighing feedback selectively is part of being an AI-fluent student, the revision note can be presented as a learning task in its own right.

Preflight Testing Under Realistic Conditions

Before students arrive, the instructor and a support colleague should test the configured tool with a modest set of prompts that includes typical drafts, a draft with a deliberate methodological error, requests for the assistant to write the explanation outright, questions outside the syllabus and a prompt containing synthetic sensitive information. They should also walk through a login or screen-reader problem to confirm that the escalation route and response window promised to students work as described.

The voluntary NIST AI RMF Playbook's MEASURE function offers a related structure, including metrics suited to the context of use, acceptable performance limits defined in advance, tracking of errors, comparison of performance before and after deployment, and testing under realistic conditions with human review. The Playbook was written for AI systems generally, and its logic translates well to a single course.

Evidence Beyond Usage Counts

Usage counts measure adoption only, and frequent use can coexist with weak learning. Evidence of reasoning comes from student work: sampled first attempts and revisions scored on the existing rubric, short oral explanations from a subset of students, and a brief explanation task completed without AI early and late in the term. Error data comes from the flag log and spot-checks, time data from instructor and support logs, and access data from support tickets, use of the alternative path and a short survey. None of this requires capturing every private chat, and identifiable chats should not be repurposed for research without approval.

The decision rules and their thresholds belong to the institution and should be set before launch. In this example, expansion would require that participation in the revision step meets an agreed level across student groups and that reasoning gains hold on the no-AI task. Error above an agreed rate would trigger a change in materials or configuration, recurring staff time above an agreed ceiling would trigger a change or a stop, and an unresolved access barrier or data incident would pause or stop the pilot regardless of usage.

Interpretation calls for restraint, because grades and comparisons with earlier cohorts cannot on their own show that AI caused a learning gain when students, assessments and teaching change between terms. One course can show whether the tool fits locally, and stronger causal claims need a planned design with comparison groups, adequate sample size and attention to other explanations.

A Proportionate Plan for Large and Small Institutions

Where several departments are piloting tools, common evidence categories help leaders compare findings without imposing identical targets on different courses. Where staff hold several roles, use lighter paperwork while keeping the baseline, correction process, accessible alternative, exit terms and decision date. The EDUCAUSE procurement research also notes smaller institutions' difficulties engaging vendors and negotiating pricing, another reason to bound the commitment before expansion.

Nuubi in a Course Pilot

Nuubi brings course questions into a shared space where the AI assistant, classmates and faculty can respond. It also offers private, rubric-aware draft feedback before submission. A pilot can evaluate one of these workflows without assuming that the whole platform must be adopted at once. The institution should document the visibility and data handling of the chosen activity, then judge its value through student work and the full human feedback cycle rather than activity counts alone.

The Expand, Change or Stop Decision

An illustrative 15-week term could put reviews, testing and baseline timing before classes, a midterm check around week eight, and the decision after final grades. Other calendars can use the same sequence.

Define the next step before launch: success might justify one more section, not a campus-wide rollout. An instructor who owns the pilot should agree thresholds with a department or teaching-and-learning colleague.

At the decision date, expand if the evidence meets those thresholds, including independent reasoning, manageable error and workload, and adequate access. Change the design if the need is real but evidence is mixed, such as useful feedback with excessive correction time. Stop if the problem remains unaddressed, costs exceed the ceiling or a safety or access failure remains unresolved.

Count licensing, onboarding and exit work, including data deletion, required record retention and a return to the previous workflow. Separate setup from recurring work. Saved minutes are recovered capacity; they become budget savings only when spending falls. Compare the result with a non-AI option, such as clearer course guidance or a redesigned feedback process.

UNESCO's guidance on generative AI in education and research emphasizes human agency, privacy and pedagogical and ethical suitability. A pilot that keeps the instructor responsible for judgment and grading, protects student data and tests fit against a real learning objective gives an institution a defensible basis for whichever of the three outcomes it reaches.