Questions to Ask Before Choosing an AI Tool for a University Course
If you work at a university and have been asked to evaluate an AI tool for courses, you have probably noticed that every demo looks about the same. The interface is clean, the chatbot answers fluently, and the vendor tells you it will improve learning. Then the demo ends and you still have to decide.
I've seen this from both sides. In my consulting work with Fortune 100 companies, the AI initiatives that lasted were tied to one team, one use case, and a problem someone was already struggling with. The polished demo was rarely what decided it.
This post is a set of questions I would bring to any vendor conversation, including one with us. They are meant to be useful whether or not you ever talk to Nuubi.
Define What Success Looks Like Before the Demo
Before comparing tools, write down what better would look like in a course. Maybe students ask more specific questions before an assignment is due. Maybe they stop using a general chatbot to write their drafts and start using it to check their reasoning. Maybe the instructor simply wants to see where students get stuck. Any of these is a fair goal. What matters is that you name it before the demo, because a tool will always look good against a goal that has not been defined.
A useful test: could you tell, at the end of one term, whether it worked? If the answer depends on the vendor's own dashboard and nothing you can observe in the course, push on that. Our post on the ROI of AI in higher ed goes further into what to measure.
Ask How the Tool Learns Your Course
A general chatbot knows a lot about a lot, but it does not know your syllabus, your rubric, or what you told students in week two. Ask how the tool learns about a specific course. Can the instructor load the materials? Can the instructor set expectations for how the tool should respond, for example by pointing students back to the reading rather than writing the answer for them? And when the course changes next term, how much work is it to update?
I learned this in my own course. The first version of Nuubi I piloted gave students a generic first answer because it wasn't yet grounded in the course materials. Most answers needed a follow-up from me, either confirming they were on track or explaining where they missed. That's a big reason Cuubi now answers from the materials faculty load.
Ask to see a live example using one of your own courses. You will learn more from how it handles a question the vendor did not anticipate.
Decide Between Standalone and LMS Integration
Some tools run on their own site and some connect to your learning management system. Neither is automatically better. A tool inside the LMS is easier for students to find and usually fits your single sign-on and roster setup. A standalone tool can often start sooner, because it does not wait on an integration review. The standard most vendors use to connect to an LMS is called LTI, maintained by 1EdTech, and your IT team will probably ask which version a vendor supports.
Find Out What Instructors and Students Can See
This question gets skipped, and it matters a great deal. Ask the vendor to show you both screens. What does an instructor see about how students used the tool: every conversation, a summary, nothing? What do students know about that visibility before they start typing? Students may well use these tools differently when they know someone can read the conversation, and they deserve to be told plainly what is shared.
In my own online course, the LMS discussion forum got zero student posts across an entire term, even though every student had access. Students emailed me instead, often late at night, and Monday mornings meant the same question asked several different ways. Once questions moved into a shared thread, students said they liked learning by watching classmates ask first, and many late-night questions were resolved by morning.
There is no single right answer here. Some programs want full transcripts and some want only patterns. What you want is a vendor who can explain the choice and show you where it is set.
Meet Procurement and IT Security Questions Early
Your procurement and IT security reviewers will have their own list, and it is better to meet it early. EDUCAUSE has published guidance on how institutions are approaching AI procurement and where the risks sit, and it is a good place to see the kinds of questions other schools are asking. Typical topics include where student data is stored, whether it is used to train models, how long it is kept, and how accessibility is documented.
Ask the vendor for their answers in writing, and be wary of anything that sounds certain but cannot be pointed to in a document. A vendor who says a thing is on the way is telling you it is not there yet, which is fine as long as everyone says so.
Plan for Setup Time and Support
A tool that needs an instructor to spend a weekend configuring it will not survive a busy term. Ask how long setup takes for a single course, who answers when a student cannot log in at 10 p.m., and what happens if the instructor leaves. These practical details often decide whether a pilot gets a fair try.
Pilot the Tool in One Course First
Whatever the vendor says, try it in one course before committing to more. Pick a course with an instructor who is curious, agree on what you will look at afterward, and keep the first run short. For an example of what this can look like, see our graduate course case study.
Start at the beginning of the term if you can. In one of my sections, I introduced Nuubi in week 3 of 8, and only 2 of 13 students became active. In a section where it was part of the course from day one, about two-thirds of students used it, with no grade attached. I wrote more about what that taught me in my post on the ROI of AI in higher ed.
Nuubi Should Answer These Questions Too
I build Nuubi, an AI-integrated tool that helps students build critical thinking as they use AI in their coursework, with feedback along the way and faculty in the loop on every interaction. It works inside an LMS or on its own. I would still want you to ask us every question above. To see how Nuubi answers them in a real course, request a demo at nuubi.com.
Sources
EDUCAUSE Review, AI Procurement in Higher Education: Benefits and Risks of Emerging Tools (March 2025): https://er.educause.edu/articles/2025/3/ai-procurement-in-higher-education-benefits-and-risks-of-emerging-tools
1EdTech, Learning Tools Interoperability (LTI): https://www.1edtech.org/standards/lti