Get in touch
Have a project in mind? Tell us a bit about it.
The vendor evaluation for most AI tools happens in a single meeting: someone runs a demo, the room nods, and a contract gets signed a week later based on that fifteen minutes. Nobody asks what happens when the vendor changes the underlying model, what your data actually gets used for, or what “cancel anytime” really looks like six months in. By the time those questions matter, you’re already locked in, and asking them after signing is a very different conversation than asking them before.
None of this requires a legal team or a procurement department. It requires knowing which six questions actually separate a vendor you can build on from one that quietly becomes a liability. Here’s what should be on that list before you sign anything.
Ask exactly what happens to your data, not just where it’s stored
“We take data security seriously” is not an answer, it’s a sentence designed to end the question. Ask specifically: is your data used to train or fine-tune the vendor’s models for other customers, is it retained after you cancel, and can you get a straight answer on how long, in writing. A vendor that hedges or redirects you to a generic privacy policy page instead of answering directly is telling you something. A vendor that answers in one sentence, no caveats, usually means they’ve actually thought about it.
Ask who decides when the model underneath you changes
Every AI vendor is built on top of models that get updated, sometimes without much notice. A prompt or workflow tuned against one model version can behave differently after a silent upgrade. Ask whether you can pin a model version, whether you get advance notice before a change that could affect output, and whether there’s a way to test a new version against your own cases before it goes live for you. If the answer is “we handle that for you,” ask what that actually means in practice, because “we handle it” and “we tell you after it already happened” get described with the same sentence.
Ask for their real numbers on a task like yours, not their marketing benchmark
Published benchmark scores measure performance on someone else’s test set, on someone else’s task. They tell you almost nothing about how the tool performs on your specific use case, your data, your edge cases. Ask instead for a trial period against your own real examples, run by your own team, scored against your own definition of correct, before committing to a full year contract. A vendor confident in their product will offer this without much resistance. A vendor who steers you toward a subset of examples they already know work well is showing you their best case, not your likely case.
Ask what “human in the loop” actually costs once you’re using it daily
Sales conversations describe human review as a lightweight safety net. In practice, review time scales with volume, and nobody quotes that cost upfront because it’s not the vendor’s cost to bear, it’s yours. Ask for a realistic estimate of review time per output based on similar customers, not a hypothetical minimum. If the vendor can’t or won’t give you a number, budget for the pessimistic case rather than the pitch-deck case, because the actual number tends to land closer to the former.
Ask what leaving looks like before you need to leave
This is the question most businesses skip entirely, and it’s the one that costs the most later. Can you export your data, your configurations, your historical outputs, in a usable format, and how long does that process take. Is there a contractual notice period, and does pricing change if you try to reduce usage before cancelling. A vendor who’s confident in their product will have a straightforward answer, because they’re not counting on you being unable to leave. A vague or evasive answer here is a preview of what a real exit will feel like.
Ask who owns the output, plainly
Most vendor contracts address this, but not always in your favor, and it’s rarely the clause anyone reads carefully before signing. Confirm in writing that outputs generated using your inputs belong to you, not the vendor, and that there’s no clause granting them a license to reuse your specific outputs beyond what’s needed to run the service. This matters more the more your business builds on top of what the tool produces.
A one-page version you can actually use before your next demo
You don’t need a formal RFP process to apply this. Before your next vendor call, write down these six questions on one page: what happens to our data after we cancel, can we pin or preview a model version change, can we test against our own examples before committing, what’s the realistic human review time per output, what does export look like and how long does it take, and who owns the output. Bring that page to the call instead of just watching the demo. The vendors worth building on will answer plainly. The ones who can’t, or won’t, just told you what the relationship will actually be like once the contract is signed.
