The IT manager was frustrated. They’d signed a 12-month contract with an AI vendor promising “seamless integration” and “enterprise-grade security.” Three months in, they discovered:
• Integration required $40,000 in custom development — not included in the quote
• Data was stored on US servers with no Australian data residency option
• The “AI” was mostly rules-based automation with a thin ML layer
The vendor hadn’t lied. They’d just let the buyer assume.
According to Forrester’s 2025 AI Vendor Evaluation Report, 67% of organisations report significant gaps between vendor promises and delivered capabilities. The problem isn’t deception — it’s the questions that never get asked.
Here are the questions that protect you.
On Data & Privacy
1. Where is my data stored and processed?
If the answer is “our cloud” — push harder. Which region? Which provider? Is Australian data residency available? For many regulated industries, this is non-negotiable.
2. Is my data used to train your models?
Some vendors use customer data to improve their AI. If confidentiality matters, you need a contractual carve-out — not just a verbal assurance.
3. What happens to my data if I leave?
Can you export everything? In what format? How long do they retain it after contract ends? Get this in writing.
On Cost
4. What’s included in the base price — and what isn’t?
Integration, training, support, API calls, overages. Ask for a full cost breakdown over 24 months, not just Year 1 license fees.
5. How does pricing scale if usage grows?
Some models charge per user. Others per API call. Others per document processed. Know the meter before you commit.
6. What are the exit costs?
Early termination fees? Data migration costs? Re-implementation burden? Vendor lock-in is expensive.
On Capability
7. What exactly is “AI” in your product?
Is it machine learning? Large language models? Rules-based automation branded as AI? There’s nothing wrong with simpler approaches — but know what you’re buying.
8. Can I see a customer reference in my industry?
Case studies are marketing. References are reality. Ask to speak with someone who’s been live for 6+ months.
9. What does implementation actually look like?
Timeline, resources required from your side, dependencies, common failure points. If the vendor can’t answer this clearly, they haven’t done it enough times.
On Support
10. What’s included in support — and what’s extra?
Response times, escalation paths, dedicated account management, training refreshers. Get the SLA in the contract, not just the pitch deck.
11. Who will I actually be working with?
Sales teams are charming. Implementation teams are who you’ll live with. Ask to meet them before you sign.
12. What happens when something goes wrong?
AI outputs can be unpredictable. What’s the process for reporting errors, getting fixes, and preventing recurrence?
The Meta-Question
After all the specifics, ask one more:
“What’s the most common reason your customers fail to get value from this product?”
A good vendor will answer honestly — because they’ve learned from it. A bad vendor will deflect.
The goal isn’t to catch vendors out. It’s to make a decision with eyes open — not assumptions intact.
→ Learn more about AI tool selection criteria and consideration
Sources: Forrester AI Vendor Evaluation Report 2025.
