Questions to Ask AI Vendors: A Checklist Before You Buy

Before you buy AI software, ask vendors about seven areas: what the product does for your use case, how its quality is measured, how your data is stored and whether it is used for training, which security controls exist, how the vendor sees its role under the EU AI Act, how pricing scales with your volume, and how you can leave. The answers worth trusting are specific, given in writing, consistent with the contract, and confirmed in a trial with your own data.

This checklist lists the questions per area, what a good answer looks like and which red flags should stop a purchase. It complements our guide on how to choose an LLM, which covers comparing models; here the focus is on buying an AI product or service from a vendor.

Before you talk to vendors

Vendor conversations go better when you know what you need. Prepare four things:

1. The use case: what task the tool should support, who will use it and what happens with the output.
2. Success criteria: what "good enough" means, for example the share of usable drafts or the time per case.
3. The data involved: whether personal, confidential or customer data will go into the tool.
4. A first risk classification: the purpose decides which obligations apply. The free EU AI Act risk checker gives a quick orientation, and anything that looks high-risk changes which questions matter most.

If you keep an AI inventory, record the planned use there before the purchase, not after.

1. Product and fit

Question	A good answer
Which tasks is the product designed for, and which not?	Clear limits, not "it can do anything"
Which AI models does it use, and who provides them?	Named models or providers, and whether you can choose
How do you notify customers when the model changes?	Advance notice and release notes, ideally with a test period
Which customers with a similar use case can we talk to?	References in a comparable industry or size

The model question matters because behaviour can change when a vendor switches models. A prompt or workflow that worked last month may need re-testing.

2. Quality and evidence

Question	A good answer
How do you measure quality for a use case like ours?	A described method and metrics, not only testimonials
Can we run a trial with our own data?	Yes, with a defined period and support
What are the known failure modes?	Honest examples, such as types of questions it handles badly
How does the product handle uncertain or missing information?	It can say "I don't know", cite sources or hand over to a person
Which features support human review?	Review queues, edit tracking, source references, logs

A trial with your own cases is the only reliable evidence. Prepare 20 to 50 realistic examples, including difficult ones, and score the results against your success criteria. Our article on how to reduce AI hallucinations explains which product features make wrong answers less likely.

3. Data protection and confidentiality

Question	A good answer
Is our data used to train or improve models, yours or a third party's?	A clear yes or no for your plan, matching the contract
How long are inputs, outputs and logs stored?	Defined retention periods and configurable deletion
Where is data processed and stored?	Named regions, and whether data leaves the EU
Which sub-processors are involved?	A current list, with notice of changes
Do you offer a data processing agreement?	Yes, as required under the GDPR when the vendor processes personal data on your behalf
Who at the vendor can access our data, and when?	Restricted, logged access, for example only for support with your permission

Check that the answers match the documents. Marketing pages, the terms of service, the data processing agreement and the actual product settings should say the same thing. Involve your data protection officer before any personal data goes in.

4. Security

Question	A good answer
Which security certifications or audit reports can you share?	Recognised frameworks such as ISO/IEC 27001 or SOC 2 reports, with scope and date
Do you support single sign-on, roles and permissions?	Yes, so access follows your user management
Which activity is logged, and can we access the logs?	Admin and usage logs available to you
How do you protect against prompt injection and data leakage between customers?	Concrete measures, testing, and separation of customer data
How and how fast are customers informed about security incidents?	A defined process and time frame in the contract

AI products add specific risks to the usual software checks, such as manipulated inputs in documents or emails that try to override the system's instructions. A vendor should be able to explain how it tests for this.

5. EU AI Act and compliance

Question	A good answer
How do you classify the intended purpose of the product under the AI Act?	A reasoned classification, not a blanket "we are compliant"
For high-risk uses: what documentation and instructions for use do you provide?	Instructions covering purpose, limits, human oversight and logs
Which features help us meet transparency duties?	For example, chatbot disclosure or labelling of generated content
What changes if we customise or rebrand the product?	Awareness that substantial changes or a new purpose can shift provider duties to you
Do you provide training material for users?	Material that supports your AI literacy measures

Under the AI Act, most buyers are deployers and the vendor is the provider. For high-risk uses, you rely on the vendor for much of what you need: the overview of deployer obligations explains what remains your responsibility. According to the European Commission, rules for high-risk systems in the listed areas such as employment apply from 2 December 2027, so ask how the vendor is preparing. This is general information, not legal advice; involve counsel for anything that could be high-risk.

6. Costs and commercial terms

Question	A good answer
What exactly is billed: seats, usage, conversations, tokens?	A pricing model you can calculate for your volume
What counts as one unit of usage, and what happens above the limit?	Clear definitions, overage prices, spending caps
How much notice do you give before price changes?	A defined notice period in the contract
What costs come on top: setup, integration, premium support?	A complete list, not only the licence
Is there a minimum term or commitment?	Terms that fit a pilot, for example a short initial period

Model the total cost for your expected volume over at least a year, including your internal time. For usage-based pricing, the LLM API cost calculator helps you estimate token-based charges, and the AI ROI calculator shows whether the expected time savings cover the costs.

7. Integration, support and exit

Question	A good answer
Which of our systems can it connect to, and how?	Documented integrations or APIs, not "anything is possible"
What support is included, and with which response times?	Defined channels and service levels
Can we export our data, configuration and prompts?	Yes, in usable formats, at any time
What happens to our data when the contract ends?	Deletion within a defined period, with confirmation
What is on the roadmap for the next twelve months?	A plausible roadmap, without making your decision depend on promised features

The exit questions are easy to skip when you are excited about a product. They determine how expensive it will be to switch if the vendor raises prices, changes direction or disappears.

Red flags

• Claims of perfect or near-perfect accuracy.
• No trial with your own data, or only a scripted demo.
• Vague or changing answers about training use and data storage.
• No data processing agreement, or sub-processors that cannot be named.
• Pricing you cannot calculate for your volume.
• No way to export your data.
• Dismissing compliance questions as irrelevant for your use case without explaining why.

How to compare vendors

Turn the answers into a simple scorecard. Weight each area by its importance for your use case, score each vendor from 1 to 5, and multiply. A hypothetical example:

Area	Weight	Vendor A	Vendor B
Quality in our trial	30%	4	3
Data protection	20%	3	5
Security	15%	4	4
Costs over 12 months	15%	3	4
Integration and support	10%	4	3
Exit and portability	10%	2	4
Weighted score		3.45	3.8

The scorecard does not make the decision, but it makes trade-offs visible: here, Vendor A did better in the trial, while Vendor B scores higher on data protection and exit. If either vendor fails a must-have, such as no data processing agreement, it is out regardless of the score.

A short question list to copy

1. What is the product designed for, and what not?
2. Which models does it use, and how are we told about changes?
3. Can we run a trial with our own data?
4. Is our data used for training, and how long is it stored?
5. Where is data processed, and which sub-processors are involved?
6. Do you offer a data processing agreement?
7. Which security audits or certifications can you share?
8. How do you protect against prompt injection?
9. How do you classify the product under the EU AI Act, and what do you provide to deployers?
10. How exactly is usage billed, and what happens above the limit?
11. What notice do we get before price changes?
12. Can we export everything, and what happens to our data at the end?

Summary

A good AI purchase starts with your use case, data and risk classification, not with a demo. Ask specific questions in seven areas, get the important answers in writing, check them against the contract and test the product with your own data. Before you sign, make sure the purchase fits your AI acceptable use policy and that the team can use the tool safely.