How to Choose an LLM for Your Business

To choose an LLM for your business, start from the task, not the model. Define what the model must do and what a good result looks like, set your constraints (data protection, budget, speed, integration), shortlist a few candidates, test them all on the same set of real cases, and compare quality, cost per task and response time in a scorecard. The best choice is the cheapest, fastest model that reliably meets your quality bar, under terms you can accept. Because models and prices change frequently, keep your test set and re-run it when the market shifts.

This guide gives you a repeatable process. It deliberately does not rank specific models: any such list would be outdated quickly. Check current model lineups, capabilities and prices directly with providers.

Step 1: Define the task precisely

"We need AI" is not a requirement. Write down:

• The task: for example, "draft replies to customer emails about orders and returns" or "extract fields from supplier invoices".
• Inputs: what the model receives (short messages, long documents, images, tables).
• Outputs: free text, structured data, code, decisions.
• Languages: which languages inputs and outputs use.
• Volume: requests per day or month, and peaks.
• Who sees the result: internal staff, customers, the public.
• Risk: what happens if the output is wrong.

Different tasks favour different models. A classification step with high volume needs a fast, inexpensive model. Analysis of long contracts needs a model with a large context window and strong reasoning. Many applications use several models for different steps.

Step 2: Set your criteria

Criterion	Questions to ask
Quality	Does it meet the bar on our test cases? How does it handle hard and unusual cases?
Instruction following	Does it follow format rules and constraints reliably?
Grounding	Does it stick to provided sources and admit when information is missing?
Context window	Can it handle our longest inputs with room for instructions and output?
Languages	How good is it in each language we need?
Modalities	Do we need images, documents, audio?
Structured output	Does it support schemas or reliable JSON?
Tool use	Can it call functions or tools if we need that?
Speed	Time to first token and total response time at our input sizes
Cost	Cost per task at our token volumes, including caching and batch options
Data terms	Retention, use of inputs for training, data location, certifications, contractual terms
Availability	Rate limits, uptime commitments, regional availability
Ecosystem	SDKs, documentation, integration with our existing platforms
Longevity	How often models are deprecated, and how much notice is given

Weight the criteria for your task. For an internal summarisation tool, quality and cost may dominate. For a customer-facing assistant handling personal data, data terms and grounding may be decisive.

Step 3: Decide on the deployment model

Option	What it means	Pros	Cons
Proprietary model via provider API	Model hosted by its developer	Fast start, strong capabilities, no infrastructure	Dependency on one vendor, data leaves your environment under their terms
Proprietary model via cloud platform	Same or similar models offered through a major cloud provider	Fits existing cloud contracts, regional hosting options	Model availability and features may differ
Open-weight model, hosted by a third party	Openly available model run by a hosting provider	More choice and portability	Varying quality of hosting and terms
Open-weight model, self-hosted	You run the model on your own infrastructure	Maximum control over data and customisation	Requires hardware, expertise, maintenance and security work

For most small and mid-sized firms, starting with an API is the pragmatic choice. Revisit self-hosting if data requirements, volume or customisation needs justify the effort.

Step 4: Build a test set

A test set is the most valuable asset in model selection. It lets you compare candidates fairly and re-evaluate quickly later.

• Collect 30 to 100 real cases, anonymised where needed.
• Include the distribution you expect: mostly typical cases, plus difficult ones, edge cases and cases that should be refused or escalated.
• Write down the expected result or the criteria a good answer must meet.
• Define scoring: for example pass/fail per criterion, or a 1–5 scale with clear descriptions.
• Keep it private and stable, so results are comparable over time.

For tasks with a single correct answer, such as extraction or classification, scoring can be automated. For writing tasks, use a rubric and have people score a sample, ideally without knowing which model produced which answer.

Step 5: Shortlist and test

Pick three to five candidates across price levels: at least one small, inexpensive model, one mid-range and one top-tier. Then:

1. Use the same prompt structure for all, adapted only where a provider's documentation recommends a specific format. A well-designed system prompt makes comparisons fairer.
2. Run the full test set on each.
3. Record quality scores, token usage and response times.
4. Read the failures. Two models with the same average can fail in very different ways, and one kind of failure may be unacceptable for you.

Do not choose based on public benchmarks alone. They measure general abilities on standard tasks, not your task with your data and your prompt.

Step 6: Calculate cost per task

Price per million tokens is not the same as cost per task. A cheaper model that needs longer prompts, more retries or more human correction can cost more overall.

For each candidate, calculate:

• average input and output tokens per task from your test runs,
• cost per task at current prices,
• monthly cost at your expected volume,
• the human review effort implied by its error rate.

The LLM API cost calculator compares three price scenarios side by side, and LLM API pricing explained covers the factors that change the calculation, such as caching and batch discounts. To compare the full business case including review time, use the AI ROI calculator.

Step 7: Check data and legal terms

Before deciding, review for each provider:

• whether inputs and outputs are retained, for how long and for what purpose,
• whether your data may be used to train models, and how to opt out if relevant,
• where data is processed and stored,
• data processing agreements and security certifications,
• terms on ownership and use of outputs,
• acceptable use restrictions that may affect your use case.

If you operate in the EU and the use case could fall into a regulated category under the AI Act, consider that early. The EU AI Act risk categories article explains the tiers. Involve your data protection officer or legal adviser for anything involving personal data.

Step 8: Decide with a scorecard

A simple weighted scorecard keeps the decision transparent:

Criterion	Weight	Model A	Model B	Model C
Quality on test set	35%	4	5	3
Cost per task	25%	4	2	5
Speed	10%	4	3	5
Data terms	20%	4	4	3
Integration and ecosystem	10%	3	4	4
Weighted score		3.9	3.75	3.8

The figures are illustrative. In this example the scores are close, which is common. When that happens, look at the failure analysis and at the criteria that are deal-breakers for you, and consider routing: use the cheaper model for most cases and the stronger one for difficult cases.

Avoiding lock-in

• Abstract the model call in your code so you can switch providers with limited effort.
• Keep prompts and test sets portable, not tied to one platform's proprietary features unless the benefit is clear.
• Log inputs, outputs and token counts in your own systems, within your data rules.
• Watch deprecation notices. Providers retire model versions; plan migrations and re-test.

Common mistakes

• Choosing the most famous model by default. It may be more capable and more expensive than your task needs.
• Testing with a handful of hand-picked examples. Results will not reflect real use.
• Comparing price per token instead of cost per task.
• Ignoring data terms until after the build.
• Changing prompts and models at the same time, so you cannot tell which change mattered.
• Never re-evaluating. The market moves quickly; your test set lets you benefit from it.

Summary

Define the task, set weighted criteria, build a realistic test set, test a small range of candidates, compare cost per task and data terms, and decide with a scorecard. Keep the test set and re-run it when new models or prices appear. If cost is a major factor, continue with how to reduce LLM API costs, and if you are deciding how to give the model your company knowledge, read RAG vs. fine-tuning.