5 AI Terms Every Technical Buyer Should Know Before Evaluating a Model Vendor

AI vendor demos all look good. The demo is built to. The difference between a good purchase and an expensive lesson usually shows up in the follow-up questions, and you can’t ask them if the vocabulary is fuzzy.

These five terms come up in almost every evaluation. Know what each one means, and what to ask about it.

1. Fine-tuning

Start with what fine-tuning is: additional training that adapts an existing pretrained model to a specific job. The model already understands language. Fine-tuning shapes how it behaves on your task, like routing tickets to the right queue or returning a strict format every time.

It’s different from training from scratch, which starts with random parameters and needs enormous data and compute. Fine-tuning builds on what the model already knows, so it needs far less of both. The fine-tuning platform Oumi, for example, trained a 4B-parameter model on a public banking dataset in about 30 minutes.

Ask the vendor: What exactly gets trained, on whose data, and how do you show me it improved?

2. Retrieval-augmented generation (RAG)

RAG fetches relevant documents or records at request time and hands them to the model as context. The technique got its name from a 2020 paper by researchers at Facebook AI Research, University College London, and New York University.

RAG fits information that changes: policies, product docs, account records. Fine-tuning fits behavior that repeats. Many vendors blur the two. A model fine-tuned on last quarter’s policy will apply last quarter’s policy, consistently and wrongly.

Ask: When our data changes, what updates, and how fast?

3. Evaluation set (and holdout)

An evaluation set is the list of test cases used to score a model. A holdout is the part of your data the model never trains on, kept back so the score means something.

This is where weak vendors slip. A finished training run proves the job is complete. It doesn’t prove the model got better. The proof is a before-and-after comparison on the same untouched test cases.

Ask: Can we supply our own holdout set, and will you report results per category instead of one average?

4. LoRA and adapters

LoRA trains a small add-on (an adapter) instead of updating every parameter in the model. It’s cheaper, faster to test, and easy to roll back. QLoRA goes further by compressing the base model during training, so larger models fit on cheaper hardware. Full fine-tuning updates everything and costs the most. For a fuller comparison of LoRA vs. QLoRA and full fine-tuning, the tradeoffs come down to hardware, quality, and how you’ll serve the model.

Ask: Which method do you use, and why that one for our task?

5. Open-weight model

An open-weight model publishes its trained parameters, so you can download it, fine-tune it, and run it on infrastructure you control. Closed models are only reachable through the vendor’s API.

The difference matters most at renewal time. With open weights, the trained model can be yours. With a closed API, you’re renting.

Ask: If we leave, what do we take with us?

Bring this list to the next demo

Print these five questions. Ask every vendor all five. Score the answers on specifics: named methods, real test results on your data, and a clear exit path. Vague answers to any of them are your shortlist’s first cut.