24 questions to ask before hiring an AI agency, and what each answer tells you
Agency calls are designed to make you feel good. That is fine, but feeling good is not information. These are the questions I would bring to a first and second call with any AI agency, including mine, grouped by the phase of the project they protect you in. For each one I have written down what the answer actually reveals.
the short version
- Ask questions that force specifics: names, examples, numbers from their past work, and what went wrong.
- Pricing questions matter less than change questions. How scope changes are priced decides your final bill.
- Delivery questions should establish when you will see working software and who you will talk to every week.
- After-launch questions are the ones most buyers skip, and the ones that decide whether the system is still useful in a year.
01how to use this list
Do not read all twenty-four questions off a sheet. Pick the four or five from each phase that matter most for your project, and listen for two things: whether the answer is specific, and whether the person answering is someone who will actually be working on your system. Vague answers from the right person are worrying. Precise answers from someone you will never see again are not much better.
Take notes in the same format for every agency you talk to. When you compare later, the differences will be obvious in a way they never are from memory.
02discovery: before they quote anything
- "What do you need from us to scope this?" A builder asks for sample data, current process documentation, and access to the person who does the work today. A sales team asks for a budget.
- "What do you think this is worth to us?" You want an agency that thinks about hours, error costs, and revenue before it thinks about models. If they cannot engage with the economics, they cannot prioritize for you.
- "What would you not build here?" The best agencies narrow the scope. If every idea is a good idea, the agency is optimizing for project size.
- "Have you integrated with our systems before?" Name your CRM, ERP, EHR, or document store. Experience matters less than whether they ask sharp questions about the API and permissions.
- "Who on this call will write code for us?" If nobody, ask to meet them before signing.
- "Can you show me something live?" Not a video. A system you can use, ideally with the person who built it explaining a decision they regret.
03pricing: what you will pay and why
- "Is this a fixed price or an estimate?" An estimate is a hope. A fixed price against a written scope means the agency has thought hard about the work.
- "How are changes priced?" This is the most important pricing question. The right answer is that every change is priced and agreed before anyone starts on it.
- "What is not included?" Hosting, model usage, app store fees, third-party licenses, data cleanup, and post-launch support are common exclusions. Get the list in writing.
- "What will this cost to run each month?" Model calls are billed by usage. Ask for an estimate at your actual volume, and ask what levers bring it down.
- "What happens if you underestimated?" On a fixed scope, the agency should absorb its own estimating mistakes. Listen for whether they say that plainly.
- "How do you get paid?" Payments tied to delivered milestones keep incentives aligned. Large upfront payments for undefined work do not.
04delivery: how the work actually runs
- "When will I see working software?" Every two weeks is a reasonable standard. If the first real demo is months away, you cannot steer.
- "Who do I talk to every week, and can I reach the engineers?" Layers of account management between you and the builders slow every decision.
- "How will you know the AI is giving correct answers?" Look for a test set built from your examples, measured on every change. See hallucination for why this matters.
- "What happens when the model is wrong in production?" Good answers include confidence thresholds, human review queues, and logging that lets you trace any output back to its inputs.
- "How will our data be handled?" Ask where it is stored, which model providers see it, what is logged, and who has access. If you are regulated, ask how they meet your specific obligations.
- "What do you need from our team, and how much of their time?" AI projects need an operator who knows the work. If the agency says it needs nothing from you, it will build the wrong thing.
05after launch: the questions most buyers skip
- "Do you stay after launch?" Some agencies hand off and leave. AI systems need someone watching quality, costs, and model updates.
- "How do you handle model version changes?" Providers update models. A responsible agency tests new versions against your evaluation set before switching.
- "Who owns the code, prompts, and accounts?" You should. Ask to see it in the contract.
- "Could another team take this over?" Ask what documentation and handover they provide. Portability is a sign of confidence.
- "How do you decide something should be retired?" Not everything earns its keep. An agency that measures usage and outcomes will tell you when to switch something off.
- "What did a past client stop using, and why?" An honest answer to this one is worth more than any case study.
06reading the answers
| Pattern | What it usually means |
|---|---|
| Specific answers with examples from their own work | They have done this before and remember the hard parts |
| "It depends" followed by what it depends on | Healthy; they are reasoning about your situation |
| "It depends" with nothing after it | They are avoiding commitment |
| Every question answered by the same senior salesperson | The builders are elsewhere and may not have seen your brief |
| They ask you more questions than you ask them | Usually a builder trying to scope accurately |
| Quick quote after one short call | They are pricing a template, not your problem |
07one more test: the small paid scope
If two agencies are close, ask each for a small paid scoping exercise on one real workflow. You get a written scope, a fixed price for the build, and a feel for how they work with your people. It costs a little and saves a lot. How to run that exercise yourself is covered in how to scope an AI project.
08how we answer these
Since I am asking you to put these questions to agencies, here is how Insomnia Club answers the important ones. I scope and ship every engagement, so the person on the first call writes code on your project. Every project is a fixed budget agreed before work starts, and change requests are priced before they begin. You get working software every two weeks. We stay after launch to measure, harden, and retire what does not earn its keep. You own the code and accounts. That applies whether the work is AI workflow automation, AI agents, or a broader custom software build.
Where we are the wrong agency: if you need a large team billed hourly under your own project managers, or if your timeline is set before the scope is. We fix the scope first and the timeline follows from it. If those constraints suit you, ask us all twenty-four.
common questions
What should I ask an AI agency on the first call?
Ask who would build your project, what they would need from you to scope it, and what working software you would have after the first month. Those three answers tell you whether you are talking to a builder or a sales team.
How do I know if an AI agency is legitimate?
Ask for live systems you can see, named people you can talk to, and a reference who can describe a time something went wrong and how the agency handled it. Legitimate agencies have all three and offer them without being pushed.
What is a fair way for an AI agency to price a project?
A fixed price against a written scope, with change requests priced before they start. Hourly billing is reasonable for open-ended support, but for a defined build it moves all of the estimating risk onto you.
What red flags should I watch for when hiring an AI agency?
Promises of accuracy without a testing method, a team that will be assigned after signature, a first demo scheduled months out, and systems that run on the agency's accounts with no way to take them with you.
Should the AI agency own the code it writes for us?
No. You should own the code, prompts, test sets, and cloud accounts from the first day. The agency can reuse its general know-how, but your system should be portable to another team without permission.