Choosing an AI agency: what to look for

Why this choice has become hard

Two years ago it was manageable. A handful of parties did something with language models and you picked whoever could start soonest. Today nearly every digital agency, every ERP implementation partner and every staffing firm calls itself an AI partner.

That makes choosing hard, because on the website it all looks the same. Everyone talks about agents, automation and ROI. And almost everyone can show you an impressive demo within two weeks, because building a demo is genuinely not difficult any more.

The difference is not in what someone can show, but in what is still running three months later. That is what you should select on, and it calls for different questions than most procurement processes ask.

The demo is not the product

A demo shows that a model can read a document. We already knew that. The real question is what happens with the document that is slightly different: the supplier who changed their layout, the order with an item that is not in your master data, the invoice with a missing amount.

Those cases do not appear in a demo, because the demo runs on documents that go well. In production they are half of your work. So do not ask for a demo of the successful case, ask for a story about the failed one.

• What does the system do when it is not sure? If the answer is guess, end the conversation.
• Where does a case the agent cannot handle end up, and who sees it?
• How do you find out afterwards why something was posted the way it was?
• What happens when a supplier changes their document layout? Does someone have to come in?

Ask about production, not about projects

Many agencies work in projects: there is an assignment, something gets built, it gets delivered, and after that you own something you have to understand and maintain yourself. For a website that works fine. For an agent posting into your administration daily, it is a problem.

AI solutions degrade quietly. Models get updated, suppliers change their documents, your own ERP gets a release, and the share that flows through automatically slides from 92 to 78 without anyone noticing. Who watches that, and who pays for it?

So ask explicitly: how many processes do you have running in production, for how long, and who does the maintenance. A party that can only name pilots has never had to answer that question.

Do they build in your system or beside it?

This is the question that best predicts whether you will still be happy a year from now. There are two kinds of solutions: those that work inside the systems you already have, and those that put a shell alongside them that people also have to work in.

The second is tempting for the vendor, because an own interface is easier to build and easier to sell. But it means your team maintains two places and data has to be synchronised somewhere. That is exactly where adoption breaks down.

Our starting point is that the agent works in your existing environment: SAP, AFAS, Exact, Dynamics 365, Odoo or Bouwworks. The order simply sits in your ERP. Your colleague does not open a new tool, they see less work in the tool they already use.

What about security and your data?

The moment an external party gets access to your administration, this stops being a tick-box question. There are a few things you want in writing, and that you should get answered without hesitation.

• Where is your data processed, and does it stay within the EU?
• Is anything trained on your data, and if not, how is that arranged contractually?
• Is there an independent certification, ISO 27001 for instance, or only a promise?
• Which permissions does the agent get in your systems, and how are they bounded?
• What happens to your data if you stop?

A partner who welcomes these questions is a partner who has had them before. A partner who turns vague will cost you time later in conversations with your own security team.

Price says more than you think

Watch the pricing model, because it reveals how someone sees the work. Time and materials means the risk sits with you: if it overruns, you pay. A fixed price per process means the vendor takes risk on their own estimate.

That second model only works if someone has done it before and knows what it costs. It is therefore indirect evidence of experience. Also ask what is included after go-live, because maintenance and model renewal are not extras but the bulk of the lifespan.

And do the maths yourself. If a process costs 20 hours a week at an internal rate, within ten minutes you know what automating it is worth. That number is your negotiating baseline, and it keeps the conversation honest.

The questions that separate fastest

• Which process have you automated most often, and how often did it go wrong the first time?
• May I speak with a customer who has had this in production for over a year?
• What is your honest estimate of the share that will stay manual?
• Who do I call when nothing comes through on a Tuesday morning?
• What happens if the model you use is retired next year?

The last one is a good litmus test. A party that used the model as a foundation gets uncomfortable here. A party that treats the model as a replaceable component has a short and boring answer. Boring is the right answer here.

Getting started

Do not pick on the most impressive demo but on the dullest evidence: running processes, measurable numbers, a clear maintenance story and a partner willing to say what cannot be done. That last one is rarer than it should be.

If you want to see how we do it, plan a Quick Scan. We look at one concrete process, show what an AI coworker can take over and what it cannot, and give an honest estimate of the gain. Typically you are live within eight weeks. Plan your go-live whenever it suits you.