Scaling your Agentic AI solution and ensuring good governance

In my third and final blogpost looking at implementing agentic AI in #intled, I wanted to build on what we’ve already looked at. So, we identified our candidate workflow and built a pilot taking factors into account like access to other systems, memory, scope & action, levels of autonomy and the “human in the loop” escalation points.

So the next question is: how do you scale the pilot without losing control? - and just how autonomous should your agent be?

Agentic AI autonomy can be considered something of spectrum. At one end is a fully constrained agent that proposes certain actions and a human approves every action. At the other is an essentially autonomous agent that acts alone and calls for help only when an anomaly is flagged. 

Most workflows will most likely require something in between, and different tasks within the same workflow may require different settings. Drafting a reply to a routine question is not the same as changing a student record, for example. 

In terms of good governance, ensuring you have the right guardrails in place is critical.  There are a number of approaches to consider:

• Hard-coded constraints. Some rules should never depend on the model’s judgement - e.g. spending limits, data the agent cannot access, topics it must not advise on.

• RAG (Retrieval Augmented Generation). This tech solution grounds answers in your approved, current sources rather than the model’s general memory. Only being able to respond from your approved knowledge base effectively reduces hallucination to zero - although it’s always advisable to have standard AI disclaimers in place.

• Output verification. Checking at launch is not enough and a regular on-going checking process against an agreed baseline is recommended. 

• Reversibility by design. Although RAG enables highly accurate responses, it’s best to assume mistakes may happen and make them easy to fix. Keep logs and version history, use soft deletes, and make actions easy to undo.

• Escalation architecture. Define what triggers a handover, who receives it, how quickly, and with what context. Clear escalation is what lets you safely turn the autonomy dial up.
  

Lessons for scaling from MIT and AWS:

Paraphrasing Tim Kraska (1), Associate Professor of Electrical Engineering and Computer Science at MIT’s CSAIL and a former co-director of applied science at AWS, there are two points that are critical to consider:

First, multi-agent systems are useful not because they are conceptually elegant, but because they help organise complex work and let tasks run in parallel. Splitting a job between specialist agents is a practical solution rather than having one agent trying to do everything. 

Second, the harder question is not architecture but measurement. What matters is whether the system succeeds in practice, not whether it performs well against an abstract or more technical benchmark. A strong benchmark score tells you little about how your agents cope with your messy, real-world enquiries, so it’s best to try and define success in operational terms, such as accuracy, resolution, escalation rates and user satisfaction, and track it continuously.

Conclusion

So just to conclude this series of blog posts looking at implementing agentic AI in #intled, the key take outs that can help inform the process comprise:

1. Start small with a clearly defined process and required outcomes

2. Define scope around your agent’s action and access to the outside world

3. Develop appropriate guardrail architecture and escalation governance

4. Establish suitable metrics and monitor as you scale 

5. Involve the team from the get-go!

Here at AskEd, we've developed our product workflow to provide instant, engaging and accurate conversational responses to out-of-hours prospective students. The AI agent is voice-first but also offers a text-based option and is fluent in over 80 languages.

Our product workflow also provides a lead and config dashboard integrating with CRM, website, telephony and whatsapp depending on client requirements. We have taken decisions around reasoning and action built around a Retrieval Augmented Generation (RAG) knowledge base to ensure accuracy guardrails and deployed a human escalation point where relevant data is not available. We have also put control of data retention / deletion in the hands of clients.

If you would like to learn more, please email me at: john@askedai.co.uk

(1) MIT Sloan Implementing Agentic AI: Building Your Organizational Playbook program 2026-06-08