AI325: Agent Harness Engineering – Designing, Building, and Evaluating Agents NEW

Agenda:

• Loop, Tools, and Context

  • Anatomy of an agent: understanding the model, loop, tools, and context as a harness

  • What coding agents like Claude Code do under the hood: the harness as the key difference

  • Basic rule of harness design: the simplest harness that works wins

  • The agent loop in detail: reason/act cycle, tool results as feedback, and termination criteria

  • Identifying and fixing typical error patterns: infinite loops, context loss, premature termination

  • Verify loop as a basic pattern: modify, test, repair

  • Tool design: tailoring tools specifically rather than providing an unrestricted shell

  • Schemas, descriptions, and error returns as control instruments; enforcing structured output
    
    

Hands-on-Lab: Building an agent loop without an SDK (looping via the Messages API, tool definitions, dispatch, feeding back results, termination criterion); subsequently implementing the verify loop and deliberately triggering error scenarios

• Scaling, securing, and measuring

  • Context engineering: managing the context budget as a scarce resource

  • Summarization and compaction, memory files, and repository maps

  • Prompt caching: impact on cost and latency

  • Subagents and orchestration: context isolation via a second instance of your own loop

  • Handoff or coordinator: choosing the right pattern and determining when splitting tasks is worthwhile

  • SDKs and frameworks classified: what they handle for you compared to a custom-built loop, and what they cost

  • Multi-agent frameworks using CrewAI as an example: agents, roles, tasks, and crews; comparison with your own harness

  • Security and control: sandboxing, permission models, and hooks as hard guardrails

  • Prompt injection and excessive agency in practice: understand the attack, then secure against it

  • Evaluation: building your own eval harness with test cases, success criteria, and regression detection

  • Realistically assessing public benchmarks and their limitations
    
    

Hands-on-Lab: Extending your own harness with a subagent, guardrails, and an eval suite, implementing the same task using CrewAI, and measuring both variants against the test cases

Objectives:

Upon completion of the workshop AI325 Agent Harness Engineering – design, build, evaluate agents the participants will be able to:

• explain and evaluate the architecture of modern LLM agents, including loops, tools, context, and guardrails

• implement an agent harness directly against the API without using a framework

• design tools with clean schemas, descriptions, and error returns

• economically control the context budget with compaction, memory, and prompt caching

• structure agents using subagents

• make informed comparisons between SDKs/multi-agent frameworks such as CrewAI and custom implementations

• secure agents against prompt injection and excessive agency

• measure agent quality using a custom eval harness and detect regressions

Target audience:

The training AI325 Agent Harness Engineering – Designing, Building, and Evaluating Agents is aimed at:

•      Software developers who want to integrate LLM agents into their own products and workflows

•      AI engineers and ML engineers who are transitioning from prompting and individual API calls to agentic systems

•      Tech leads and architects who want to evaluate self-build and agent frameworks against each other

Prerequisites:

For participation in the course AI325 Agent Harness Engineering – Designing, Building, and Evaluating Agents the following prior knowledge is required:

•      solid programming skills in Python

•      first practical experience with an LLM API or a coding agent

•      basic understanding of HTTP APIs and JSON

•      prior knowledge of agent frameworks is not necessary

Description:

An agent is more than just a model: The harness—that is, the loop, the tools, context management, and the guardrails—determines whether an LLM agent operates reliably or fails uncontrollably. The course AI325 Agent Harness Engineering – Designing, Building, and Evaluating Agents focuses on this harness: Participants build it themselves from scratch, without a framework, directly against the API.

In the first part, participants work out the agent loop in detail, design clean tools with schemas and error responses, and implement their own harness in the lab including a verify loop.This process reveals that the core of an agent consists of just a few dozen lines of code, and that the actual engineering work lies in the design of tools, context, and termination criteria.

The second part prepares the agent for production operation: context budget and prompt caching, scaling with subagents, protection against prompt injection and excessive agency, as well as an own eval harness with test cases and regression detection. Finally, participants implement the same task using the CrewAI multi-agent framework and evaluate both variants against the same test cases. Anyone who has built the loop themselves will recognize what a framework actually handles and what it does not.