Lesson 0: Build a Research and Writing Agent with MCP

LLMs are solving Math Olympiad problems and folding proteins into structures that match experimental data. But ask one to write a LinkedIn post about AI engineering, and here is what comes back:

Lets delve into the intricate tapestry of the AI landscape! In today's rapidly evolving ecosystem, leveraging cutting-edge frameworks is paramount to unlocking unprecedented synergies. This groundbreaking, revolutionary approach fosters a vibrant paradigm shift that empowers stakeholders to navigate the multifaceted realm of agentic AI. The robust, scalable architecture stands as a beacon of innovation, poised to disrupt and enrich the very fabric of our digital journey. Buckle up!

Okay yes, this is a little exaggerated. But take out the parody, and the average LLM-generated post still has zero original thought, no sources behind any claim, and a voice that sounds like every other AI post you have ever scrolled past.

So, we built a system that makes LLMs take content generation as seriously as protein folding. And now, we are giving you this as a free lesson in this course. It takes any topic, autonomously researches it using grounded web search and YouTube video analysis, then runs the result through a writing workflow that drafts, reviews, and edits in a loop, catching AI slop, enforcing structure, and maintaining a consistent voice.

You build the complete system across 34 lessons in the full course, covering foundations (prompt chaining, structured outputs, tool calling, memory, RAG), the research agent, and the writing workflow, component by component, using LangGraph and FastMCP, followed by evaluation, observability, and deployment.

This lesson walks you through a pocket version of the full system. Clone the repo, add one API key, and you have it running in under five minutes. It is a fraction of what the full course builds, but enough to see what even a basic layer of infrastructure does to LLM output. Here is an example of what the same model produces with research grounding and a structured writing workflow behind it:

We planned 12 AI agents and shipped 1. It worked better. Sounds crazy, right? But it's a common story. A client built an AI marketing chatbot. Their initial design had dozens of agents: orchestrator, validators, spam prevention. It failed. A single agent with tools won. Tasks were tightly coupled. One brain maintained context. Tools were still specialized.

The first draft (v0) on the left is verbose with a weak hook. After three rounds of automated review and editing, the final draft (v3) on the right is tighter, punchier, and structurally sound. Each intermediate version (post_0.md, post_1.md, etc.) is saved so you can see exactly what changed per pass

We originally built this system as a live workshop for the AI Engineering conference, which you can watch here:

GitHub repo: github.com/iusztinpaul/designing-real-world-ai-agents-workshop

Why Build This?

Off-the-shelf LLMs generate text by predicting the most likely next token. That likelihood is shaped by what the model has seen across its training corpus and what it was rewarded for during alignment. The outputs gravitate toward the most common patterns in the data: safe, general, and stylistically average.

Getting the model to break those patterns and produce something with an original voice, grounded claims, and structural precision is partly solved by existing techniques, but cannot be consistently enforced across runs.

Prompting and context engineering get you the furthest with the least effort. A well-constructed system prompt with structural rules, audience definition, tone constraints, and few-shot examples can shift the output distribution away from the generic baseline. Context engineering extends this by controlling what enters the context window and how it is formatted.

Thinking mode adds a reasoning step before generation, improving instruction-following and coherence on complex tasks. Tool use and function calling go further by giving the model access to external systems at inference time: web search for up-to-date information, APIs to access data from services, and code execution for computation. With search access, the model can ground its claims in real sources rather than generating plausible approximations from training data.

Together, these techniques produce strong first drafts. But they all operate within the same constraint: generation and evaluation happen in a single forward pass. The model writes and self-assesses in a single pass, so errors that survive generation have no second mechanism to catch them. There is no persistence across runs, no external quality criteria to evaluate against, and no loop in which output is revised based on structured feedback.

The resulting failure mode is specific and measurable. Run the same prompt ten times and compare outputs: structure will be inconsistent, banned terminology will appear in some runs and not others, factual claims will vary in specificity and accuracy, and voice will drift across generations.

The model is not failing to follow instructions. It is following them probabilistically, which means compliance is stochastic rather than enforced. In practice, human reviewers end up applying the same corrections on every output: grounding vague claims, removing terminology the prompt explicitly banned, and restructuring sections that drifted from the guidelines. The corrections are consistent enough to automate, but there is no layer in a single-pass pipeline where those rules can be applied and verified after generation.

This system adds that layer. A research pipeline runs before generation, compiling source-backed material into a structured document that enters the context window as grounded input rather than a vague topic direction. Writing profiles define structure, voice, terminology, and formatting as machine-readable constraints. These are explicit rules, dynamically compiled into the system prompt, that the evaluator checks against.

The evaluator-optimizer loop is the core architectural difference: one LLM generates a draft, a second reviews it against the user input, profiles, and research, and returns a structured list of violations. The generator edits the draft to resolve them. This runs for multiple passes, each intermediate version saved to disk, so you can diff any two and see exactly which violations were caught and how they were addressed. The loop converts quality criteria from suggestions the model may or may not follow into enforceable constraints with a feedback mechanism.

📓

This lesson presents a working prototype that uses Claude Code as the orchestrator, rather than a full agent framework with state management. It lacks persistent memory, a deployment strategy, and has minimal cost controls. There's minimal context engineering and observability. These limitations are addressed in the full course, which covers each aspect in separate lessons, ultimately building a reliable system with the same architecture and infrastructure.

What You’re Building

The system runs on two MCP servers: a Deep Research MCP Server and a LinkedIn Writer MCP Server , both built with FastMCP , a Python library that implements the MCP protocol and exposes Python functions as tools any MCP-compatible agent can call. Claude Code is the orchestrating agent. It reasons about what to research, decides when it has enough material, and delegates the actual work to whichever server’s tools are appropriate. The two servers connect through two shared files: research.md (the compiled research output) and guideline.md (the topic, angle, audience, and key points for the post).

The system architecture of the whole multi-agent system.

Deep Research MCP Server. You give Claude Code a topic like “What are AI agent skills?” and it breaks that into targeted queries. Each query goes to Gemini with Google Search grounding, which returns answers backed by real sources. If you include YouTube URLs, Claude Code sends them to Gemini for native video analysis. The extracted insights serve as context seeds: the agent uses them to better understand the problem space and to construct more focused follow-up queries, rather than relying on open-ended guesses. All results accumulate in a .memory/ folder on disk and get compiled into research.md at the end.

The guideline.md ties both servers together. It defines the topic, angle, audience, and key points, and can seed the research step with a focused starting point instead of a broad topic. The research agent uses it to narrow its queries. The writing workflow uses it alongside research.md to generate a post that matches the intended direction.

Claude Code connects to the FastMCP server, which dispatches to three tools. The tools call Gemini for web search and video understanding, and read/write results to the .memory/ folder.

LinkedIn Writer MCP Server. The writing system takes that research.md plus the guideline.md and produces a polished LinkedIn post. It generates an initial draft, then runs an evaluator-optimizer loop. A reviewer LLM checks the draft against the user guideline, research, and writing profiles. Then, a separate editor LLM applies corrections. This loop runs multiple times. The writing profiles ban 40+ AI-slop words (“delve”, “tapestry”, “leverage”, “vibrant”, etc.), enforce structural rules (800-1500 characters, a hook in the first two lines, and a CTA at the end), and maintain a consistent voice. The final output is post.md.

The research agent produces research.md. The writing workflow reads it, generates a post, reviews it, edits it, and repeats until the quality is high enough.

The system's research and writing components have distinct execution models, a key architectural decision discussed in depth in Lesson 2.

In an agent, the LLM controls the execution flow: it decides what to do next based on what it has observed so far, so the sequence of steps differs between runs. Research requires this because the search path depends on intermediate results. A query about AI agent architectures might surface a paper on tool-calling patterns, leading the agent to search for failure modes in tool-calling and surface a case study it decides to pursue. The number of queries, the follow-up directions, and the stopping point are all determined at runtime. The research system was originally built as a fixed-step workflow, but in practice, its rigidity produced shallow results because it could not adapt when early queries revealed that the interesting questions differed from those it had started with.

A workflow has a fixed execution path that runs the same way regardless of input. The writing system generates a draft, reviews it, edits it based on the review, and repeats the process. The input varies, but the process does not. No runtime decisions about what step comes next, which means lower cost per run, a predictable sequence of LLM calls, and easier debugging when something goes wrong.

The system also includes an observability pipeline built with Opik that monitors all traces generated by the MCP servers and evaluates the quality of the generated posts using a binary LLM judge.

Model Context Protocol (MCP)

MCP is an open protocol for serving business logic to agents. An MCP server exposes functionality as tools, data as resources, and workflow instructions as prompts, and supports authentication, user sessions, and custom UIs rendered directly within agent harnesses.

Most agent frameworks can already turn a Python function into a tool with minimal setup. What they do not provide is a standalone server that packages business logic, data, and instructions together and makes them accessible to any MCP-compatible client like Claude Code, Cursor, VS Code, a LangGraph agent via the LangChain MCP Adapters library, or anything else that speaks the protocol. You write the server once, and any client can discover and call its tools, read its resources, and load its prompts without modification.

The research and writing servers in this project are both FastMCP servers, each exposing a set of capabilities that Claude Code connects to in this lesson, but that any MCP client could use. The protocol defines three primitives for what a server can expose:

Tools are actions with side effects: calling APIs, writing files, and changing state. The research server exposes deep_research, analyze_youtube_video, and compile_research. The writing server exposes generate_post, edit_post, and generate_image. When an agent calls a tool, something in the outside world changes.

Prompts are reusable instruction templates that describe how to use the tools. The research server’s research_workflow prompt lays out the full research pipeline, and the writing server’s linkedin_post_workflow does the same for the generate-review-edit cycle.

Skills are not part of the MCP protocol yet, but they are headed there. They started as a Claude Code feature but have since become an open standard (agentskills.io) supported across Claude Code, Cursor, Gemini CLI, GitHub Copilot, and other agent harnesses. In this project, each skill is a markdown file at .claude/skills/<skill_name>/SKILL.md that tells the agent when to trigger a workflow and which MCP prompt to load. The research skill triggers on phrases like “research this topic” and points the agent to the research_workflow prompt. The writing skill does the same for linkedin_post_workflow. The workflow logic itself stays on the server inside the MCP prompt, so the skill only needs to know when to invoke it and where to find it. Once skills are supported directly within MCP, we can merge the logic under a single dedicated skill under each server.

Resources are read-only data endpoints with no side effects. They provide the agent with context: the research server exposes its configuration (server name, version, model names), and the writing server exposes its config and all writing profiles.

Project Structure

The project structure follows a pattern commonly used in FastAPI servers (routers, models, config, utils), adapted here for FastMCP. If you have built a FastAPI app before, this layout will feel familiar. The two MCP servers are under src/, each with the same modular layout:

designing-real-world-ai-agents-workshop/
├── .env.example            # Template for API keys
├── .mcp.json               # Registers both MCP servers
├── pyproject.toml           # Dependencies
├── datasets/               # Few-shot examples for the writer
│   └── linkedin_paul_iusztin/
│       └── index.yaml      # Post examples with guidelines and images
├── src/
│   ├── research/            # Research agent MCP server
│   │   ├── server.py
│   │   ├── routers/         # tools.py, prompts.py, resources.py
│   │   ├── tools/           # deep_research, youtube, compile
│   │   ├── app/             # Business logic handlers
│   │   ├── config/          # Settings, constants
│   │   ├── models/          # Pydantic schemas
│   │   └── utils/           # File I/O, LLM client, logging
│   └── writing/             # Writing workflow MCP server
│       ├── server.py
│       ├── routers/         # tools.py, prompts.py, resources.py
│       ├── tools/           # generate_post, edit_post, generate_image
│       ├── app/             # Post writer, reviewer, editor, image handler
│       ├── config/          # Settings, prompts
│       ├── models/          # Pydantic schemas (Post, Review, Profiles)
│       ├── profiles/        # Writing profiles (structure, terminology, character, branding)
│       └── utils/           # LLM client, file I/O
└── .claude/
    └── skills/
        ├── research/SKILL.md    # /research skill
        └── write-post/SKILL.md  # /write-post skill
        └── research-and-write/SKILL.md # /research-and-write skill (combines both)

Both servers follow the same pattern: routers/ registers the MCP endpoints (tools, prompts, resources), tools/ defines what each tool does, and app/ contains the core business logic that tools call into. Configuration is in config/, including model names and prompt templates. Data schemas are in models/ as Pydantic classes. Since the model names are just settings in config/settings.py, you can swap Gemini Flash for Gemini Pro (or any other model) by changing an environment variable, without touching any MCP or tool code.

Setup

The architecture is easier to follow once you can see the components running. Setup takes about five minutes. Remember that this is the GitHub repository we will use for this lesson.

Step 1: Install Claude Code

Claude Code (an AI agent made by Anthropic) runs in your terminal or as a desktop app and can use tools, read files, and execute multi-step tasks. In this project, it acts as the harness that orchestrates both MCP servers.

A harness is the runtime layer between the model and external systems. An agent is a model plus a harness. The harness is every piece of code, configuration, and execution logic surrounding the model: memory, tools, orchestration, guardrails, and context management. It manages which tools are available, assembles the context for each call, dispatches tool invocations, and controls the execution flow. Think of it the way you would a horse harness: the animal has the strength, and the harness is what makes it usable for specific work.

A basic agent combines a model, a prompt, tools, and a planning loop. A harness extends this with memory systems, guardrails, advanced orchestration, context engineering, multi-agent coordination, and a serving layer that connects the agent to terminals, web dashboards, IDE plugins, and messaging platforms. Claude Code, OpenCode, and Codex are all harnesses. You could swap the model inside any of them, but the engineering value is in the harness, not the weights.

In this project, the harness is simpler: Claude Code manages which MCP tools are available, assembles the context for each call, dispatches tool invocations, and controls the execution flow.

The three levels of engineering: Prompt engineering is crafting instructions, context engineering is managing what the model sees, and harness engineering is the full infrastructure.

Install it by following the official instructions at docs.anthropic.com/en/docs/claude-code. You will need an Anthropic API key or a Claude Pro/Max subscription.

curl -LsSf https://astral.sh/uv/install.sh | sh

Step 3: Clone and Configure

git clone https://github.com/iusztinpaul/designing-real-world-ai-agents-workshop.git
cd designing-real-world-ai-agents-workshop
cp .env.example .env

Open .env and add your Google API key. You can get one for free at aistudio.google.com.

# Required
GOOGLE_API_KEY=your-key-here

# Optional: enables Opik tracing
OPIK_API_KEY=your-opik-key

💡 Without GOOGLE_API_KEY, nothing works. The Opik keys are optional and only needed if you want to explore the observability dashboard, which we cover later in the course.

Step 4: Start Claude Code

cd designing-real-world-ai-agents-workshop
claude

When Claude Code starts, it reads a file called .mcp.json in the project root.

/mcp      # Shows connected servers

You should see both deep-research and linkedin-writer MCP servers listed. Selecting each should reveal six tools total: three for research (deep_research, analyze_youtube_video, compile_research) and three for writing (generate_post, edit_post, generate_image). If a server is not connected, check that your .env file has a valid GOOGLE_API_KEY.

Step 6: Run Your First Research

Now give the agent a topic to research:

Research what AI agent skills are, how autonomous agents use
skills and tools to complete tasks, and the latest architecture
and design patterns for 2025. Also analyze this YouTube video:
https://www.youtube.com/watch?v=CEvIs9y1uog

Claude Code reads your prompt, breaks the topic into multiple targeted queries, and calls deep_research for each one. It reads the results, identifies coverage gaps, and generates additional queries to fill them. It also calls analyze_youtube_video to extract insights from the video. These insights act as context seeds, helping the agent understand the problem space and generate more targeted follow-up queries rather than guessing blindly. When it has enough material, it calls compile_research to assemble everything into research.md. The agent decides what to search, how many queries to run, and when it has enough to stop.

To keep runs predictable, the server caps total exploration calls at 6 (MAX_EXPLORATION_CALLS in src/research/config/constants.py). Every deep_research and analyze_youtube_video call counts against this shared budget, recorded in .memory/. Each successful call returns the number of calls remaining, giving the agent a visible countdown so it can pace itself before the hard stop. The agent is free to stop earlier when it judges coverage is sufficient, but once the budget is exhausted, the next exploration call returns an error whose message instructs the agent to run compile_research immediately and finalize research.md from the results gathered so far. compile_research then resets the budget, so any follow-up research after a finalized brief starts fresh. This is the guardrail that keeps cost and runtime bounded even when the agent is otherwise free to plan its own search path.

Step 7: Write a Guideline

Before generating a post, you need a guideline.md. In this case, the brief includes what the post should cover, the angle, the target audience, and the key points. Here is an example:

# LinkedIn Post Guideline

## Topic
Why most AI teams should use 1 agent instead of 12.

## Angle
Open with the counterintuitive "12 agents → 1" hook.
Introduce the complexity spectrum.
End with a clear mental model.

## Target Audience
AI engineers and technical leads building LLM-powered applications.

## Key Points
- A team planned 12 agents but shipped 1 — it worked better.
- The spectrum: workflows → single agent + tools → multi-agent. Stay left.
- Only 4 valid reasons for multi-agent.

## Tone
Direct, opinionated, engineer-to-engineer. No fluff.

Save this as guideline.md in the same working directory as research.md. Often, you use the key points from the guideline, plus some reference URIs, as input to the deep research agent.

Step 8: Generate a Post From the Research

Now run the writing workflow:

Write a LinkedIn post based on the research you just completed.
Use the research.md and guideline.md as source material.

The agent calls generate_post, which reads both files, generates an initial draft, and then runs the evaluator-optimizer loop: review the draft against the guideline, research, and writing profiles, edit to fix issues, repeat. You will see intermediate versions saved as post_0.md, post_1.md, etc. The final version is saved as post.md.

The initial draft (v0) is verbose with a weak hook. After three rounds of review and editing, the final version (v3) is tighter and structurally sound:

Each intermediate version is saved to disk so you can trace exactly what changed per pass. Browse more full examples in the [examples/](<https://github.com/iusztinpaul/designing-real-world-ai-agents-workshop/tree/main/examples>) directory.

How Claude Code Finds the Server: .mcp.json

In step 4, claude automatically launched and connected to both servers. Here is how that works.

Claude Code looks for a file called .mcp.json in the project root on startup. This file is the configuration that tells Claude Code which MCP servers exist, how to launch them, and what environment variables to pass. Here is the .mcp.json for this project:

{
  "mcpServers": {
    "deep-research": {
      "command": "uv",
      "args": ["run", "fastmcp", "run", "src/research/server.py"],
      "env": { "ENV_FILE_PATH": ".env" }
    },
    "linkedin-writer": {
      "command": "uv",
      "args": ["run", "fastmcp", "run", "src/writing/server.py"],
      "env": { "ENV_FILE_PATH": ".env" }
    }
  }
}

Each entry has a name (deep-research, linkedin-writer), the command to launch the server, and the path to the .env file so the server can read your API keys.

Both servers use stdio transport , meaning Claude Code launches each server locally as a child process and communicates with it via stdin/stdout. There are no ports to configure and no network setup. The alternative is HTTP transport , where the server runs independently (possibly on a remote machine) and the client connects via URL. For local development, stdio is simpler. We cover HTTP transport and remote deployment in Part 3 of the course.

from fastmcp import FastMCP
 
def create_mcp_server() -> FastMCP:
    settings = get_settings()
    mcp = FastMCP(name=settings.server_name, version=settings.version)
 
    register_mcp_tools(mcp)
    register_mcp_resources(mcp)
    register_mcp_prompts(mcp)
 
    return mcp
 
mcp = create_mcp_server()

Research Tools

The research server exposes three tools:

All tools also take a working_dir parameter for storing intermediate results. deep_research and analyze_youtube_video additionally share an exploration call budget tracked in .memory/, capped by MAX_EXPLORATION_CALLS. compile_research resets the budget when it runs, so a fresh research session starts with a clean counter.

deep_research sends a query to Gemini with Google Search grounding (live web search), gets back an answer with sources, and appends the result to .memory/research_results.json. It also returns the result directly to the agent. This dual-write pattern is important: the agent gets immediate feedback to reason about gaps, and the data persists on disk for compile_research to assemble later.

# src/research/app/research_handler.py
prompt = PROMPT_RESEARCH.format(query=query)
answer_text, raw_sources = await call_gemini_search(prompt)
sources = [
    ResearchSource(url=s["url"], title=s.get("title", ""), snippet="")
    for s in raw_sources
]
return ResearchResult(query=query, answer=answer_text, sources=sources)

The query is wrapped in a research prompt template, sent to Gemini with Google Search grounding via call_gemini_search, and the grounded sources are parsed into a typed ResearchResult. The tool wrapper around this (tools/deep_research_tool.py) handles persistence to .memory/research_results.json and budget tracking.

analyze_youtube_video passes a YouTube URL directly to Gemini using FileData(file_uri=url). Gemini watches the video natively and produces a transcript with timestamps and key insights. The output goes to .memory/transcripts/{video_id}.md.

# src/research/app/youtube_handler.py
prompt = PROMPT_YOUTUBE_TRANSCRIPTION.format(timestamp=timestamp)
parts = [
    types.Part(file_data=types.FileData(file_uri=url)),  # the video itself
    types.Part(text=prompt),                              # transcription instructions
]
response = await client.aio.models.generate_content(
    model=model_name, contents=types.Content(parts=parts)
)
output_path.write_text(response.text or "", encoding="utf-8")

The key line is FileData(file_uri=url). Gemini accepts the YouTube URL directly as a multimodal input, so no download or audio extraction is needed. The prompt asks for a transcript with [MM:SS] markers at a configurable interval (30 seconds by default), and the response is written straight to a markdown file under .memory/transcripts/.

💡 Most LLMs require you to download the video, extract the audio, transcribe it separately, and then pass the text to the model. Gemini handles all of this in one call. You just give it the URL. Google also offers a generous free tier, which makes it great for learning.

compile_research reads everything in .memory/ and assembles it into a single research.md file. It does not call any LLM; it is pure assembly logic that merges the accumulated search results and YouTube transcripts produced by the other two tools into one structured document.

# src/research/app/research_file_handler.py
research_results = load_json(memory_path / RESEARCH_RESULTS_FILE, default=[])
youtube_sources = [
    (f.stem, read_file(f))
    for f in sorted((memory_path / TRANSCRIPTS_FOLDER).glob("*.md"))
]
research_section = build_research_results_section(research_results)
youtube_section = build_sources_section(
    "## YouTube Video Transcripts", youtube_sources, "No YouTube video transcripts found."
)
return combine_research_sections(research_section, youtube_section)

Load the accumulated search results JSON, load every transcript markdown file, render each into a collapsible <details> section via the markdown helpers, and concatenate. The tool wrapper (tools/compile_research_tool.py) writes the result to research.md and resets the exploration budget.

What Happens While It Runs

Claude Code reads the topic and breaks it into sub-queries. It calls deep_research for each one, reads the results, identifies gaps, and fires off more queries to fill them. It repeats this process in a loop. Once it has enough coverage and has analyzed any provided YouTube videos, it calls compile_research to assemble everything.

The agent makes the substantive decisions: what to search, what to follow up on, and when coverage is sufficient. The MCP server does not decide what to search, but it does enforce an upper bound on how many exploration calls the agent can make before it must finalize. Within that budget, the agent is in control.

The .memory/ Folder

As the agent works, a .memory/ folder grows in the working directory:

working_dir/
├── .memory/
│   ├── research_results.json   # One entry appended per deep_research call
│   ├── exploration_state.json  # Tracks calls used against MAX_EXPLORATION_CALLS
│   └── transcripts/
│       └── CEvIs9y1uog.md      # One file per YouTube video
└── research.md                  # Final output

Research results can get large, and context windows have limits. Files also persist across tool calls, so nothing is lost if something fails mid-run. Since compile_research needs all results at once, having them on disk makes assembly straightforward.

The Research Workflow Prompt

The research_workflow MCP prompt (defined in src/research/routers/prompts.py) tells the agent how to run the full research pipeline:

You are a deep research agent. Use the available tools
to thoroughly research a topic.

**Workflow:**
1. Break the user's topic into multiple specific research queries.
2. Call deep_research for each query. Review results and identify gaps.
3. If the user provided YouTube URLs, call analyze_youtube_video.
4. If needed, run additional deep_research calls to fill knowledge gaps.
5. Call compile_research to generate the final research.md file.

The prompt tells the agent what to do but leaves most of the how to its judgment: what questions to ask and when coverage is sufficient. The number of exploration calls is bounded by the shared budget described in Step 6, and the call count returned by each tool gives the agent a running view of how much room it has left before it must call compile_research.

Writing Workflow: Code Walkthrough

The writing server (src/writing/server.py) follows the same FastMCP setup and router pattern as the research server. Its tools implement the generate-review-edit loop, taking research.md for grounded source material and guideline.md for the topic, angle, and audience.

Writing Tools

How generate_post Works

The evaluator-optimizer pattern separates generation from evaluation: one LLM writes, another reviews, and a third edits based on the review. Splitting these into separate calls with distinct prompts lets each step focus on a single concern.

When the agent calls generate_post, the entire pipeline runs inside the tool:

Phase 1: Load context (guideline, research, profiles, few-shot examples). Phase 2: Generate initial draft. Phase 3: Review and edit loop runs multiple iterations to improve the post.

• Context loading. The tool reads guideline.md (the user input on what to write about), research.md (factual material from the research agent), all four writing profiles (structure, terminology, character, branding), and a set of real LinkedIn posts from the datasets/ folder as few-shot examples. All of this goes into the prompt. The datasets/linkedin_paul_iusztin folder contains 20 real LinkedIn posts from Paul Iusztin’s profile sampled by engagement (>150 reactions). The dataset is indexed through the index.yaml file which contains the original text, media, a seed, a guideline, research, and a generated version. The dataset is split by purpose: some posts serve as few-shot examples during generation, some as reference images for the image generator, and others as dev and test splits for evaluation. During generation, the system pulls ~5 few-shot examples to show the LLM what good posts look like. During evaluation, the system uses the dev/test splits to comparable quality.
• Generation calls write_post(), which sends everything to Gemini to produce the first draft.
• Review-edit loop (runs 4 iterations by default):
  • review_post() evaluates the draft against the guideline (the user input), the research, and every writing profile, and returns structured feedback: a list of Review objects specifying which rule was violated, where in the post, and what is wrong.
  • If issues are found, edit_post() receives both the draft, the writing context, and the reviews, then rewrites to fix the issues while maintaining flow and style.
  • If no issues are found, the edit is skipped for that iteration, but the loop does not exit. The next review pass still runs against the unchanged draft, which can surface issues the previous pass missed.

The whole loop fits in about 10 lines of Python:

# src/writing/app/generate_post.py
post = await write_post(guideline, research, profiles, post_examples_text)
versions = [post]
all_reviews = []

for i in range(settings.num_reviews):  # 4 by default
    reviews_result = await review_post(post, guideline, profiles)
    all_reviews.append(reviews_result)
    if not reviews_result.reviews:
        continue  # no issues: skip the edit, run the next review
    post = await edit_post(
        post, reviews_result.reviews, guideline, research, profiles, post_examples_text
    )
    versions.append(post)

write_post produces the first draft. The loop then runs settings.num_reviews times (4 by default). Each iteration calls review_post to get a list of structured Review objects. If the list is empty, the iteration skips straight to the next review without rewriting; if not, edit_post rewrites the draft using the reviews as targeted feedback, and the new version is appended to the versions. Because versions only grows on edits, the number of saved files reflects the number of edits, not the number of review passes. A run with one clean iteration produces fewer post_N.md files than the configured iteration count. The filenames stay contiguous (post_0.md, post_1.md, ...), but there are simply fewer of them. The MCP tool wrapper (tools/generate_post_tool.py) handles I/O around this. It reads guideline.md and research.md, passes them in, then writes each entry in versions to post_0.md, post_1.md, and so on, with the final draft also saved as post.md. The structured Review objects collected in all_reviews are also persisted, written to .memory/reviews_1.json, .memory/reviews_2.json, and so on, so you can inspect not just every draft but also the exact violations that triggered each rewrite.

• The**** finalOutput is saved as post.md. All intermediate versions are also stored (post_0.md, post_1.md, etc.) so you can see how the post improves with each pass.

Writing Profiles

Writing profiles are plain Markdown files in src/writing/profiles/that define how the post should look, read, and sound. Every prompt in the writing workflow (generation, review, and editing) includes the structure, terminology, and character profile.

The structure profile defines the format and length. For example: 800-1500 characters, the hook in the first two lines (bold claim, surprising stat, or contrarian take), 1-3-sentence paragraphs, end with a CTA, and zero hashtags.

The terminology profile defines the language rules, sentence style, banned words, and phrases. This agent includes a list of 40+ banned AI slop words and marketing terms:

• Slop words: “delve”, “tapestry”, “vibrant”, “landscape”, “realm”, “embark”, “foster”, “unleash”, “beacon”, “poised”, “unravel”, “enrich”, “multifaceted”, “intricate”, “paramount”, “groundbreaking”, “cutting-edge”.
• Marketing terms: “seamlessly”, “leverage”, “robust”, “scalable”, “innovative”, “revolutionary”, “disruptive”, “synergy”, “paradigm shift”.

The profiles are configured once and injected into every prompt. They control format, language, voice, and visual identity.

The character profile defines the voice and tones. For example, to be confident and direct, professional but conversational, with real examples over theory. The profile also defines on-brand tones (excited, frank, curious) and off-brand tones (salesy, preachy, condescending).

The branding profile is used only for image generation. It defines image characteristics such as color palette, style, text, and design elements. For example, a black/white/orange color palette, minimalist style, abstract/conceptual, no text overlay, no faces.

How the Review Works

The reviewer returns structured, targeted output using Pydantic models, a technique you will learn in Lesson 4 (Structured Outputs). Each review is a Python object with three fields:

class Review(BaseModel):
    profile: str     # Which rule was violated ("terminology_profile", "guideline", etc.)
    location: str    # Where in the post ("Hook", "Paragraph 3", "CTA")
    comment: str     # What's wrong ("Uses 'leverage' - banned term")

The reviewer produces structured feedback: which profile was violated, where in the post, and what specifically is wrong.

It returns which rule was broken, where in the post it was broken, and what to fix. For example, if the reviewer finds that the post uses “leverage”, it returns:

Review(
    profile="terminology_profile",
    location="Paragraph 3",
    comment="Uses 'leverage' — banned term. Replace with a direct verb like 'use' or 'apply'."
)

The system also enforces a priority order when multiple issues exist: human feedback (when provided) > guideline > research > structure > terminology > character.

Once the reviewer produces its list of Review objects, the internal editor rewrites the post to address every flagged issue while preserving flow and style. It receives the current draft, all reviews, the guideline, the research, and the writing profiles.

The complete loop looks like this: generate, review, edit, repeat for 4 iterations.

Why Is There No Scoring Threshold

In many evaluator-optimizer systems, the loop stops when a quality score crosses a threshold. We deliberately chose not to do that here. Creative writing is highly subjective. There is no reliable numeric score that captures whether a LinkedIn post “sounds right.” Instead, the loop runs all 4 review-edit cycles unconditionally, and if a review round finds no issues, the edit is simply skipped for that round.

The automated loop gives you a solid baseline, but it cannot guarantee the result matches your taste.

Editing a Post

The edit_post function is intentionally used in two places.

1. Inside**generate_post**, the internal evaluator-optimizer loop calls edit_post during each review-edit cycle. This happens automatically and behind the scenes as part of the iterations using the reviews as feedback.
2. As an MCP tool , the edit_post is called after the generation is complete. It runs one review-edit pass with adjusting the post based on your feedback.

The same functions run for both, but in different contexts

How edit_post Works With Human Feedback

After generate_post finishes its 4 internal rounds and saves post.md, you read the result and decide whether it needs changes. If it does, you call edit_post with your feedback. It can be something like “Make the opening more provocative” or “Add a concrete number from the research.” The tool runs one review-edit pass with your feedback injected at the highest priority. It produces structured Review objects the same way the internal loop does, and the editor rewrites accordingly. The updated post.md is saved in place.

You can call edit_post as many times as needed. Each call is one pass, and there is no limit on manual iterations.

Image Generation

The generate_image tool creates a LinkedIn post image in two steps. First, it reads post.md and uses a text LLM to extract a visual scene description from the content (abstract, geometric, no text overlays). Then it passes that scene description along with the branding profile (black/white/orange palette, minimalist style) and reference images from the datasets/ folder to Gemini Flash Image. The output is a 1200x1200 post_image.png.

Agent Skills

The MCP servers expose tools and prompts, but Claude Code needs a way to know which prompt to load and when. Without that, you would type out the full instruction every time: “load the research_workflow prompt from the deep-research server and follow it.”

Agent Skills (agentskills.io) is an open standard for packaging reusable, portable instructions that any AI coding agent can follow. A skill is a markdown file placed at .claude/skills/<name>/SKILL.md. It works across Claude Code, Cursor, Gemini CLI, GitHub Copilot, and others. No installation, no registration, no configuration. You drop the file in the folder, and the agent picks it up.

Here is the research skill from this project:

---
name: research
description: "Run deep research on any topic using the Deep
  Research MCP server. Triggers on: 'research this', 'find out
  about', 'gather information on', 'deep dive into'..."
---

# Research

Research a topic using the `deep-research` MCP server.

## Execution
1. Load the research_workflow MCP prompt from the deep-research server.
2. Follow the workflow instructions using the available tools.
3. Use the current working directory as the working_dir.

Note: The version shown above is simplified for clarity. The actual SKILL.md in the repo includes additional logic for deriving a working directory from the topic name (e.g., outputs/{slug}/). Check the repo for the full version: .claude/skills

The file has two parts. The YAML frontmatter (between the --- markers) is metadata. The name field becomes a slash command (/research), so you can type /research in Claude Code to trigger it. The description field tells the agent when the skill is relevant, so even without a slash command, saying “research this topic” in natural language automatically triggers it.

The markdown body is the playbook. It tells the agent what to do: load the MCP prompt, follow the workflow, and use the current directory. Notice that the skill does not contain any research logic itself. It just points to the MCP prompt, which is the single source of truth. If the server’s workflow changes, the skill does not need to be updated. This is a design choice. You could put all the instructions directly in the skill, but keeping the workflow in the MCP prompt means the logic lives on the server, and any MCP client that connects can load the same prompt. Note that this works here because the servers run locally via stdio. With remote HTTP servers, prompt loading works differently.

Skills use progressive disclosure to stay efficient. At startup, Claude Code loads only the frontmatter of all installed skills, keeping the context window lightweight. When a skill triggers, the full markdown body loads into context. After execution, it leaves context. This means you can have dozens of skills installed without wasting tokens or slowing the agent down.

Try it: type /research “What are agent skills” in Claude Code and watch the full research pipeline run. Then create a guideline.md with your topic, angle, and audience, and type /write-post to generate a LinkedIn post from the research and guidelines.

Monitoring

Before you can score anything, you have to be able to see what the system is actually doing. Debugging workflows and agents through terminal logs is painful, especially once you have agent thinking traces, multiple LLM calls per run, and tool call branching. The fix is monitoring: a tool that captures every trace (each LLM and tool call with full input, output, and metadata), plus per-step latency and cost. Those stored traces also become the input for evals later. One dataset, two uses.

Opik does this for both servers in this project automatically. The monitoring is organized in three layers:

• Threads group the full run end-to-end. For the writing workflow, a thread captures the whole "generate post plus image" sequence, including all the back-and-forth between the user and the LLM. A popular example is the chat session from Gemini or ChatGPT, which is modeled as a single thread.
• Traces sit inside threads. A trace is one request-response between the client and the server, like a single generate_post call, and it shows total latency, total tokens, and total cost.
• Spans sit inside traces. They are the individual tool and LLM calls that make up the operation, with per-call model, latency, and token counts.

Every LLM and tool call in the pipeline is traced. A failure links back to the exact prompts that produced it.

The same setup runs for the research server, where threads capture the full deep_research plus compile_research sequence and each tool call lands as its own trace.

Evals

The review loop catches violations during generation, but it does not tell you whether the system as a whole is improving. Changing a writing profile, adjusting a prompt, or swapping the underlying model could push output up or down, and without a way to measure across runs, you are relying on intuition. When you have one generated post, you can read it carefully and decide if it is good. With ten posts, you read a few and skim the rest. With a hundred, you stop reading altogether. The problem gets worse as the system grows. Every new feature you ship can break a prompt that was working yesterday, because the surrounding context has shifted. Catching regressions like that is standard practice in software engineering. Catching them in prompts, where a single word can change the output, is much harder.

Evals are the layer that catches that. Production systems need an answer to two questions: is this run better than the last one, and is the metric I am using actually trustworthy?

Why evals matter, and where they fit

Evals show up in three layers in this system, in order from development to production:

• Optimization. This is similar to training a model. You have a quality score, you change something (a profile, a prompt, the model), you re-run the eval, and you see whether the score moved in the right direction.
• Regression testing. Same as in software engineering: when you add a feature, you re-run the eval in CI/CD to make sure you did not break things that were already working.
• Production monitoring. Run the judge continuously on live outputs and surface alerts when quality drops.

The thing that makes this different from unit tests is that evals start with a dataset, not assertions. The pipeline behaves like a small ML training loop: you build a dataset, build a classifier on top of it, validate it on held-out data, then run it. The dataset is the lever everything else rests on.

Building the eval dataset

Building the dataset is the hardest part of setting up an LLM-as-judge, and most implementations skip it.

The datasets/linkedin_paul_iusztin/ folder contains 20 real LinkedIn posts, sampled by engagement (>150 reactions). Twenty is small. For a workshop, it works, but in practice, you want at least 100. The construction process matters more than the size:

• Extract. Start with real outputs. The 20 posts are real LinkedIn content that already worked.
• Reverse-engineer. For each post, write the guideline that could have produced it (topic, angle, audience, key points), then run the deep research agent against that guideline to produce the supporting research material.
• Generate. Feed the guidelines and research into the writing workflow. Save the generated post.

The Generate step has one rule. Do not ask an LLM to write the post you are going to evaluate. Ask the LLM to help you write the guideline, then feed that guideline into the writing workflow to produce the post. The post must come from the writing workflow itself. If you let the LLM write the post directly and then evaluate it, your eval is measuring how good that LLM is at writing posts, not how good your writing workflow is.

• Label. Read each generated post, give it a binary pass or fail, and write a one-to-three-sentence critique explaining the decision. To keep labeling fast, stop at the first reason to fail. Once you find one, write the critique for that reason and move on. This also gives the judge a consistent failure pattern to learn from.
• Split. Split by purpose, not by row. A row in dev_evaluator or test_evaluator never appears in any training split. If it does, the judge labels a post that it has already seen, and F1 inflates. Generator and judge few-shot can overlap when data is scarce. The judge stays independent because its examples carry a signal the generator never sees: the expert label and critique.

The dataset is split as follows:

Building the LLM judge

The judge is a BinaryLLMJudgeMetric defined in src/writing/evals/metric.py. It takes a generated post plus the context that was used to produce it (guideline, research, profiles), and returns a pass or fail label with a short critique. Three design choices are worth calling out, because each one is a place where this kind of system usually goes wrong.

It is binary. The judge returns a pass or fail, not a numeric score on a scale from 1 to 5. Numeric scores easily drift during the label processing and are harder for the LLM to predict correctly. Ultimately, what’s the difference between 2 and 3? A binary verdict is harder to game and easier to compare against expert labels. The nuance lives in the short critique the judge returns alongside it.

The system prompt is simple. The few-shot examples do the work. The judge prompt has profile rules and labeling instructions, but the part that actually drives accuracy is the handful of labeled examples loaded from train_evaluator and added to the prompt:

# src/writing/evals/metric.py
train_entries = load_by_scope("train_evaluator")
for entry in train_entries:
    if entry.label is None or entry.critique is None:
        continue
    guideline = entry.guideline_content(DATASET_DIR)
    research = entry.research_content(DATASET_DIR)
    generated = entry.generated_content(DATASET_DIR)
    if generated and guideline:
        few_shot_examples.append(
		        _FewShotExample(
                        guideline=guideline,
                        research=research,
                        generated_post=generated,
                        label=entry.label.value,
                        critique=entry.critique,
            )
        )
        
self._few_shot_section = _build_few_shot_section(few_shot_examples)

Each example shows the judge a guideline, research, a generated post, the expert label, and the critique that justifies the label. The judge calibrates to your metric through these examples. The prompt rules are necessary but not sufficient.

As we “train” a binary classifier, there is no ground truth post. The judge evaluates the generated post only against the guideline, the research, and the profiles. Then it outputs the pass/fail label. There is no "real" post to compare against. This matches production conditions, where no ground truth is available.

Measuring whether the judge is reliable

Validating the judge is part of the eval pipeline, not an optional step. The judge is itself an LLM, and its verdicts need to be checked against expert labels before its scores can be trusted.

The judge is a binary classifier. We chose to use an LLM as the classifier because it’s the simplest solution, but we could have used any other AI model that takes text as input. The standard tool for evaluating a binary classifier is F1, the harmonic mean of precision and recall. The pipeline runs the judge on every entry in the dev_evaluator and test_evaluator splits, compares the judge's verdict to the expert label, and computes F1.

The workflow is the same one you'd use to train any binary classifier:

• Run the judge on the dev split and compute F1.
• Adjust the judge: tweak the few-shot examples, tighten the critiques, swap the underlying model.
• Repeat until F1 converges at a number you are happy with.
• Run on the test split as the final validation. This is the held-out check, run only when you think you are done.

If F1 is high on dev and low on test, the judge has overfit to the dev split. The fix is to expand the dev split with new samples and recalibrate. If F1 is high on both, the absolute number reflects how strict your labels are. Two teams with different tolerance levels can both hit F1 = 0.9 on their own data and disagree on most actual posts.

Wiring it all together

The full eval architecture has four pieces and one platform underneath: the dataset, the LLM judge built on top of it, the calibration loop on dev/test, and the run on real data, all observed through Opik.

You can run all three modes from the terminal:

make eval-dev               # LLM judge on dev split, scored against expert labels
make eval-test              # LLM judge on test split, scored against expert labels
make eval-online            # Generate + judge a fresh post against online_test (no labels)

eval-dev and eval-test upload the dataset to Opik, run the judge over each entry, and report F1 against the expert labels. eval-online runs against the online_test split, which has no expert labels by design. You generate a post from scratch and judge it, simulating production conditions where automated scoring is all you have. There is no F1 to report. You get the raw pass rate plus the per-post critiques. This is what production scoring looks like, and the point of running it is to see whether the same judge that scored well on dev and test still produces sensible verdicts on posts it has never seen.

You can view every run in the Opik dashboard at comet.com/site/products/opik, where each LLM call is traced with inputs, outputs, scores, and latency.

View from Opik after running the LLM judge on a sample from the online_test split, where we can see the generated post, the binary score and critique.

What’s Next

You have a working research agent and writing workflow running on your machine. It takes a topic, runs autonomous web research and video analysis, compiles grounded source material, generates a draft against writing profiles, reviews its own output for violations, and edits through multiple passes with every intermediate version saved to disk. You can inject your own feedback at any point and run as many manual editing passes as you want.

The full course teaches you to build every part of this from scratch. Part 1 focuses on building a foundational understanding of workflow and agentic patterns from scratch. It starts with how LLMs handle context and how to control the information flow into each call, then moves to extracting structured data from model responses, which is how the review objects in the writing workflow get their precise format. From there, you build tool-calling agents, implement ReAct-style reasoning loops, and learn the workflow patterns that determine how components connect: chaining, routing, parallelization, and orchestrator-worker.

Part 2 extends the research and writing system well beyond this prototype. The research agent stays with Claude Code as the orchestrator but becomes significantly more capable: it gains support for code execution, notebook analysis, and web scraping as additional research sources, plus a post-processing pipeline that ranks and filters retrieved material so only high-quality sources make it into the final research document.

The writing workflow is rebuilt in LangGraph with explicit state management, checkpointing between review-edit passes, and the ability to add new stages, such as semantic memory, that draws on how a concept was explained in previous outputs. The evaluation pipeline expands from the binary judge in this lesson to multi-dimensional scoring, with deeper treatment of how to build datasets, calibrate judges against human annotations, measure judge stability across runs, and understand the reasoning behind each design choice.

Part 3 takes the system off your laptop. Deployment to Google Cloud Run with Docker, a PostgreSQL backend for persistent state, authentication, and user sessions with Descope, and CI/CD with GitHub Actions. The course also walks through design decisions that a code walkthrough cannot cover: why the research side started as a hardcoded LangGraph workflow and had to be restructured into an MCP server with natural language prompts, why the writing workflow looks as it is today, and how to choose between reasoning and non-reasoning models for different pipeline steps as a trade-off between cost, latency, and performance.

By the end, you will have the production version of this system, a full understanding of how every component was built, and a capstone project where you apply everything to a domain of your choice.

Let’s get started.