Lesson 18: The Research Loop: Query Generation, Perplexity, and Human Feedback

3. Repeat the following research loop for 3 rounds:

    3.1. Run the "generate_next_queries" tool to analyze the ARTICLE_GUIDELINE_FILE, the already-scraped guideline
    URLs, and the existing PERPLEXITY_RESULTS_FILE. The tool identifies knowledge gaps, proposes new web-search
    questions, and writes them - together with a short justification for each - to the NEXT_QUERIES_FILE within
    NOVA_FOLDER.

    3.2. Run the "run_perplexity_research" tool with the new queries. This tool executes the queries with
    Perplexity and appends the results to the PERPLEXITY_RESULTS_FILE within NOVA_FOLDER.

async def generate_next_queries_tool(research_directory: str, n_queries: int = 5) -> Dict[str, Any]:
    """
    Generate candidate web-search queries for the next research round.
 
    Analyzes the article guidelines, already-scraped content, and existing Perplexity
    results to identify knowledge gaps and propose new web-search questions.
    Each query includes a rationale explaining why it's important for the article.
    Results are saved to next_queries.md in the research directory.
    """
    # Convert to Path object
    research_path = Path(research_directory)
    nova_path = research_path / NOVA_FOLDER
 
    # Gather context from the research folder
    guidelines_path = research_path / ARTICLE_GUIDELINE_FILE
    results_path = nova_path / PERPLEXITY_RESULTS_FILE
    urls_from_guidelines_dir = nova_path / URLS_FROM_GUIDELINES_FOLDER
 
    article_guidelines = read_file_safe(guidelines_path)
    past_research = read_file_safe(results_path)
 
    # Collect all scraped content for context
    scraped_ctx_parts: List[str] = []
    if urls_from_guidelines_dir.exists():
        for md_file in sorted(urls_from_guidelines_dir.glob(f"*{MARKDOWN_EXTENSION}")):
            scraped_ctx_parts.append(md_file.read_text(encoding="utf-8"))
    scraped_ctx_str = "\n\n".join(scraped_ctx_parts)
 
    # Generate queries using LLM
    queries_and_reasons = await generate_queries_with_reasons(
        article_guidelines, past_research, scraped_ctx_str, n_queries=n_queries
    )
 
    # Write to next_queries.md (overwrite)
    next_q_path = nova_path / NEXT_QUERIES_FILE
    write_queries_to_file(next_q_path, queries_and_reasons)
 
    return {
        "status": "success",
        "queries_count": len(queries_and_reasons),
        "queries": queries_and_reasons,
        "output_path": str(next_q_path.resolve()),
        "message": f"Successfully generated {len(queries_and_reasons)} candidate queries..."
    }

async def generate_queries_with_reasons(
    article_guidelines: str,
    past_research: str,
    scraped_ctx: str,
    n_queries: int = 5,
) -> List[Tuple[str, str]]:
    """Return a list of tuples (query, reason)."""
 
    prompt = PROMPT_GENERATE_QUERIES_AND_REASONS.format(
        n_queries=n_queries,
        article_guidelines=article_guidelines or "",
        past_research=past_research or "",
        scraped_ctx=scraped_ctx or "",
    )
 
    chat_llm = get_chat_model(settings.query_generation_model, GeneratedQueries)
    response = await chat_llm.ainvoke(prompt)
 
    queries_and_reasons = [(item.question, item.reason) for item in response.queries]
    return queries_and_reasons[:n_queries]

PROMPT_GENERATE_QUERIES_AND_REASONS = """
You are a research assistant helping to craft an article.

Your task: propose {n_queries} diverse, insightful web-search questions
that, taken **as a group**, will collect authoritative sources for the
article **and** provide a short explanation of why each question is
important.


{article_guidelines}


{past_research}


{scraped_ctx}


Guidelines for the set of queries:
• Give priority to sections/topics from the article guidelines that
  currently lack supporting sources in  and
  .
• Cover any remaining major sections to ensure balanced coverage.
• Avoid duplication; each query should target a distinct aspect.
• The web search queries should be natural language questions, not just keywords.
""".strip()

class QueryAndReason(BaseModel):
    """A single web-search query and the reason for it."""
 
    question: str = Field(description="The web-search question to research.")
    reason: str = Field(description="The reason why this question is important for the research.")
 
class GeneratedQueries(BaseModel):
    """A list of generated web-search queries and their reasons."""
 
    queries: List[QueryAndReason] = Field(description="A list of web-search queries and their reasons.")

from research_agent_part_2.mcp_server.src.app.generate_queries_handler import generate_queries_with_reasons
 
# Example inputs (simplified for demonstration)
article_guidelines = '''
# Article: Advanced Function Calling with LLMs
 
## Sections to cover:
1. Error handling strategies
2. Performance optimization
3. Security considerations
4. Best practices for production
'''
 
past_research = '''
### Source [1]: https://example.com/basic-function-calling
Query: What is function calling in LLMs?
Answer: Function calling allows LLMs to invoke external tools and APIs...
 
### Source [2]: https://example.com/simple-examples  
Query: How to implement basic function calling?
Answer: Basic implementation involves defining function schemas...
'''
 
scraped_ctx = '''
# Function Calling Documentation
This guide covers the fundamentals of function calling...
[Basic examples and simple use cases already covered]
'''
 
# Generate queries based on this context
queries_and_reasons = await generate_queries_with_reasons(
    article_guidelines=article_guidelines,
    past_research=past_research,
    scraped_ctx=scraped_ctx,
    n_queries=3
)
 
for query, reason in queries_and_reasons:
    print(f"Query: {query}")
    print(f"Reason: {reason}")
    print("---")

Query: What are the most common security vulnerabilities when giving LLMs access to external tools via function calling, and what are the established mitigation strategies?
Reason: This question directly targets the 'Security considerations' section, a critical topic for production systems that is not covered in the existing research. It seeks to find authoritative sources on risks and specific, actionable solutions.
---
Query: How can latency be minimized and token usage be optimized when using chained or parallel function calls with large language models?
Reason: This addresses the 'Performance optimization' section. It focuses on advanced, practical challenges like speed and cost-efficiency, especially in complex scenarios that go beyond single, basic function calls.
---
Query: What are robust error handling patterns for LLM function calling when external API calls fail, return unexpected data, or the LLM hallucinates a function call?
Reason: This covers the 'Error handling strategies' section. It aims to find expert advice on building resilient systems, which is a cornerstone of production-readiness and a key theme for an 'advanced' article.
---

async def run_perplexity_research_tool(research_directory: str, queries: List[str]) -> Dict[str, Any]:
    """
    Run Perplexity research queries for the research folder.
 
    Executes the provided queries using Perplexity's Sonar-Pro model and appends
    the results to perplexity_results.md in the research directory. Each query
    result includes the answer and source citations.
    """
    research_path = Path(research_directory)
    nova_path = research_path / NOVA_FOLDER
    results_path = nova_path / PERPLEXITY_RESULTS_FILE
 
    if not queries:
        return {
            "status": "success",
            "message": "No queries provided – nothing to do.",
            "queries_processed": 0,
            "sources_added": 0,
        }
 
    # Execute all queries concurrently
    tasks = [run_perplexity_search(query) for query in queries]
    search_results = await asyncio.gather(*tasks)
 
    # Process and append search results to file
    total_sources = append_search_results_to_file(results_path, queries, search_results)
 
    return {
        "status": "success",
        "queries_processed": len(queries),
        "sources_added": total_sources,
        "output_path": str(results_path.resolve()),
        "message": f"Successfully completed Perplexity research round..."
    }

class SourceAnswer(BaseModel):
    """A single source answer with URL and content."""
 
    url: str = Field(description="The URL of the source")
    answer: str = Field(description="The detailed answer extracted from that source")
 
class PerplexityResponse(BaseModel):
    """Structured response from Perplexity search containing multiple sources."""
 
    sources: List[SourceAnswer] = Field(description="List of sources with their answers")
 
async def run_perplexity_search(query: str) -> Tuple[str, Dict[int, str], Dict[int, str]]:
    """Run a Perplexity Sonar-Pro search and return full answer + parsed sections."""
 
    # run perplexity search with structured output
    llm = get_chat_model("perplexity", PerplexityResponse)
 
    prompt = PROMPT_WEB_SEARCH.format(query=query)
    logger.debug(f"Searching web for: {query} …")
 
    response = await llm.ainvoke(prompt)
 
    # Convert structured response to the expected format
    answer_by_source = {}
    citations = {}
    for i, source in enumerate(response.sources, 1):
        answer_by_source[i] = source.answer
        citations[i] = source.url
 
    # Create a readable full answer string for compatibility
    full_answer_lines = []
    for i, source in enumerate(response.sources, 1):
        full_answer_lines.append(f"### [{i}]: {source.url}")
        full_answer_lines.append(source.answer)
        full_answer_lines.append("")
    full_answer = "\n".join(full_answer_lines)
 
    return full_answer, answer_by_source, citations

# Example of what run_perplexity_search returns for a query about "LLM function calling best practices"

full_answer = '''### [1]: https://openai.com/docs/guides/function-calling
Function calling allows models to connect to external tools and APIs...

### [2]: https://docs.anthropic.com/claude/docs/tool-use
Claude can use tools to perform actions beyond text generation...

### [3]: https://ai.google.dev/gemini-api/docs/function-calling
Gemini models support function calling for structured interactions...
'''

answer_by_source = {
    1: "Function calling allows models to connect to external tools and APIs. Best practices include defining clear schemas, handling errors gracefully, and validating inputs...",
    2: "Claude can use tools to perform actions beyond text generation. Key considerations include security, rate limiting, and proper error handling...",
    3: "Gemini models support function calling for structured interactions. Important aspects include schema design, response validation, and performance optimization..."
}

citations = {
    1: "https://openai.com/docs/guides/function-calling",
    2: "https://docs.anthropic.com/claude/docs/tool-use",
    3: "https://ai.google.dev/gemini-api/docs/function-calling"
}

PROMPT_WEB_SEARCH = """
Question: {query}

Provide a detailed answer to the question above.
The answer should be organized into source sections, where each source section
contains all the information coming from a single source.
Never use multiple source citations in the same source section. A source section should refer to a single source.
Focus on the official sources and avoid personal opinions.
For each source, write as much information as possible coming from the source
and that is relevant to the question (at most 300 words).

Return a list of objects, where each object represents a source and has the following fields:
- url: The URL of the source
- answer: The detailed answer extracted from that source
""".strip()

from research_agent_part_2.mcp_server.src.tools import generate_next_queries_tool
 
# Update this path to your actual sample research folder
research_folder = "/path/to/research_folder"
result = await generate_next_queries_tool(research_directory=research_folder, n_queries=3)
print(result)

{
  "status": "success",
  "queries_count": 3,
  "queries": [
    ("What are the latest best practices for implementing function calling with LLMs?", "This query addresses implementation details that may not be fully covered in the basic documentation from the guidelines."),
    ("How do different LLM providers handle function calling differently?", "Understanding provider-specific approaches will help create more comprehensive guidance."),
    ("What are common pitfalls and debugging techniques for function calling?", "Practical troubleshooting information is essential for developers implementing these systems.")
  ],
  "output_path": "/path/to/research_folder/.nova/next_queries.md",
  "message": "Successfully generated 3 candidate queries for research folder..."
}

from research_agent_part_2.mcp_server.src.tools import run_perplexity_research_tool
 
research_folder = "/path/to/research_folder"
 
# Example queries to test
test_queries = [
    "What are the latest developments in LLM function calling?",
    "How do you handle errors in AI agent tool execution?"
]
 
result = await run_perplexity_research_tool(
    research_directory=research_folder,
    queries=test_queries
)
print("Tool call output:")
print(result)
print()
 
print("Content of the resulting file:")
with open(result["output_path"], "r") as f:
    print(f.read())

Tool call output:
{'status': 'success', 'queries_processed': 2, 'sources_added': 7, 'output_path': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/perplexity_results.md', 'message': "Successfully completed Perplexity research round for research folder '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder'. Processed 2 queries and added 7 source sections to perplexity_results.md"}

Content of the resulting file:
### Source [1]: https://arxiv.org/html/2505.20192v1

Query: What are the latest developments in LLM function calling?

Answer: The paper introduces **FunReason**, a novel framework designed to enhance large language models’ (LLMs) function calling capabilities. FunReason addresses the challenge of combining detailed reasoning with accurate function execution—a known limitation of prior approaches—by employing an automated data refinement strategy and a Self-Refinement Multiscale Loss (SRML) technique. This approach allows LLMs to generate high-quality training examples, focusing on three aspects: query parseability, reasoning coherence, and function call precision. The SRML dynamically balances reasoning and function call accuracy during training, overcoming the typical trade-off between these factors. The authors report that FunReason achieves performance on par with GPT-4o and effectively mitigates catastrophic forgetting during fine-tuning. The work positions function calling as central for LLM practical utility, highlighting a shift from prompt engineering to more data-driven and fine-tuned approaches, including reinforcement learning (RL)-based optimization for task success. The availability of code and datasets is emphasized, supporting reproducibility and practical adoption.

-----
...

from research_agent_part_2.mcp_client.src.client import main as client_main
import sys
 
async def run_client():
    _argv_backup = sys.argv[:]
    sys.argv = ["client"]
    try:
        await client_main()
    finally:
        sys.argv = _argv_backup
 
# Start client with in-memory server
await run_client()

🛠️  Available tools: 11
📚 Available resources: 2
💬 Available prompts: 1

Available Commands: /tools, /resources, /prompts, /prompt/, /resource/, /model-thinking-switch, /quit


 🤔 LLM's Thoughts:
**Ready to Begin the Research Workflow!**

Okay, so I'm geared up and ready to dive into this research project. The user has given me the go-ahead, so let's get rolling. First things first, I need to outline the workflow. I'll make sure the user understands what I'm about to do, so they know what to expect.
...
💬 LLM Response: Here's a breakdown of the research workflow steps:

1.  **Setup**: Extracts URLs (GitHub, YouTube, other web links) and local file references from your `ARTICLE_GUIDELINE_FILE`.
2.  **Process Extracted Resources**: Simultaneously processes the extracted resources:
    *   Copies referenced local files to a dedicated folder.
    *   Scrapes and cleans content from other web URLs.
    *   Processes GitHub URLs to create summaries and content.
    *   Transcribes YouTube videos.
3.  **Research Loop**: Repeats for 3 rounds:
    ...