Lesson 19: Final Outputs and Agent Completion

4. Filter Perplexity results by quality:
    4.1 Run the "select_research_sources_to_keep" tool to automatically evaluate each source for trustworthiness, authority and relevance.

5. Identify which of the accepted sources deserve a full scrape:
    5.1 Run the "select_research_sources_to_scrape" tool to choose up to 5 diverse, authoritative sources whose full content will add most value.
    5.2 Run the "scrape_research_urls" tool to scrape/clean each selected URL's full content.

6. Write final research file:
    6.1 Run the "create_research_file" tool to combine all research data into a
    comprehensive research.md file.

async def select_research_sources_to_keep_tool(research_directory: str) -> Dict[str, Any]:
    """
    Automatically select high-quality sources from Perplexity results.
 
    Uses an LLM to evaluate each source in perplexity_results.md for trustworthiness,
    authority, and relevance based on the article guidelines. Writes the comma-separated
    IDs of accepted sources to perplexity_sources_selected.md and saves a filtered
    markdown file perplexity_results_selected.md containing only the accepted sources.
 
    Args:
        research_directory: Path to the research directory containing article guidelines and research data
 
    Returns:
        Dict with status, selection results, and file paths
    """
		# Convert to Path object
    research_path = Path(research_directory)
    nova_path = research_path / NOVA_FOLDER
 
		# Gather context from the research folder
    guidelines_path = research_path / ARTICLE_GUIDELINE_FILE
    results_path = nova_path / PERPLEXITY_RESULTS_FILE
 
    article_guidelines = read_file_safe(guidelines_path)
    perplexity_results = read_file_safe(results_path)
 
		# Use LLM to select sources
    selected_ids = await select_sources(
        article_guidelines, perplexity_results, settings.source_selection_model
    )
 
		# Write selected source IDs to file
    sources_selected_path = nova_path / PERPLEXITY_SOURCES_SELECTED_FILE
    sources_selected_path.write_text(",".join(map(str, selected_ids)), encoding="utf-8")
 
		# Extract and save filtered content
    filtered_content = extract_selected_blocks_content(selected_ids, perplexity_results)
    results_selected_path = nova_path / PERPLEXITY_RESULTS_SELECTED_FILE
    results_selected_path.write_text(filtered_content, encoding="utf-8")
 
    return {
        "status": "success",
        "sources_selected_count": len(selected_ids),
        "selected_source_ids": selected_ids,
        "sources_selected_path": str(sources_selected_path.resolve()),
        "results_selected_path": str(results_selected_path.resolve()),
        "message": f"Successfully selected {len(selected_ids)} high-quality sources..."
    }

PROMPT_SELECT_SOURCES = """
You are a research quality evaluator. Your task is to evaluate web sources for an upcoming article
and select only the high-quality, trustworthy sources that are relevant to the article guidelines.


{article_guidelines}


Here are the sources to evaluate:

{sources_data}


**Selection Criteria:**
- ACCEPT sources from trustworthy domains (e.g., .edu, .gov, established news sites,
official documentation, reputable organizations)
- ACCEPT sources with high-quality, relevant content that directly supports the article guidelines
- REJECT sources from obscure, untrustworthy, or potentially biased websites
- REJECT sources with low-quality, irrelevant, or superficial content
- REJECT sources that seem to be marketing materials, advertisements, or self-promotional content

Return your decision as a structured output with:
1. selection_type: "none" if no sources meet the quality standards, "all" if all sources are acceptable,
or "specific" for specific source IDs
2. source_ids: List of selected source IDs
""".strip()

from research_agent_part_2.mcp_server.src.tools import select_research_sources_to_keep_tool
 
# Test the source selection tool
research_folder = "/path/to/research_folder"
result = await select_research_sources_to_keep_tool(research_directory=research_folder)
print(result)

{'status': 'success', 'sources_selected_count': 12, 'selected_source_ids': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], 'sources_selected_path': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/perplexity_sources_selected.md', 'results_selected_path': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/perplexity_results_selected.md', 'message': '✅ Selected 12 source(s). IDs written to /Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/perplexity_sources_selected.md. Filtered results written to /Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/perplexity_results_selected.md.'}

async def select_research_sources_to_scrape_tool(research_directory: str, max_sources: int = 5) -> Dict[str, Any]:
    """
    Select up to max_sources priority research sources to scrape in full.
    
    Analyzes the filtered Perplexity results together with the article guidelines and
    the material already scraped from guideline URLs, then chooses up to max_sources diverse,
    authoritative sources whose full content will add most value. The chosen URLs are
    written (one per line) to urls_to_scrape_from_research.md.
    """
    # Gather context from the research folder
    guidelines_path = research_path / ARTICLE_GUIDELINE_FILE
    results_selected_path = nova_path / PERPLEXITY_RESULTS_SELECTED_FILE
    
    article_guidelines = read_file_safe(guidelines_path)
    accepted_sources_data = read_file_safe(results_selected_path)
    scraped_guideline_ctx = load_scraped_guideline_context(nova_path)
    
    # Use LLM to select top sources for scraping
    selected_urls, reasoning = await select_top_sources(
        article_guidelines, accepted_sources_data, scraped_guideline_ctx, max_sources
    )
    
    # Write selected URLs to file
    urls_out_path = nova_path / URLS_TO_SCRAPE_FROM_RESEARCH_FILE
    urls_out_path.write_text("\n".join(selected_urls) + "\n", encoding="utf-8")
    
    return {
        "status": "success",
        "sources_selected": selected_urls,
        "sources_selected_count": len(selected_urls),
        "output_path": str(urls_out_path.resolve()),
        "reasoning": reasoning,
        "message": f"Successfully selected {len(selected_urls)} sources for full scraping..."
    }

PROMPT_SELECT_TOP_SOURCES = """
You are assisting with research for an upcoming article.

Your task is to select the most relevant and trustworthy sources from the web search results.
You should consider:
1. **Relevance**: How well each source addresses the article guidelines
2. **Authority**: The credibility and reputation of the source
3. **Quality**: The depth and accuracy of the information provided
4. **Uniqueness**: Sources that provide unique insights not covered by the scraped guideline URLs

Please select the top {top_n} sources that would be most valuable for the article research.

Return your selection with the following structure:
- **selected_urls**: A list of the most valuable URLs to scrape in full, ordered by priority
- **reasoning**: A short explanation summarizing why these specific URLs were chosen
""".strip()

from research_agent_part_2.mcp_server.src.tools import select_research_sources_to_scrape_tool
 
# Test selecting sources to scrape
result = await select_research_sources_to_scrape_tool(
		research_directory=research_folder,
		max_sources=3
)
print("Selected sources:")
print(result)

Selected sources:
{'status': 'success', 'urls_selected_count': 3, 'selected_urls': ['https://www.elastic.co/what-is/large-language-models', 'https://arxiv.org/html/2412.01130v2', 'https://www.legitsecurity.com/aspm-knowledge-base/llm-security-risks'], 'selection_reasoning': "These sources were chosen for their direct relevance to the article's themes of understanding why agents need tools, the development of autonomous AI agents, and crucial security considerations. The Elastic.co source provides a comprehensive overview of LLM limitations that tool use addresses. The ArXiv paper offers authoritative insights into the role of function calling in autonomous agent development and action models. The Legit Security source details critical security risks and mitigation strategies associated with external tool-calling, which is a vital aspect not covered in the existing guideline URLs.", 'urls_output_path': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/urls_to_scrape_from_research.md', 'message': "✅ Saved 3 URL(s) to scrape to /Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/urls_to_scrape_from_research.md.\nReasoning: These sources were chosen for their direct relevance to the article's themes of understanding why agents need tools, the development of autonomous AI agents, and crucial security considerations. The Elastic.co source provides a comprehensive overview of LLM limitations that tool use addresses. The ArXiv paper offers authoritative insights into the role of function calling in autonomous agent development and action models. The Legit Security source details critical security risks and mitigation strategies associated with external tool-calling, which is a vital aspect not covered in the existing guideline URLs."}

from research_agent_part_2.mcp_server.src.tools import scrape_research_urls_tool
 
# Test scraping the selected research URLs
result = await scrape_research_urls_tool(
		research_directory=research_folder,
		concurrency_limit=3
)
print("Scraping results:")
print(result)

Scraping results:
{'status': 'success', 'urls_processed': 3, 'urls_total': 3, 'original_urls_count': 3, 'deduplicated_count': 0, 'files_saved': 3, 'output_directory': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/.nova/urls_from_research', 'saved_files': ['understanding-large-language-models-a-comprehensive-guide-el.md', 'enhancing-function-calling-capabilities-in-llms-strategies-f.md', 'large-language-model-llm-security-risks-and-best-practices.md'], 'message': "Processed 3 new URLs from urls_to_scrape_from_research.md in '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder'. Scraped 3/3 web pages."}

def create_research_file_tool(research_directory: str) -> Dict[str, Any]:
    """
    Generate comprehensive research.md file from all research data.
 
    Combines all research data including filtered Perplexity results, scraped guideline
    sources, and full research sources into a comprehensive research.md file. The file
    is organized into sections with collapsible blocks for easy navigation.
    """
		# Convert to Path object
    article_dir = Path(research_directory)
    nova_dir = article_dir / NOVA_FOLDER
 
		# Collect all research data
    perplexity_results = read_file_safe(nova_dir / PERPLEXITY_RESULTS_SELECTED_FILE)
 
		# Collect scraped sources from guidelines
    scraped_sources = collect_directory_markdowns_with_titles(nova_dir / URLS_FROM_GUIDELINES_FOLDER)
    code_sources = collect_directory_markdowns_with_titles(nova_dir / URLS_FROM_GUIDELINES_CODE_FOLDER)
    youtube_transcripts = collect_directory_markdowns_with_titles(nova_dir / URLS_FROM_GUIDELINES_YOUTUBE_FOLDER)
 
		# Collect full research sources
    additional_sources = collect_directory_markdowns_with_titles(nova_dir / URLS_FROM_RESEARCH_FOLDER)
 
		# Build comprehensive research sections
    research_results_section = build_research_results_section(perplexity_results)
    scraped_sources_section = build_sources_section("Scraped Sources from Guidelines", scraped_sources)
    code_sources_section = build_sources_section("Code Sources from Guidelines", code_sources)
    youtube_section = build_sources_section("YouTube Transcripts from Guidelines", youtube_transcripts)
    additional_section = build_sources_section("Additional Research Sources", additional_sources)
 
		# Combine all sections into final research file
    research_content = combine_research_sections([
        research_results_section,
        scraped_sources_section,
        code_sources_section,
        youtube_section,
        additional_section
    ])
 
		# Write final research file
    research_file_path = article_dir / RESEARCH_MD_FILE
    research_file_path.write_text(research_content, encoding="utf-8")
 
    return {
        "status": "success",
        "markdown_file": str(research_file_path.resolve()),
        "research_results_count": len(extract_perplexity_chunks(perplexity_results)),
        "scraped_sources_count": len(scraped_sources),
        "code_sources_count": len(code_sources),
        "youtube_transcripts_count": len(youtube_transcripts),
        "additional_sources_count": len(additional_sources),
        "message": f"Successfully created comprehensive research file: {research_file_path.name}"
    }

## Research Results from Web Search
Query: [Original Query]

### Source [1]: [URL]
[Content from source]

### Source [2]: [URL]
[Content from source]


## Scraped Sources from Guidelines
Source: [Filename]
[Full scraped content]


## Code Sources from Guidelines
Repository: [Repository Name]
[Repository analysis and code content]


## YouTube Transcripts from Guidelines
Video: [Video Title]
[Full video transcript]


## Additional Research Sources
Source: [URL]
[Full scraped research content]

from research_agent_part_2.mcp_server.src.tools import create_research_file_tool
 
# Test creating the final research file
result = create_research_file_tool(research_directory=research_folder)
print("Research file creation results:")
print(result)
 
# Read and display a sample of the generated research file
with open(result["markdown_file"], "r") as f:
    content = f.read()
    print("\nFirst 1000 characters of the research file:")
    print(content[:1000] + "...")

Research file creation results:
{'status': 'success', 'markdown_file': '/Users/fabio/Desktop/course-ai-agents/lessons/research_agent_part_2/data/sample_research_folder/research.md', 'research_results_count': 3, 'scraped_sources_count': 6, 'code_sources_count': 2, 'youtube_transcripts_count': 1, 'additional_sources_count': 2, 'message': '✅ Generated research markdown file:\n  - research.md'}

First 1000 characters of the research file:
# Research

## Research Results


What are the fundamental limitations of large language models that tool use and function calling aim to solve?

### Source [1]: https://hatchworks.com/blog/gen-ai/large-language-models-guide/

Query: What are the fundamental limitations of large language models that tool use and function calling aim to solve?

Answer: Large Language Models (LLMs) have several fundamental technical limitations that tool use and function calling aim to solve:

- **Domain Mismatch**: LLMs trained on general datasets struggle with providing accurate or detailed responses in specialized or niche domains, leading to generic or incorrect outputs when specific expertise is required.

- **Word Prediction Issues**: LLMs often fail with less common words or phrases, affecting their ability to generate or translate technical or domain-specific content accurately.

- **Hallucinations**: LLMs sometimes produce highly original but fabricated information. F...

from research_agent_part_2.mcp_client.src.client import main as client_main
import sys
 
async def run_client():
    _argv_backup = sys.argv[:]
    sys.argv = ["client"]
    try:
        await client_main()
    finally:
        sys.argv = _argv_backup
 
# Start client with in-memory server
await run_client()

🤔 LLM's Thoughts:
Let's Get This Research Started!

Okay, so I'm ready to roll. My first step is to lay out this research workflow – gotta make sure we're on the same page and, critically, that I have the right starting point. I need to explain the steps, and then I'll politely ask for that all-important `research_directory`. I'm also going to check if there are any tweaks or modifications you'd like to make to the process.

Once that's clear, I'll kick things off by calling `extract_guidelines_urls` to grab the URLs and local files listed in `ARTICLE_GUIDELINE_FILE`. That gives me a solid initial pool to work with.

Then, the heavy lifting begins. I'll launch several parallel processes to handle different types of resources efficiently. I'll call `process_local_files` for local content, `scrape_and_clean_other_urls` for general web pages, `process_github_urls` for those GitHub repositories (always a good source!), and `transcribe_youtube_urls` for any relevant YouTube videos. Parallel processing is key here for speed.

Next comes the core research loop. We'll run this three times, each time generating new search queries with `generate_next_queries`. Those queries will then be fed into `run_perplexity_research` to pull in the initial research results.

Now, we filter. I'll use `select_research_sources_to_keep` to identify and keep only the highest-quality sources from the Perplexity results. This is where expertise comes into play – I need to make sure we're focusing on the good stuff.

Once the good stuff is identified, I'll identify sources for a complete scrape using `select_research_sources_to_scrape`. Then I'll call `scrape_research_urls` to scrape the selected sources.

Finally, I'll synthesize everything. I'll call `create_research_file` to compile all the research, cleaned content, scraped pages, and initial sources into a comprehensive `research.md` file.

Now, a critical point: I need to be prepared for potential hiccups. If *any* of the tools run into trouble and successfully process zero items, I'm going to halt the entire process. I'll state which tool failed and the associated error message, and then I'll ask for your guidance on how to proceed. Better safe than sorry – a failed start could compromise the integrity of the entire process!

💬 LLM Response:  Hello! I will help you execute the Nova research agent workflow. Here are the steps involved:

1.  **Setup**: I will begin by extracting URLs and local file references from your `ARTICLE_GUIDELINE_FILE`.
2.  **Process Extracted Resources**: I will then process these extracted resources in parallel:
    *   Copying referenced local files to a dedicated folder.
    *   Scraping and cleaning content from other web URLs.
    *   Processing GitHub URLs using gitinge...
...

{
  "mcpServers": {
    "research-agent": {
      "transport": "stdio",
      "command": "uv",
      "args": [
        "--directory",
        "/absolute/path/to/nova/mcp_server",
        "run",
        "-m",
        "src.server",
        "--transport",
        "stdio"
      ]
    }
  }
}