Connect to any LLMs

CrewAI can talk to many LLM providers (cloud and local) through one unified LLM class. You choose the model you want, configure its keys once, and then plug that LLM into agents, crews, or flows.

1. LLM Basics for Agentic Architects

LLMs are the “brain” of your agents: they read context, decide what to do, and generate text.
Context window = how much text fits in one call (e.g., 8K vs 128K vs 1M tokens).
Temperature (0.0–1.0) controls randomness: low for deterministic tools, higher for creative writing.
Provider = where the model runs (OpenAI, Anthropic, Google, local Ollama, etc.).

As an architect, you pick which brain and how it should think (temperature, max tokens, etc.).

2. The Core LLM Class

Most providers follow this basic pattern:

# python
from crewai import LLM

llm = LLM(
    model="openai/gpt-4o",   # or "anthropic/claude-3-5-sonnet-20241022", etc.
    api_key="your-api-key",  # or via env vars
    temperature=0.7,
)

You can then:

Pass llm= into an Agent, Crew, or Flow.
Or call it directly:

response = llm.call("Explain retrieval-augmented generation.")

CrewAI also supports:

stream=True for streaming tokens.
await llm.acall(...) for async calls.
response_format=MyPydanticModel for structured outputs.

3. Fastest Setup: Use MODEL= in .env

For quick scaffolds (especially with crewai create):

MODEL=openai/gpt-4o        # or gemini-2.0-flash, claude-3-5-sonnet-20241022, ...
OPENAI_API_KEY=sk-...

CrewAI will default to that model whenever you don’t pass a custom LLM.
For production, it’s still good practice to be explicit with LLM(...) on critical agents.

4. Popular Cloud Providers (Cheat Sheet)

Below is a didactical “menu” of the most common providers and how to wire each one.

4.1 OpenAI (GPT‑4.x, GPT‑4o, o‑series)

Env vars:

OPENAI_API_KEY=sk-...
# Optional:
OPENAI_BASE_URL=<custom-base-url>

Basic LLM:

llm = LLM(
    model="openai/gpt-4o",
    api_key="your-api-key",     # or via env var
    temperature=0.7,
    max_tokens=4000,
)

Advanced (Responses API, reasoning, streaming, etc.) are also supported (e.g., api="responses", reasoning_effort="medium").

When to use: general‑purpose, best‑in‑class, strong tools + structured outputs.

4.2 Anthropic (Claude 3 / 4 family)

Env vars:

ANTHROPIC_API_KEY=sk-ant-...

Basic:

llm = LLM(
    model="anthropic/claude-3-5-sonnet-20241022",
    api_key="your-api-key",
    max_tokens=4096,   # required by Anthropic
)

Extended Thinking (for Claude Sonnet 4+):

llm = LLM(
    model="anthropic/claude-sonnet-4",
    thinking={"type": "enabled", "budget_tokens": 5000},
    max_tokens=10000,
)

When to use: deep reasoning, agentic workflows, and long‑context analysis.

4.3 Google Gemini (Gemini API + Vertex)

Env vars (Gemini API):

GOOGLE_API_KEY=<your-api-key>   # or GEMINI_API_KEY

Basic Gemini API usage:

llm = LLM(
    model="gemini/gemini-2.0-flash",
    api_key="your-api-key",
    temperature=0.7,
)

Vertex AI Express mode (API key):

GOOGLE_GENAI_USE_VERTEXAI=true
GOOGLE_API_KEY=<your-api-key>

llm = LLM(model="gemini/gemini-2.0-flash", temperature=0.7)

Full Vertex with service account uses project, location, and vertex_credentials JSON.

When to use: multimodal work, Google Cloud integration, very large context windows.

4.4 Azure OpenAI / Azure AI Inference

Env vars:

AZURE_API_KEY=<your-api-key>
AZURE_ENDPOINT=<your-endpoint-url>
AZURE_API_VERSION=2024-06-01

Code:

llm = LLM(
    model="azure/gpt-4o",
    api_key="your-api-key",     # or env
    endpoint="https://your-resource.openai.azure.com/openai/deployments/your-deployment",
    api_version="2024-06-01",
    temperature=0.7,
)

When to use: enterprise compliance, VNet, data residency.

4.5 AWS Bedrock

Env vars:

AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
AWS_DEFAULT_REGION=us-east-1

Code:

llm = LLM(
    model="bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0",
    region_name="us-east-1",
    temperature=0.7,
)

Supports Anthropic, Llama, Nova, Mistral, etc. with Guardrails and model‑specific params.

4.6 Other major providers (same pattern)

All below use the same idea: set an API key env var, then call LLM(model="provider/model-id", ...).

Groq – GROQ_API_KEY, models like "groq/llama-3.2-90b-text-preview".
Nvidia NIM (cloud) – NVIDIA_API_KEY, models under nvidia_nim/....
IBM watsonx.ai – WATSONX_URL, WATSONX_APIKEY, WATSONX_PROJECT_ID.
Hugging Face Hub – HF_TOKEN, model="huggingface/meta-llama/Meta-Llama-3.1-8B-Instruct".
Perplexity, Fireworks, SambaNova, Cerebras, OpenRouter, Nebius, etc.

The docs list model IDs, context windows, and best‑for columns to help you choose.

5. Local and Self‑Hosted Models

For privacy, latency, or cost reasons you can also run models yourself.

5.1 Ollama (local LLMs)

Steps:

Install Ollama and pull a model:

ollama run llama3

Point CrewAI at it:

from crewai import LLM

llm = LLM(
    model="ollama/llama3:70b",
    base_url="http://localhost:11434",
)

5.2 Local Nvidia NIM (OpenAI‑compatible)

If you expose NIM as an OpenAI‑compatible endpoint:

from crewai.llm import LLM

local_nvidia_nim_llm = LLM(
    model="openai/meta/llama-3.1-8b-instruct",
    base_url="http://localhost:8000/v1",
    api_key="anything",   # required but can be dummy if server ignores it
)

Then reuse this llm in your agents (e.g., via @CrewBase).

6. Streaming, Async, and Structured Responses

6.1 Streaming

To stream tokens as they’re generated:

llm = LLM(
    model="openai/gpt-4o",
    stream=True,
)

Useful for UIs and long‑running tasks (and is also supported at the crew/flow level).

6.2 Async calls

For concurrent LLM calls

import asyncio
from crewai import LLM

async def main():
    llm = LLM(model="openai/gpt-4o")
    response = await llm.acall("What is the capital of France?")
    print(response)

asyncio.run(main())

acall accepts the same parameters as call (messages, tools, callbacks, etc.).

6.3 Structured outputs (Pydantic)

You can ask the LLM to return typed objects instead of raw JSON/text:

from pydantic import BaseModel
from crewai import LLM

class Dog(BaseModel):
    name: str
    age: int
    breed: str

llm = LLM(model="openai/gpt-4o", response_format=Dog)

response = llm.call(
    "Meet Kona! She is 3 years old and is a black german shepherd. "
    "Return name, age, and breed."
)

print(response)  # Dog(name='Kona', age=3, breed='black german shepherd')

This is powerful for building reliable agent outputs (e.g., plans, tool configs, JSON APIs).

7. Design Tips for “Connect Any LLM” Lessons

When architecting your solution, consider these patterns:

Separate what the agent does (role, tools, knowledge) from which brain it uses (LLM(model=...)).
Start learners on one provider (e.g., openai/gpt-4o), then swap to Groq, Gemini, or Ollama by changing only:
- env variables and
- the model= string.
Use:
- Low temperature for tools, planning, evaluations.
- Higher temperature for ideation, copy, creative work.
Show streaming + async for UX and throughput, and structured outputs for robust integrations.

This framing makes “Connect to Any LLM” a matter of configuration, not code rewrites, which is exactly the mindset you want for Agentic Architects.

Connect to any LLMs

On this page