LLM Providers Guide¶

The anchor.llm module provides a unified interface for multiple LLM providers through the LLMProvider protocol. Provider-specific SDKs are optional dependencies -- install only the ones you need.

Overview¶

The LLM provider system is built around three layers:

1. LLMProvider protocol -- defines the standard interface (invoke, stream, ainvoke, astream)
2. BaseLLMProvider -- abstract base class with built-in retries, error mapping, and fallback support
3. Concrete providers -- thin adapters for each SDK (Anthropic, OpenAI, Gemini, etc.)

Model string "openai/gpt-4o"
    |
    v
create_provider()  -->  resolves prefix  -->  OpenAIProvider
    |
    v
Agent or direct usage
    |
    v
invoke / stream / ainvoke / astream

Quick Start¶

from anchor import Agent

# Anthropic (default when no prefix)
agent = Agent(model="claude-haiku-4-5-20251001")

# OpenAI
agent = Agent(model="openai/gpt-4o")

# Google Gemini
agent = Agent(model="gemini/gemini-2.0-flash")

# Local Ollama
agent = Agent(model="ollama/llama3")

# Claude Code CLI — your subscription, no API key
agent = Agent(model="claude_cli/sonnet")

Installation¶

Each provider is packaged as an optional extra:

pip install astro-anchor[anthropic]   # Anthropic
pip install astro-anchor[claude-cli]   # Claude Code CLI (no API key)
pip install astro-anchor[openai]      # OpenAI, Grok, OpenRouter
pip install astro-anchor[gemini]      # Google Gemini
pip install astro-anchor[ollama]      # Ollama (local models)
pip install astro-anchor[litellm]     # LiteLLM (catch-all)

Install multiple extras at once:

pip install astro-anchor[anthropic,openai,gemini]

Supported Providers¶

Claude Code CLI (no API key)¶

claude_cli/ runs Anchor on the claude CLI's own credentials — a Pro/Max subscription (OAuth) or a token from claude setup-token. No ANTHROPIC_API_KEY anywhere.

pip install astro-anchor[claude-cli]
claude auth login      # or: claude setup-token

agent = Agent(model="claude_cli/sonnet")   # aliases or full model ids

Tool calls still come back to you¶

The CLI is an agent: it runs its own loop and executes its own tools. Anchor keeps the loop anyway. Your tools are registered as an in-process MCP server and a PreToolUse hook answers defer, which stops the run without executing and reports the call back. Agent.with_tools(...) works exactly as it does on any other provider — your Python function runs in your process.

Closing the loop needs no CLI session: once the history carries a result for a call, the hook lets a replayed call through and the MCP handler returns that stored result.

Agentic mode¶

Set builtin_tools to also let the CLI's own tools run inside the run:

from anchor.llm.providers.claude_cli import ClaudeCLIProvider

provider = ClaudeCLIProvider(
    model="sonnet",
    builtin_tools=["Read", "Grep"],   # or "claude_code" for the full preset
    cwd="/path/to/project",
)
agent = Agent(llm=provider)

What differs from an API-key provider¶

Anchor loads none of your CLI configuration: setting_sources=[] and strict_mcp_config=True are set for you. Without them the CLI's default system prompt, your CLAUDE.md and your MCP servers ride along — measured at ~178k tokens of overhead per call instead of a few hundred.

Model String Format¶

Model strings follow the pattern "provider/model-name". The prefix is split on the first / only, so model names containing slashes (e.g. openrouter/meta-llama/llama-3-70b) work correctly.

# Explicit provider prefix
agent = Agent(model="openai/gpt-4o")

# No prefix defaults to Anthropic (backward compatible)
agent = Agent(model="claude-haiku-4-5-20251001")
# Equivalent to:
agent = Agent(model="anthropic/claude-haiku-4-5-20251001")

Fallback Chains¶

Configure fallback providers to handle transient failures gracefully:

# Primary: Anthropic, fallback: OpenAI
agent = Agent(
    model="anthropic/claude-sonnet-4-20250514",
    fallbacks=["openai/gpt-4o"],
)

Fallback behavior depends on the call method:

Injecting a Pre-Built Provider¶

For advanced configuration (custom base URLs, headers, or timeouts), create a provider instance directly and pass it to the Agent:

from anchor.llm import create_provider

provider = create_provider("openai/gpt-4o", api_key="sk-...")
agent = Agent(llm=provider)

Direct Provider Usage¶

Providers implement the LLMProvider protocol and can be used independently of the Agent:

from anchor.llm import create_provider, Message, Role

llm = create_provider("openai/gpt-4o")
response = llm.invoke([Message(role=Role.USER, content="Hello!")])
print(response.content)

All four call methods are available:

# Synchronous
response = llm.invoke(messages)

# Synchronous streaming
for chunk in llm.stream(messages):
    print(chunk.content, end="")

# Async
response = await llm.ainvoke(messages)

# Async streaming
async for chunk in llm.astream(messages):
    print(chunk.content, end="")

Body Parameters and Billed Cost¶

extra_body forwards JSON body parameters the SDK has no field for — OpenRouter's provider routing and reasoning, vendor extensions. Set it once on the provider (every request) or per call; the two are merged key by key and the call wins.

from anchor.llm import create_provider

llm = create_provider(
    "openrouter/google/gemini-2.5-flash",
    extra_body={"provider": {"order": ["Google AI Studio"], "allow_fallbacks": False}},
)
llm.invoke(messages, extra_body={"reasoning": {"effort": "low"}})

Applies to every OpenAI-compatible provider (openai, grok, openrouter, ollama, litellm; the option lives on the base provider). Streaming requests the final usage-only chunk (stream_options.include_usage), so RoundUsage carries the provider's real token counts instead of tokenizer estimates, and prompt_tokens_details.cached_tokens becomes Usage.cache_read_tokens. OpenRouterProvider also asks for usage: {"include": true} by default: the billed usage.cost lands in Usage.total_cost, which prices the round (RoundUsage.cost_usd) and feeds UsageLimits.cost_limit.

Custom Providers¶

Implement BaseLLMProvider to add support for a new backend:

from anchor.llm import BaseLLMProvider, register_provider

class MyProvider(BaseLLMProvider):
    provider_name = "my_provider"

    def _resolve_api_key(self): ...
    def _do_invoke(self, messages, tools, **kwargs): ...
    def _do_stream(self, messages, tools, **kwargs): ...
    async def _do_ainvoke(self, messages, tools, **kwargs): ...
    async def _do_astream(self, messages, tools, **kwargs): ...

register_provider("my_provider", MyProvider)

Once registered, the prefix routes automatically:

agent = Agent(model="my_provider/my-model")

Error Handling¶

All provider errors are normalized into a common hierarchy:

from anchor.llm import ProviderError, RateLimitError

try:
    response = llm.invoke(messages)
except RateLimitError as e:
    print(f"Rate limited by {e.provider}, retry after {e.retry_after}s")
except ProviderError as e:
    print(f"Error from {e.provider}: {e}")

BaseLLMProvider includes built-in retry with exponential backoff for transient errors (rate limits, connection timeouts, server errors). Non-transient errors (authentication failures, invalid requests) propagate immediately.

See Also¶

• LLM API Reference -- complete API signatures and classes
• Agent Guide -- using providers through the Agent
• Protocols Reference -- LLMProvider protocol definition