Skip to main content

​
Overview

LiteLLM integrates with multiple secret management systems to securely store and retrieve API keys and sensitive configuration. This eliminates hardcoded credentials and enables centralized secret management.

​
Supported Secret Managers

  • AWS Secrets Manager: Full-featured AWS secret storage
  • Azure Key Vault: Microsoft Azure secret management
  • Google Cloud Secret Manager: GCP secret storage
  • HashiCorp Vault: Enterprise secret management
  • CyberArk: Enterprise privileged access management
  • Google KMS: Encryption key management
  • AWS KMS: AWS encryption service
  • Custom: Build your own integration

​
Quick Start

1

Choose Your Secret Manager

Install required dependencies:
# AWS Secrets Manager
pip install boto3>=1.28.57

# Azure Key Vault
pip install azure-identity azure-keyvault-secrets

# Google Secret Manager
pip install google-cloud-secret-manager

# HashiCorp Vault
pip install hvac
2

Configure Environment

Set up authentication:
# AWS
export AWS_REGION_NAME="us-east-1"
export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"

# Azure
export AZURE_KEY_VAULT_URI="https://your-vault.vault.azure.net/"

# Google Cloud
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"

# HashiCorp Vault
export VAULT_ADDR="https://vault.example.com"
export VAULT_TOKEN="your-vault-token"
3

Initialize Secret Manager

Configure LiteLLM to use your secret manager:
from litellm.secret_managers import AWSSecretsManagerV2
import litellm

# Initialize AWS Secrets Manager
litellm.secret_manager_client = AWSSecretsManagerV2(
    aws_region_name="us-east-1"
)

litellm._key_management_system = "aws_secret_manager"
4

Use Secrets in API Calls

Reference secrets by name:
# Secret name format: os.environ/SECRET_NAME
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/OPENAI_API_KEY",  # Fetched from secret manager
    messages=[{"role": "user", "content": "Hello!"}]
)

​
AWS Secrets Manager

​
Setup

from litellm.secret_managers.aws_secret_manager_v2 import AWSSecretsManagerV2
import litellm

# Initialize
litellm.secret_manager_client = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_role_name="optional-role-arn",  # For cross-account access
    aws_session_name="litellm-session"
)

litellm._key_management_system = "aws_secret_manager"

​
Read Secrets

from litellm.secret_managers.aws_secret_manager_v2 import AWSSecretsManagerV2

sm = AWSSecretsManagerV2(aws_region_name="us-east-1")

# Read single secret
api_key = await sm.async_read_secret(
    secret_name="openai-api-key"
)

print(api_key)

​
Write Secrets

# Create new secret
response = await sm.async_write_secret(
    secret_name="new-api-key",
    secret_value="sk-1234567890",
    description="OpenAI API key for production",
    tags={"Environment": "Production", "Team": "ML"}
)

print(f"Secret ARN: {response['ARN']}")

​
Rotate Secrets

# Rotate by creating new version and deleting old
response = await sm.async_rotate_secret(
    current_secret_name="old-api-key",
    new_secret_name="new-api-key",
    new_secret_value="sk-new-value"
)

# Old secret scheduled for deletion (7 day recovery window)

​
Delete Secrets

# Schedule deletion with recovery window
response = await sm.async_delete_secret(
    secret_name="unused-key",
    recovery_window_in_days=30  # Can be recovered within 30 days
)

print(f"Deletion date: {response['DeletionDate']}")

​
Azure Key Vault

​
Setup

from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
import litellm
import os

# Initialize Azure Key Vault client
kv_uri = os.environ["AZURE_KEY_VAULT_URI"]
credential = DefaultAzureCredential()

litellm.secret_manager_client = SecretClient(
    vault_url=kv_uri,
    credential=credential
)

litellm._key_management_system = "azure_key_vault"

​
Use with LiteLLM

# Secrets automatically fetched from Key Vault
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/openai-api-key",  # Retrieved from Azure Key Vault
    messages=[{"role": "user", "content": "Hello!"}]
)

​
Google Cloud Secret Manager

​
Setup

from litellm.secret_managers.google_secret_manager import GoogleSecretManager
import litellm

# Initialize
litellm.secret_manager_client = GoogleSecretManager(
    project_id="your-gcp-project"
)

litellm._key_management_system = "google_secret_manager"

​
Operations

sm = GoogleSecretManager(project_id="your-project")

# Read secret (latest version)
api_key = sm.get_secret_from_google_secret_manager(
    secret_name="openai-api-key"
)

# Read specific version
api_key = sm.get_secret_from_google_secret_manager(
    secret_name="openai-api-key",
    version_id="3"
)

​
HashiCorp Vault

​
Setup

from litellm.secret_managers.hashicorp_secret_manager import HashiCorpVault
import litellm
import os

# Initialize
vault = HashiCorpVault(
    vault_url=os.environ["VAULT_ADDR"],
    vault_token=os.environ["VAULT_TOKEN"],
    vault_namespace="litellm",  # Optional
    mount_point="secret"  # Default KV mount point
)

litellm.secret_manager_client = vault
litellm._key_management_system = "hashicorp_vault"

​
Operations

# Async read
api_key = await vault.async_read_secret(
    secret_name="openai/api-key"  # Path in Vault
)

# Sync read
api_key = vault.sync_read_secret(
    secret_name="openai/api-key"
)

​
CyberArk

​
Setup

from litellm.secret_managers.cyberark_secret_manager import CyberArkSecretManager
import litellm

# Initialize
cyberark = CyberArkSecretManager(
    cyberark_url="https://cyberark.company.com",
    cyberark_app_id="litellm-app",
    cyberark_safe="litellm-safe",
    cyberark_cert_path="/path/to/client-cert.pem"
)

litellm.secret_manager_client = cyberark
litellm._key_management_system = "cyberark"

​
Operations

# Read secret
api_key = await cyberark.async_read_secret(
    secret_name="openai-api-key",
    optional_params={
        "folder": "Root",
        "object": "OpenAI-Key"
    }
)

​
Custom Secret Manager

​
Implement Base Class

from litellm.secret_managers.base_secret_manager import BaseSecretManager
from typing import Optional, Dict, Any, Union
import httpx

class MySecretManager(BaseSecretManager):
    def __init__(self, api_url: str, api_token: str):
        super().__init__()
        self.api_url = api_url
        self.api_token = api_token
    
    async def async_read_secret(
        self,
        secret_name: str,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> Optional[str]:
        """
        Fetch secret from your backend.
        
        Returns:
            Secret value as string, or None if not found
        """
        async with httpx.AsyncClient() as client:
            response = await client.get(
                f"{self.api_url}/secrets/{secret_name}",
                headers={"Authorization": f"Bearer {self.api_token}"},
                timeout=timeout or 30.0
            )
            
            if response.status_code == 404:
                return None
            
            response.raise_for_status()
            data = response.json()
            return data["value"]
    
    def sync_read_secret(
        self,
        secret_name: str,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> Optional[str]:
        """
        Synchronous version of async_read_secret.
        """
        response = httpx.get(
            f"{self.api_url}/secrets/{secret_name}",
            headers={"Authorization": f"Bearer {self.api_token}"},
            timeout=timeout or 30.0
        )
        
        if response.status_code == 404:
            return None
        
        response.raise_for_status()
        data = response.json()
        return data["value"]
    
    async def async_write_secret(
        self,
        secret_name: str,
        secret_value: str,
        description: Optional[str] = None,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
        tags: Optional[Union[dict, list]] = None
    ) -> Dict[str, Any]:
        """
        Write a secret to your backend.
        """
        async with httpx.AsyncClient() as client:
            response = await client.post(
                f"{self.api_url}/secrets",
                json={
                    "name": secret_name,
                    "value": secret_value,
                    "description": description,
                    "tags": tags
                },
                headers={"Authorization": f"Bearer {self.api_token}"},
                timeout=timeout or 30.0
            )
            
            response.raise_for_status()
            return response.json()
    
    async def async_delete_secret(
        self,
        secret_name: str,
        recovery_window_in_days: Optional[int] = 7,
        optional_params: Optional[dict] = None,
        timeout: Optional[Union[float, httpx.Timeout]] = None,
    ) -> dict:
        """
        Delete a secret from your backend.
        """
        async with httpx.AsyncClient() as client:
            response = await client.delete(
                f"{self.api_url}/secrets/{secret_name}",
                headers={"Authorization": f"Bearer {self.api_token}"},
                params={"recovery_days": recovery_window_in_days},
                timeout=timeout or 30.0
            )
            
            response.raise_for_status()
            return response.json()

# Register with LiteLLM
import litellm

litellm.secret_manager_client = MySecretManager(
    api_url="https://secrets.company.com",
    api_token="your-token"
)

litellm._key_management_system = "custom"

​
Use Custom Secret Manager

# Secrets fetched from your backend
response = await litellm.acompletion(
    model="gpt-4",
    api_key="os.environ/openai-key",
    messages=[{"role": "user", "content": "Hello!"}]
)

​
Using with LiteLLM Proxy

​
Configuration

config.yaml
general_settings:
  use_aws_secret_manager: true
  key_management_system: "aws_secret_manager"

key_management_settings:
  aws_region_name: us-east-1
  aws_role_name: arn:aws:iam::123456789:role/LiteLLMRole

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/OPENAI_API_KEY  # From AWS Secrets Manager
config.yaml
environment_variables:
  AZURE_KEY_VAULT_URI: https://your-vault.vault.azure.net/

general_settings:
  key_management_system: azure_key_vault

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai-api-key
config.yaml
environment_variables:
  GOOGLE_APPLICATION_CREDENTIALS: /path/to/credentials.json

general_settings:
  key_management_system: google_secret_manager
  google_kms_project_id: your-gcp-project

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai-api-key
config.yaml
environment_variables:
  VAULT_ADDR: https://vault.company.com
  VAULT_TOKEN: your-vault-token

general_settings:
  key_management_system: hashicorp_vault

model_list:
  - model_name: gpt-4
    litellm_params:
      model: gpt-4
      api_key: os.environ/openai/api-key

​
Best Practices

For AWS, prefer IAM roles over access keys:
# Good: IAM role-based authentication
sm = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_role_name="arn:aws:iam::123456789:role/LiteLLMRole"
)

# Bad: Hardcoded credentials
sm = AWSSecretsManagerV2(
    aws_region_name="us-east-1",
    aws_access_key_id="AKIAIOSFODNN7EXAMPLE",
    aws_secret_access_key="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)
Regularly rotate secrets:
import asyncio
from datetime import datetime, timedelta

async def rotate_if_needed(sm, secret_name, max_age_days=90):
    # Check secret age
    metadata = await sm.get_secret_metadata(secret_name)
    created = metadata["CreatedDate"]
    
    if datetime.now() - created > timedelta(days=max_age_days):
        # Generate new key (implement your logic)
        new_value = generate_new_api_key()
        
        # Rotate
        await sm.async_rotate_secret(
            current_secret_name=secret_name,
            new_secret_name=f"{secret_name}-v2",
            new_secret_value=new_value
        )
        
        print(f"Rotated {secret_name}")
Reduce API calls with caching:
from functools import lru_cache
from datetime import datetime, timedelta

class CachedSecretManager:
    def __init__(self, sm):
        self.sm = sm
        self.cache = {}
        self.cache_ttl = timedelta(minutes=5)
    
    async def get_secret(self, secret_name):
        # Check cache
        if secret_name in self.cache:
            value, timestamp = self.cache[secret_name]
            if datetime.now() - timestamp < self.cache_ttl:
                return value
        
        # Fetch from secret manager
        value = await self.sm.async_read_secret(secret_name)
        
        # Update cache
        self.cache[secret_name] = (value, datetime.now())
        
        return value
Gracefully handle missing secrets:
async def get_secret_with_fallback(sm, secret_name, fallback=None):
    try:
        value = await sm.async_read_secret(secret_name)
        
        if value is None:
            if fallback:
                return fallback
            raise ValueError(f"Secret not found: {secret_name}")
        
        return value
    except Exception as e:
        print(f"Error fetching secret: {e}")
        if fallback:
            return fallback
        raise
Log all secret access for security auditing:
import logging

logger = logging.getLogger("secret_access")

class AuditedSecretManager:
    def __init__(self, sm):
        self.sm = sm
    
    async def async_read_secret(self, secret_name, **kwargs):
        logger.info(f"Accessing secret: {secret_name}")
        
        try:
            value = await self.sm.async_read_secret(secret_name, **kwargs)
            logger.info(f"Successfully retrieved: {secret_name}")
            return value
        except Exception as e:
            logger.error(f"Failed to retrieve {secret_name}: {e}")
            raise

​
Reference

​
Source Code

  • Base secret manager: litellm/secret_managers/base_secret_manager.py:9
  • AWS Secrets Manager: litellm/secret_managers/aws_secret_manager_v2.py:36
  • Secret manager handler: litellm/secret_managers/secret_manager_handler.py:24

​
Environment Variables

# AWS
AWS_REGION_NAME=us-east-1
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

# Azure
AZURE_KEY_VAULT_URI=https://vault.azure.net/
AZURE_TENANT_ID=your-tenant-id
AZURE_CLIENT_ID=your-client-id
AZURE_CLIENT_SECRET=your-secret

# Google Cloud
GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json

# HashiCorp Vault
VAULT_ADDR=https://vault.example.com
VAULT_TOKEN=your-token
VAULT_NAMESPACE=litellm

Build docs developers (and LLMs) love

Get started for freeTalk to us