diff --git a/README.md b/README.md index 96bca25f3b..7827ed2b28 100644 --- a/README.md +++ b/README.md @@ -303,51 +303,107 @@ curl 'http://0.0.0.0:4000/key/generate' \ ## Supported Providers ([Docs](https://docs.litellm.ai/docs/providers)) -| Provider | `/chat/completions` | `/embeddings` | `/image/generations` | -|-------------------------------------------------------------------------------------|---------------------|---------------|----------------------| -| [OpenAI](https://docs.litellm.ai/docs/providers/openai) | ✅ | ✅ | ✅ | -| [Meta - Llama API](https://docs.litellm.ai/docs/providers/meta_llama) | ✅ | | | -| [Azure](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | -| [AI/ML API](https://docs.litellm.ai/docs/providers/aiml) | ✅ | ✅ | ✅ | -| [AWS - Sagemaker](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | | -| [AWS - Bedrock](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | | -| [Google - Vertex AI](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | -| [Google - PaLM](https://docs.litellm.ai/docs/providers/palm) | ✅ | | | -| [Google AI Studio - Gemini](https://docs.litellm.ai/docs/providers/gemini) | ✅ | | | -| [Mistral AI API](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | | -| [Cloudflare AI Workers](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | | | -| [CompactifAI](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | | | -| [Cohere](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | | -| [Anthropic](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | | | -| [Empower](https://docs.litellm.ai/docs/providers/empower) | ✅ | | | -| [Huggingface](https://docs.litellm.ai/docs/providers/huggingface) | ✅ | ✅ | | -| [Replicate](https://docs.litellm.ai/docs/providers/replicate) | ✅ | | | -| [Together AI](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | | | -| [OpenRouter](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | | | -| [AI21](https://docs.litellm.ai/docs/providers/ai21) | ✅ | | | -| [Baseten](https://docs.litellm.ai/docs/providers/baseten) | ✅ | | | -| [VLLM](https://docs.litellm.ai/docs/providers/vllm) | ✅ | | | -| [NLP Cloud](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | | | -| [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | | | -| [Petals](https://docs.litellm.ai/docs/providers/petals) | ✅ | | | -| [Ollama](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | | -| [DeepInfra](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | | | -| [Perplexity AI](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | | | -| [Groq AI](https://docs.litellm.ai/docs/providers/groq) | ✅ | | | -| [Deepseek](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | | | -| [Anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | | | -| [IBM - Watsonx.ai](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | | -| [Voyage AI](https://docs.litellm.ai/docs/providers/voyage) | | ✅ | | -| [Xinference [Xorbits Inference]](https://docs.litellm.ai/docs/providers/xinference) | | ✅ | | -| [FriendliAI](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | | | -| [Galadriel](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | | | -| [GradientAI](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | | | -| [Novita AI](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | | | -| [Featherless AI](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | | | -| [Nebius AI Studio](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | | -| [Heroku](https://docs.litellm.ai/docs/providers/heroku) | ✅ | | | -| [OVHCloud AI Endpoints](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | | | -| [CometAPI](https://docs.litellm.ai/docs/providers/cometapi) | ✅ | ✅ | ✅ | + +| Provider | `/chat/completions` | `/messages` | `/responses` | `/embeddings` | `/image/generations` | `/audio/transcriptions` | `/audio/speech` | `/moderations` | `/batches` | `/rerank` | +|-------------------------------------------------------------------------------------|---------------------|-------------|--------------|---------------|----------------------|-------------------------|-----------------|----------------|-----------|-----------| +| [AI/ML API (`aiml`)](https://docs.litellm.ai/docs/providers/aiml) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | +| [AI21 (`ai21`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | | +| [AI21 Chat (`ai21_chat`)](https://docs.litellm.ai/docs/providers/ai21) | ✅ | ✅ | ✅ | | | | | | | | +| [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha) | ✅ | ✅ | ✅ | | | | | | | | +| [Anthropic (`anthropic`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | | +| [Anthropic Text (`anthropic_text`)](https://docs.litellm.ai/docs/providers/anthropic) | ✅ | ✅ | ✅ | | | | | | ✅ | | +| [Anyscale](https://docs.litellm.ai/docs/providers/anyscale) | ✅ | ✅ | ✅ | | | | | | | | +| [AssemblyAI (`assemblyai`)](https://docs.litellm.ai/docs/pass_through/assembly_ai) | ✅ | ✅ | ✅ | | | ✅ | | | | | +| [Auto Router (`auto_router`)](https://docs.litellm.ai/docs/proxy/auto_routing) | ✅ | ✅ | ✅ | | | | | | | | +| [AWS - Bedrock (`bedrock`)](https://docs.litellm.ai/docs/providers/bedrock) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | +| [AWS - Sagemaker (`sagemaker`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [Azure (`azure`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | +| [Azure AI (`azure_ai`)](https://docs.litellm.ai/docs/providers/azure_ai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | +| [Azure Text (`azure_text`)](https://docs.litellm.ai/docs/providers/azure) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | +| [Baseten (`baseten`)](https://docs.litellm.ai/docs/providers/baseten) | ✅ | ✅ | ✅ | | | | | | | | +| [Bytez (`bytez`)](https://docs.litellm.ai/docs/providers/bytez) | ✅ | ✅ | ✅ | | | | | | | | +| [Cerebras (`cerebras`)](https://docs.litellm.ai/docs/providers/cerebras) | ✅ | ✅ | ✅ | | | | | | | | +| [Clarifai (`clarifai`)](https://docs.litellm.ai/docs/providers/clarifai) | ✅ | ✅ | ✅ | | | | | | | | +| [Cloudflare AI Workers (`cloudflare`)](https://docs.litellm.ai/docs/providers/cloudflare_workers) | ✅ | ✅ | ✅ | | | | | | | | +| [Codestral (`codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | | +| [Cohere (`cohere`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | +| [Cohere Chat (`cohere_chat`)](https://docs.litellm.ai/docs/providers/cohere) | ✅ | ✅ | ✅ | | | | | | | | +| [CometAPI (`cometapi`)](https://docs.litellm.ai/docs/providers/cometapi) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [CompactifAI (`compactifai`)](https://docs.litellm.ai/docs/providers/compactifai) | ✅ | ✅ | ✅ | | | | | | | | +| [Custom (`custom`)](https://docs.litellm.ai/docs/providers/custom_llm_server) | ✅ | ✅ | ✅ | | | | | | | | +| [Custom OpenAI (`custom_openai`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | +| [Dashscope (`dashscope`)](https://docs.litellm.ai/docs/providers/dashscope) | ✅ | ✅ | ✅ | | | | | | | | +| [Databricks (`databricks`)](https://docs.litellm.ai/docs/providers/databricks) | ✅ | ✅ | ✅ | | | | | | | | +| [DataRobot (`datarobot`)](https://docs.litellm.ai/docs/providers/datarobot) | ✅ | ✅ | ✅ | | | | | | | | +| [Deepgram (`deepgram`)](https://docs.litellm.ai/docs/providers/deepgram) | ✅ | ✅ | ✅ | | | ✅ | | | | | +| [DeepInfra (`deepinfra`)](https://docs.litellm.ai/docs/providers/deepinfra) | ✅ | ✅ | ✅ | | | | | | | | +| [Deepseek (`deepseek`)](https://docs.litellm.ai/docs/providers/deepseek) | ✅ | ✅ | ✅ | | | | | | | | +| [ElevenLabs (`elevenlabs`)](https://docs.litellm.ai/docs/providers/elevenlabs) | ✅ | ✅ | ✅ | | | | ✅ | | | | +| [Empower (`empower`)](https://docs.litellm.ai/docs/providers/empower) | ✅ | ✅ | ✅ | | | | | | | | +| [Fal AI (`fal_ai`)](https://docs.litellm.ai/docs/providers/fal_ai) | ✅ | ✅ | ✅ | | ✅ | | | | | | +| [Featherless AI (`featherless_ai`)](https://docs.litellm.ai/docs/providers/featherless_ai) | ✅ | ✅ | ✅ | | | | | | | | +| [Fireworks AI (`fireworks_ai`)](https://docs.litellm.ai/docs/providers/fireworks_ai) | ✅ | ✅ | ✅ | | | | | | | | +| [FriendliAI (`friendliai`)](https://docs.litellm.ai/docs/providers/friendliai) | ✅ | ✅ | ✅ | | | | | | | | +| [Galadriel (`galadriel`)](https://docs.litellm.ai/docs/providers/galadriel) | ✅ | ✅ | ✅ | | | | | | | | +| [GitHub Copilot (`github_copilot`)](https://docs.litellm.ai/docs/providers/github_copilot) | ✅ | ✅ | ✅ | | | | | | | | +| [GitHub Models (`github`)](https://docs.litellm.ai/docs/providers/github) | ✅ | ✅ | ✅ | | | | | | | | +| [Google - PaLM](https://docs.litellm.ai/docs/providers/palm) | ✅ | ✅ | ✅ | | | | | | | | +| [Google - Vertex AI (`vertex_ai`)](https://docs.litellm.ai/docs/providers/vertex) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | +| [Google AI Studio - Gemini (`gemini`)](https://docs.litellm.ai/docs/providers/gemini) | ✅ | ✅ | ✅ | | | | | | | | +| [GradientAI (`gradient_ai`)](https://docs.litellm.ai/docs/providers/gradient_ai) | ✅ | ✅ | ✅ | | | | | | | | +| [Groq AI (`groq`)](https://docs.litellm.ai/docs/providers/groq) | ✅ | ✅ | ✅ | | | | | | | | +| [Heroku (`heroku`)](https://docs.litellm.ai/docs/providers/heroku) | ✅ | ✅ | ✅ | | | | | | | | +| [Hosted VLLM (`hosted_vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | | +| [Huggingface (`huggingface`)](https://docs.litellm.ai/docs/providers/huggingface) | ✅ | ✅ | ✅ | ✅ | | | | | | ✅ | +| [Hyperbolic (`hyperbolic`)](https://docs.litellm.ai/docs/providers/hyperbolic) | ✅ | ✅ | ✅ | | | | | | | | +| [IBM - Watsonx.ai (`watsonx`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [Infinity (`infinity`)](https://docs.litellm.ai/docs/providers/infinity) | | | | ✅ | | | | | | | +| [Jina AI (`jina_ai`)](https://docs.litellm.ai/docs/providers/jina_ai) | | | | ✅ | | | | | | | +| [Lambda AI (`lambda_ai`)](https://docs.litellm.ai/docs/providers/lambda_ai) | ✅ | ✅ | ✅ | | | | | | | | +| [Lemonade (`lemonade`)](https://docs.litellm.ai/docs/providers/lemonade) | ✅ | ✅ | ✅ | | | | | | | | +| [LiteLLM Proxy (`litellm_proxy`)](https://docs.litellm.ai/docs/providers/litellm_proxy) | ✅ | ✅ | ✅ | ✅ | ✅ | | | | | | +| [Llamafile (`llamafile`)](https://docs.litellm.ai/docs/providers/llamafile) | ✅ | ✅ | ✅ | | | | | | | | +| [LM Studio (`lm_studio`)](https://docs.litellm.ai/docs/providers/lm_studio) | ✅ | ✅ | ✅ | | | | | | | | +| [Maritalk (`maritalk`)](https://docs.litellm.ai/docs/providers/maritalk) | ✅ | ✅ | ✅ | | | | | | | | +| [Meta - Llama API (`meta_llama`)](https://docs.litellm.ai/docs/providers/meta_llama) | ✅ | ✅ | ✅ | | | | | | | | +| [Mistral AI API (`mistral`)](https://docs.litellm.ai/docs/providers/mistral) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [Moonshot (`moonshot`)](https://docs.litellm.ai/docs/providers/moonshot) | ✅ | ✅ | ✅ | | | | | | | | +| [Morph (`morph`)](https://docs.litellm.ai/docs/providers/morph) | ✅ | ✅ | ✅ | | | | | | | | +| [Nebius AI Studio (`nebius`)](https://docs.litellm.ai/docs/providers/nebius) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [NLP Cloud (`nlp_cloud`)](https://docs.litellm.ai/docs/providers/nlp_cloud) | ✅ | ✅ | ✅ | | | | | | | | +| [Novita AI (`novita`)](https://novita.ai/models/llm?utm_source=github_litellm&utm_medium=github_readme&utm_campaign=github_link) | ✅ | ✅ | ✅ | | | | | | | | +| [Nscale (`nscale`)](https://docs.litellm.ai/docs/providers/nscale) | ✅ | ✅ | ✅ | | | | | | | | +| [Nvidia NIM (`nvidia_nim`)](https://docs.litellm.ai/docs/providers/nvidia_nim) | ✅ | ✅ | ✅ | | | | | | | | +| [OCI (`oci`)](https://docs.litellm.ai/docs/providers/oci) | ✅ | ✅ | ✅ | | | | | | | | +| [Ollama (`ollama`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | ✅ | | | | | | | +| [Ollama Chat (`ollama_chat`)](https://docs.litellm.ai/docs/providers/ollama) | ✅ | ✅ | ✅ | | | | | | | | +| [Oobabooga (`oobabooga`)](https://docs.litellm.ai/docs/providers/openai_compatible) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | +| [OpenAI (`openai`)](https://docs.litellm.ai/docs/providers/openai) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | +| [OpenAI-like (`openai_like`)](https://docs.litellm.ai/docs/providers/openai_compatible) | | | | ✅ | | | | | | | +| [OpenRouter (`openrouter`)](https://docs.litellm.ai/docs/providers/openrouter) | ✅ | ✅ | ✅ | | | | | | | | +| [OVHCloud AI Endpoints (`ovhcloud`)](https://docs.litellm.ai/docs/providers/ovhcloud) | ✅ | ✅ | ✅ | | | | | | | | +| [Perplexity AI (`perplexity`)](https://docs.litellm.ai/docs/providers/perplexity) | ✅ | ✅ | ✅ | | | | | | | | +| [Petals (`petals`)](https://docs.litellm.ai/docs/providers/petals) | ✅ | ✅ | ✅ | | | | | | | | +| [Predibase (`predibase`)](https://docs.litellm.ai/docs/providers/predibase) | ✅ | ✅ | ✅ | | | | | | | | +| [Recraft (`recraft`)](https://docs.litellm.ai/docs/providers/recraft) | | | | | ✅ | | | | | | +| [Replicate (`replicate`)](https://docs.litellm.ai/docs/providers/replicate) | ✅ | ✅ | ✅ | | | | | | | | +| [Sagemaker Chat (`sagemaker_chat`)](https://docs.litellm.ai/docs/providers/aws_sagemaker) | ✅ | ✅ | ✅ | | | | | | | | +| [Sambanova (`sambanova`)](https://docs.litellm.ai/docs/providers/sambanova) | ✅ | ✅ | ✅ | | | | | | | | +| [Snowflake (`snowflake`)](https://docs.litellm.ai/docs/providers/snowflake) | ✅ | ✅ | ✅ | | | | | | | | +| [Text Completion Codestral (`text-completion-codestral`)](https://docs.litellm.ai/docs/providers/codestral) | ✅ | ✅ | ✅ | | | | | | | | +| [Text Completion OpenAI (`text-completion-openai`)](https://docs.litellm.ai/docs/providers/text_completion_openai) | ✅ | ✅ | ✅ | | | ✅ | ✅ | ✅ | ✅ | | +| [Together AI (`together_ai`)](https://docs.litellm.ai/docs/providers/togetherai) | ✅ | ✅ | ✅ | | | | | | | | +| [Topaz (`topaz`)](https://docs.litellm.ai/docs/providers/topaz) | ✅ | ✅ | ✅ | | | | | | | | +| [Triton (`triton`)](https://docs.litellm.ai/docs/providers/triton-inference-server) | ✅ | ✅ | ✅ | | | | | | | | +| [V0 (`v0`)](https://docs.litellm.ai/docs/providers/v0) | ✅ | ✅ | ✅ | | | | | | | | +| [Vercel AI Gateway (`vercel_ai_gateway`)](https://docs.litellm.ai/docs/providers/vercel_ai_gateway) | ✅ | ✅ | ✅ | | | | | | | | +| [VLLM (`vllm`)](https://docs.litellm.ai/docs/providers/vllm) | ✅ | ✅ | ✅ | | | | | | | | +| [Volcengine (`volcengine`)](https://docs.litellm.ai/docs/providers/volcano) | ✅ | ✅ | ✅ | | | | | | | | +| [Voyage AI (`voyage`)](https://docs.litellm.ai/docs/providers/voyage) | | | | ✅ | | | | | | | +| [WandB Inference (`wandb`)](https://docs.litellm.ai/docs/providers/wandb_inference) | ✅ | ✅ | ✅ | | | | | | | | +| [Watsonx Text (`watsonx_text`)](https://docs.litellm.ai/docs/providers/watsonx) | ✅ | ✅ | ✅ | | | | | | | | +| [xAI (`xai`)](https://docs.litellm.ai/docs/providers/xai) | ✅ | ✅ | ✅ | | | | | | | | +| [Xinference (`xinference`)](https://docs.litellm.ai/docs/providers/xinference) | | | | ✅ | | | | | | | [**Read the Docs**](https://docs.litellm.ai/docs/) diff --git a/docs/my-website/docs/proxy/config_settings.md b/docs/my-website/docs/proxy/config_settings.md index caa025cf1e..003e02d4d5 100644 --- a/docs/my-website/docs/proxy/config_settings.md +++ b/docs/my-website/docs/proxy/config_settings.md @@ -361,6 +361,10 @@ router_settings: | AIOHTTP_TRUST_ENV | Flag to enable aiohttp trust environment. When this is set to True, aiohttp will respect HTTP(S)_PROXY env vars. **Default is False** | AIOHTTP_TTL_DNS_CACHE | DNS cache time-to-live for aiohttp in seconds. **Default is 300** | ALLOWED_EMAIL_DOMAINS | List of email domains allowed for access +| APSCHEDULER_COALESCE | Whether to combine multiple pending executions of a job into one. **Default is False** +| APSCHEDULER_MAX_INSTANCES | Maximum number of concurrent instances of each job. **Default is 1** +| APSCHEDULER_MISFIRE_GRACE_TIME | Grace time in seconds for misfired jobs. **Default is 1** +| APSCHEDULER_REPLACE_EXISTING | Whether to replace existing jobs with the same ID. **Default is False** | ARIZE_API_KEY | API key for Arize platform integration | ARIZE_SPACE_KEY | Space key for Arize platform | ARGILLA_BATCH_SIZE | Batch size for Argilla logging diff --git a/tests/documentation_tests/test_readme_providers.py b/tests/documentation_tests/test_readme_providers.py new file mode 100644 index 0000000000..4ab140fa0d --- /dev/null +++ b/tests/documentation_tests/test_readme_providers.py @@ -0,0 +1,179 @@ +""" +Test to ensure all providers in LlmProviders enum are documented in README.md +""" + +import os +import re +from litellm.types.utils import LlmProviders + +# Define paths +readme_path = "./README.md" + +# Providers that shouldn't be required in README +# (specialized tools, observability, database providers that aren't LLM providers) +EXCLUDED_PROVIDERS = { + "aiohttp_openai", # internal http variant + "langfuse", # observability, not LLM provider + "humanloop", # observability, not LLM provider + "pg_vector", # database, not LLM provider + "dotprompt", # prompt management, not provider + "vertex_ai_beta", # beta variant, not needed in main table +} + +def get_enum_providers(): + """ + Get all provider values from LlmProviders enum, excluding internal variants. + """ + providers = set() + for provider in LlmProviders: + provider_value = provider.value + if provider_value not in EXCLUDED_PROVIDERS: + providers.add(provider_value) + return providers + + +def get_readme_providers(): + """ + Extract provider slugs from README.md provider table. + + Looks for provider slugs in backticks within parentheses, e.g.: + [OpenAI (`openai`)](url) -> extracts "openai" + """ + providers = set() + try: + with open(readme_path, "r", encoding="utf-8") as f: + content = f.read() + + # Find the supported providers table + # Look for the section starting with "## Supported Providers" + providers_section = re.search( + r"## Supported Providers.*?\n\n(.*?)(?=\n##|\Z)", + content, + re.DOTALL | re.MULTILINE, + ) + + if providers_section: + table_content = providers_section.group(1) + # Extract provider slugs from backticks in parentheses: (`slug`) + provider_slug_pattern = re.compile(r"\(`([^`]+)`\)") + matches = provider_slug_pattern.findall(table_content) + providers.update(matches) + else: + raise Exception("Could not find 'Supported Providers' section in README.md") + + except FileNotFoundError: + raise Exception(f"README.md not found at {readme_path}") + except Exception as e: + raise Exception(f"Error reading README.md: {e}") + + return providers + + +def get_readme_provider_names(): + """ + Extract provider display names from README.md provider table in order. + + Returns a list of provider names as they appear in the table. + """ + provider_names = [] + try: + with open(readme_path, "r", encoding="utf-8") as f: + content = f.read() + + # Find the supported providers table + providers_section = re.search( + r"## Supported Providers.*?\n\n(.*?)(?=\n\[|\n##|\Z)", + content, + re.DOTALL | re.MULTILINE, + ) + + if providers_section: + table_content = providers_section.group(1) + # Extract provider names from table rows that start with | + # Split by lines and process each line + for line in table_content.split('\n'): + # Only process lines that are table rows (start with |) + if line.strip().startswith('|') and '[' in line: + # Extract provider name from: | [Provider Name (...)](...) | + match = re.search(r'\|\s*\[([^\]]+)\]\(', line) + if match: + provider_name = match.group(1) + # Skip header row and separator row + if provider_name != "Provider" and not provider_name.startswith('-'): + provider_names.append(provider_name) + else: + raise Exception("Could not find 'Supported Providers' section in README.md") + + except FileNotFoundError: + raise Exception(f"README.md not found at {readme_path}") + except Exception as e: + raise Exception(f"Error reading README.md: {e}") + + return provider_names + + +def test_all_providers_documented(): + """ + Test that all providers in LlmProviders enum are documented in README.md. + + Verifies that provider slugs in the enum match the slugs shown in backticks + in the README provider table. + """ + enum_providers = get_enum_providers() + readme_providers = get_readme_providers() + + print(f"\nProvider slugs in LlmProviders enum (filtered): {sorted(enum_providers)}") + print(f"\nProvider slugs in README.md: {sorted(readme_providers)}") + + # Find undocumented providers + undocumented = enum_providers - readme_providers + + if undocumented: + raise AssertionError( + f"\nProvider slugs not documented in README.md: {sorted(undocumented)}\n" + f"Please add these providers to the README.md provider table with their slugs in backticks.\n" + f"Example: [Provider Name (`slug`)](url)" + ) + else: + print(f"\n✓ All {len(enum_providers)} provider slugs are documented in README.md") + + +def test_providers_alphabetically_ordered(): + """ + Test that providers in README.md are listed in alphabetical order. + """ + provider_names = get_readme_provider_names() + + if not provider_names: + raise AssertionError("No provider names found in README.md") + + # Create a sorted version for comparison + sorted_names = sorted(provider_names, key=str.lower) + + print(f"\nFound {len(provider_names)} providers in README.md") + + # Check if the list is alphabetically ordered + out_of_order = [] + for i, (actual, expected) in enumerate(zip(provider_names, sorted_names)): + if actual != expected: + out_of_order.append({ + "position": i + 1, + "actual": actual, + "expected": expected + }) + + if out_of_order: + error_msg = "\nProviders are not in alphabetical order:\n" + for item in out_of_order[:10]: # Show first 10 issues + error_msg += f" Position {item['position']}: Found '{item['actual']}', expected '{item['expected']}'\n" + if len(out_of_order) > 10: + error_msg += f" ... and {len(out_of_order) - 10} more issues\n" + raise AssertionError(error_msg) + else: + print(f"✓ All {len(provider_names)} providers are in alphabetical order") + + +if __name__ == "__main__": + test_all_providers_documented() + test_providers_alphabetically_ordered() +