LLM#

classDiagram BaseModelResponseParser <|-- CodeBlocksResponseParser BaseModelResponseParser <|-- JSONResponseParserBase BaseModelResponseParser <|-- YAMLResponseParserBase ChatGPTConfigurationBase <|-- AzureChatGPTConfiguration ChatGPTConfigurationBase <|-- OpenAIChatGPTConfiguration DataObject <|-- LLMConfigObject DataObject <|-- LLMCostManagerObject DataObjectSpecBase <|-- LLMConfigSpec DefaultLLMConsumptionCalculatorHelper <|-- DefaultLLMConsumptionCalculator JSONResponseParserBase <|-- JSONBlockResponseParser JSONResponseParserBase <|-- JSONResponseParser LLMBase <|-- AnthropicLLM LLMBase <|-- GeminiLLM LLMBase <|-- GroqLLM LLMBase <|-- LLMFallback LLMBase <|-- OllamaLLM LLMBase <|-- OpenAIChatCompletionsModel LLMConfigurationBase <|-- AnthropicLLMConfiguration LLMConfigurationBase <|-- ChatGPTConfigurationBase LLMConfigurationBase <|-- GeminiLLMConfiguration LLMConfigurationBase <|-- GroqLLMConfiguration LLMConfigurationBase <|-- OllamaLLMConfiguration LLMConsumptionCalculatorBase <|-- DefaultLLMConsumptionCalculatorHelper LLMException <|-- LLMCallException LLMException <|-- LLMCallTimeoutException LLMException <|-- LLMOutOfRetriesException LLMException <|-- LLMTokenLimitException LLMFunction <|-- LLMFunctionWithPrompt LLMFunctionError <|-- FunctionOutOfRetryError LLMLoggingMiddlewareBase <|-- LLMFileLoggingMiddleware LLMLoggingMiddlewareBase <|-- LLMLoggingMiddleware LLMMessageData <|-- LLMCacheControlData Monitorable <|-- LLMBase Monitored <|-- MonitoredLLM OpenAIChatCompletionsModel <|-- AzureLLM OpenAIChatCompletionsModel <|-- OpenAILLM YAMLResponseParserBase <|-- YAMLBlockResponseParser YAMLResponseParserBase <|-- YAMLResponseParser

This package provides clients to use various LLMs.

Overview#

The council.llm module provides a unified interface for interacting with various LLM providers, along with tools for handling responses, caching, logging and tracking consumptions metrics.

LLMs#

Create your LLM instance from YAML config file with LLMConfigObject (see for different config examples).

Currently supported providers include:

from council.llm import get_llm_from_config

# will adjust provider class automatically based on config file
llm = get_llm_from_config("data/configs/llm-config-openai.yaml")

Making Requests and Managing Costs#

Use llm.post_chat_request() method to interact with an LLM. The returned LLMResult object contains LLM response as well as list of Consumption metrics associated with the call, including duration, token usage and costs.

import dotenv
from council import LLMContext
from council.llm import LLMMessage, get_llm_from_config

llm = get_llm_from_config("data/configs/llm-config-openai.yaml")
result = llm.post_chat_request(
    LLMContext.empty(),
    messages=[LLMMessage.user_message("Hello world")]
)

print(result.first_choice)
# sample output:
# Hello! How can I assist you today?

for consumption in result.consumptions:
    print(consumption)
# sample output:
# gpt-4o-mini-2024-07-18 consumption: 1 call
# gpt-4o-mini-2024-07-18 consumption: 0.9347 second
# gpt-4o-mini-2024-07-18:prompt_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:completion_tokens consumption: 9 token
# gpt-4o-mini-2024-07-18:total_tokens consumption: 18 token
# gpt-4o-mini-2024-07-18:prompt_tokens_cost consumption: 1.3499e-06 USD
# gpt-4o-mini-2024-07-18:completion_tokens_cost consumption: 5.399e-06 USD
# gpt-4o-mini-2024-07-18:total_tokens_cost consumption: 6.7499e-06 USD

Anthropic Prompt Caching Support#

For information about enabling Anthropic prompt caching, refer to LLMCacheControlData.

LLM Functions#

LLM Functions provide structured ways to interact with LLMs including built-in response parsing, error handling and retries.

Response Parsers#

Response parsers help automate the parsing of common response formats to use LLMFunctions conveniently:

  • EchoResponseParser for raw LLMResponse

  • StringResponseParser for plain text

  • CodeBlocksResponseParser for code blocks

  • YAMLBlockResponseParser and YAMLResponseParser for YAML

  • JSONBlockResponseParser and JSONResponseParser for JSON

LLM Middleware#

Middleware components allow you to enhance LLM interactions by modifying requests and responses introducing custom logic, such as logging, caching, configuration updates, etc.

Core middlewares:

Middleware management:

Fine-tuning and Batch API#

See LLMDatasetObject for details on how to convert your YAML dataset into JSONL for fine-tuning and batch API. Currently, the functionality is limited to generating JSONL files and does not include utilities for managing fine-tuning or batch job processes.

Reference#