LLMSimilarityScorer

classDiagram Monitorable <|-- ScorerBase ScorerBase <|-- LLMSimilarityScorer

class council.scorers.LLMSimilarityScorer(llm: LLMBase, expected: str)

Bases: ScorerBase

Using an LLM to compute a similarity score between two messages.

__init__(llm: LLMBase, expected: str)

Initialize a new instance

Parameters:

• llm (LLMBase) – the LLM to be used

• expected (str) – the expected text message

score(context: ScorerContext, message: ChatMessage) → float

Score the given message

Parameters:

• context (ScorerContext) – the context for scoring

• message (ChatMessage) – the message to be scored

Returns:

similarity score. The greater the value to higher the similarity

Raises:

ScorerException – an unexpected error occurs

to_dict() → Dict[str, Any]

Serialize the instance into a dictionary. May need to be overriden in derived classes