structured-logprobs

structured-logprobs is an open-source Python library that develops The structured outputs of OpenAI by providing detailed information about the token log probabilities.

This library is designed to provide valuable insights into reliability of structured LLM outputs. It works with OpenAI’s Built-in Outputsa feature that ensures that the model always produces responses that adhere to a given JSON Schema. This eliminates worries about missing required keys or imagining invalid values.

Just install with the pip install structured-logprobs

Then use it this way:

from openai import OpenAI
from structured_logprobs.main import add_logprobs

client = OpenAI()
completion = client.chat.completions.create(
    model="gpt-4o",
    messages=(
        {
            "role": "system",
            "content": (
                "Please output metadata about 'structured-logprobs'"
            ),
        }
    ),
    logprobs=True,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "answear",
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "version": {"type": "string"},
                },
            },
        },
    },
)
chat_completion = add_logprobs(completion)
print(chat_completion)

For more details, visit dash.

The module has a function for mapping characters to token indices (map_characters_to_token_indices) and two methods of joining log probabilities:

1. Add log probabilities as a separate response field (add_logprobs).
2. Embed the log probabilities inline within the message (add_logprobs_inline).