Create Chat Completion
Create Chat Completion
Parameters
If true, the generation prompt will be added to the chat template. This is a parameter used by chat template in tokenizer config of the model.
If true, special tokens (e.g. BOS) will be added to the prompt on top of what is added by the chat template. For most models, the chat template takes care of adding the special tokens so this should be set to false (as is the default).
If specified, the prefix cache will be salted with the provided string to prevent an attacker to guess prompts in multi-user environments. The salt should be random, protected from access by 3rd parties, and long enough to be unpredictable (e.g., 43 characters base64-encoded, corresponding to 256 bit).
A Jinja template to use for this conversion. As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.
Additional keyword args to pass to the template renderer. Will be accessible by the chat template.
If this is set, the chat will be formatted so that the final message in the chat is open-ended, without any EOS tokens. The model will continue this message rather than starting a new one. This allows you to "prefill" part of the model's response for it. Cannot be used at the same time as add_generation_prompt.
A list of dicts representing documents that will be accessible to the model if it is performing RAG (retrieval-augmented generation). If the template does not support RAG, this argument will have no effect. We recommend that each document should be a dict containing "title" and "text" keys.
If true, the new message will be prepended with the last message if they belong to the same role.
KVTransfer parameters used for disaggregated serving.
Additional kwargs to pass to the HF processor.
The priority of the request (lower means earlier handling; default: 0). Any priority other than 0 will raise an error if the served model does not use priority scheduling.
The request_id related to this request. If the caller does not set it, a random_uuid will be generated. This id is used through out the inference process and return in response.
If specified, the result will include token IDs alongside the generated text. In streaming mode, prompt_token_ids is included only in the first chunk, and token_ids contains the delta tokens for each chunk. This is useful for debugging or when you need to map generated text back to input tokens.
If specified with 'logprobs', tokens are represented as strings of the form 'token_id:{token_id}' so that tokens that are not JSON-encodable can be identified.
Returns
Create Chat Completion
import os
from tzafon import Lightcone
client = Lightcone(
api_key=os.environ.get("TZAFON_API_KEY"), # This is the default and can be omitted
)
response = client.chat.create_completion(
messages=[{
"content": "string",
"role": "developer",
}],
)
print(response){}Returns Examples
{}