A first tool usage system inspired by a similar system in
ellmer has been introduced to tidyllm. At the moment tool
use is available for claude(), openai(),
mistral(), ollama(), gemini() and
groq():
get_current_time <- function(tz, format = "%Y-%m-%d %H:%M:%S") {
format(Sys.time(), tz = tz, format = format, usetz = TRUE)
}
time_tool <- tidyllm_tool(
.f = get_current_time,
.description = "Returns the current time in a specified timezone. Use this to determine the current time in any location.",
tz = field_chr("The time zone identifier (e.g., 'Europe/Berlin', 'America/New_York', 'Asia/Tokyo', 'UTC'). Required."),
format = field_chr("Format string for the time output. Default is '%Y-%m-%d %H:%M:%S'.")
)
llm_message("What's the exact time in Stuttgart?") |>
chat(openai,.tools=time_tool)
#> Message History:
#> system:
#> You are a helpful assistant
#> --------------------------------------------------------------
#> user:
#> What's the exact time in Stuttgart?
#> --------------------------------------------------------------
#> assistant:
#> The current time in Stuttgart (Europe/Berlin timezone) is
#> 2025-03-03 09:51:22 CET.
#> -------------------------------------------------------------- You can use the tidyllm_tool() function to run to make
functions available to a large language model. The model can then run
these functions in your current session, if they are needed for a chat
request.
tidyllm now supports the deepseek API as provider via
deepseek_chat() or the deepseek() provider
function. Deepseek supports logprobs just like openai(),
which you can get via get_logprobs(). At the moment tool
usage for deepseek is very inconsistent.
Voyage.ai introduces a unique multimodal embeddings
feature, allowing you to generate embeddings not only for text
but also for images. The new voyage_embedding() function in
tidyllm enables this functionality by seamlessly
handling different input types, working with both the new feature as
well as the same inputs as for other embedding functions.
The new img() function lets you create image objects for
embedding. You can mix text and img() objects in a list and
send them to Voyage AI for multimodal embeddings:
list("tidyllm", img(here::here("docs", "logo.png"))) |>
embed(voyage)
#> # A tibble: 2 × 2
#> input embeddings
#> <chr> <list>
#> 1 tidyllm <dbl [1,024]>
#> 2 [IMG] logo.png <dbl [1,024]>In this example, both text ("tidyllm") and an image
(logo.png) are embedded together. The function returns a
tibble where the input column contains the text and labeled
image names, and the embeddings column contains the
corresponding embedding vectors.
tidyllm_schema() and
tidyllm_tool()⚠️ There is a bad bug in the latest CRAN release in the
fetch_openai_batch() function that is now fixed in the
latest Github version. For the CRAN version the
fetch_openai_batch() function throws errors if the logprobs
are turned off.
.json_schema option of api-functions. Moreover,
tidyllm_schema() now accepts ellmer types as field
definitions. In addition four ellmer-inspired type-definition
functionsfield_chr(), field_dbl(),
field_lgl() and field_fct() were added that
allow you to set description fields in schemata ellmer_adress <-ellmer::type_object(
street = ellmer::type_string("A famous street"),
houseNumber = ellmer::type_number("a 3 digit number"),
postcode = ellmer::type_string(),
city = ellmer::type_string("A large city"),
region = ellmer::type_string(),
country = ellmer::type_enum(values = c("Germany", "France"))
)
person_schema <- tidyllm_schema(
person_name = "string",
age = field_dbl("An age between 25 and 40"),
is_employed = field_lgl("Employment Status in the last year")
occupation = field_fct(.levels=c("Lawyer","Butcher")),
address = ellmer_adress
)
address_message <- llm_message("imagine an address") |>
chat(openai,.json_schema = ellmer_adress)
person_message <- llm_message("imagine a person profile") |>
chat(openai,.json_schema = person_schema)openai_chat() and
send_openai_batch() and new get_logprobs()
function:badger_poem <- llm_message("Write a haiku about badgers") |>
chat(openai(.logprobs=TRUE,.top_logprobs=5))
badger_poem |> get_logprobs()
#> # A tibble: 19 × 5
#> reply_index token logprob bytes top_logprobs
#> <int> <chr> <dbl> <list> <list>
#> 1 1 "In" -0.491 <int [2]> <list [5]>
#> 2 1 " moon" -1.12 <int [5]> <list [5]>
#> 3 1 "lit" -0.00489 <int [3]> <list [5]>
#> 4 1 " forest" -1.18 <int [7]> <list [5]>
#> 5 1 "," -0.00532 <int [1]> <list [5]> ollama_delete_model() functionlist_models() is now a verb supporting most
providers.list_models(openai)
#> # A tibble: 52 × 3
#> id created owned_by
#> <chr> <chr> <chr>
#> 1 gpt-4o-mini-audio-preview-2024-12-17 2024-12-13 18:52:00 system
#> 2 gpt-4-turbo-2024-04-09 2024-04-08 18:41:17 system
#> 3 dall-e-3 2023-10-31 20:46:29 system
#> 4 dall-e-2 2023-11-01 00:22:57 system send_ollama_batch() function to make
use of the fast parallel request features of Ollama.openai() reasoning models
supportedperplexity() and
gemini()LLMMessagetidyllm 0.3.0 represents a major milestone for tidyllm
The largest changes compared to 0.2.0 are:
chat(), embed(),
send_batch(), check_batch(), and
fetch_batch() to interact with APIs. These functions always
work with a combination of verbs and providers:
chat(),
embed(), send_batch()) define the type of
action you want to perform.openai(),
claude(), ollama()) are an arguement of verbs
and specify the API to handle the action with and take provider-specific
argumentsEach verb and provider combination routes the interaction to
provider-specific functions like openai_chat() or
claude_chat() that do the work in the background. These
functions can also be called directly as an alternative more verbose and
provider-specific interface.
llm_message("Hello World") |>
openai(.model = "gpt-4o")# Recommended Verb-Based Approach
llm_message("Hello World") |>
chat(openai(.model = "gpt-4o"))
# Or even configuring a provider outside
my_ollama <- ollama(.model = "llama3.2-vision:90B",
.ollama_server = "https://ollama.example-server.de",
.temperature = 0)
llm_message("Hello World") |>
chat(my_ollama)
# Alternative Approach is to use more verbose specific functions:
llm_message("Hello World") |>
openai_chat(.model = "gpt-4o")openai(), claude(),
etc.) still work if you directly supply an LLMMessage as
arguement, but issue deprecation warnings when used directly for
chat.R6-based saved LLMMessage objects are no
longer compatible with the new version. Saved objects from earlier
versions need to be re-createdgemini() and perplexity() as new supported
API providers. gemini() brings interesting Video and Audio
features as well as search grounding to tidyllm.
perplexity() also offers well cited search grounded
assitant repliesmistral()get_reply_metadata()
to get information on token usage, or on other relevant metadata (like
sources used for grounding)R6 to S7 for the main
LLMMessage class, improving maintainability,
interoperability, and future-proofing..grounding_threshold argument
added of the gemini_chat() function allowing you to use
Google searches to ground model responses to a search result Gemini
models. For example, asking about the maintainer of an obscure R package
works with grounding but does only lead to a hallucination without:llm_message("What is tidyllm and who maintains this package?") |>
gemini_chat(.grounding_threshold = 0.3)perplexity_chat(). The neat feature of perplexity is the
up-to-date web search it does with detailed citations. Cited sources are
available in the api_specific-list column of
get_metadata().json_schema support for ollama()
available with Ollama 0.5.0get_metadata() returns a list column with API-specific
metadataR6 to S7 for the main
LLMMessage classdf_llm_message()APIProvider classesapi_openai.R,api_gemini.R,etc. filesas_tibble() S3 Generic for
LLMMessagetrack_rate_limit().onattach() removedR6-based LLMMessage-objects are not
compatible with the new version anymore! This also applies to saved
objects, like lists of batch files.here::here("local_wip","example.mp3") |> gemini_upload_file()
here::here("local_wip","legrille.mp4") |> gemini_upload_file()
file_tibble <- gemini_list_files()
llm_message("What are these two files about?") |>
gemini_chat(.fileid=file_tibble$name)Better embedding functions with improved output and error handling
and new documentation. New article on using embeddings with
tidyllm. Support for embedding models on azure with
azure_openai_embedding()
embed() and the related
API-specific functions was changed from a matrix to a tibble with an
input column and a list column containing one embedding vector and one
input per row.One disadvantage of the first iteration of the new interface was that
all arguements that needed to be passed to provider-specific functions,
were going through the provider function. This feels, unintuitive,
because users expect common arguments (e.g., .model, .temperature) to be
set directly in main verbs like chat() or
send_batch().Moreover, provider functions don’t expose
arguments for autocomplete, making it harder for users to explore
options. Therefore, the main API verbs now directly accept common
arguements, and check them against the available arguements for each
API.
tidyllm has introduced a verb-based interface overhaul
to provide a more intuitive and flexible user experience. Previously,
provider-specific functions like claude(),
openai(), and others were directly used for chat-based
workflows. Now, these functions primarily serve as provider
configuration for some general verbs like chat().
chat(), embed(),
send_batch(), check_batch(), and
fetch_batch() to interact with APIs. These functions always
work with a combination of verbs and providers:
chat(),
embed(), send_batch()) define the type of
action you want to perform.openai(),
claude(), ollama()) are an arguement of verbs
and specify the API to handle the action with and take provider-specific
argumentsEach verb and provider combination routes the interaction to
provider-specific functions like openai_chat() or
claude_chat() that do the work in the background. These
functions can also be called directly as an alternative more verbose and
provider-specific interface.
llm_message("Hello World") |>
openai(.model = "gpt-4o")# Recommended Verb-Based Approach
llm_message("Hello World") |>
chat(openai(.model = "gpt-4o"))
# Or even configuring a provider outside
my_ollama <- ollama(.model = "llama3.2-vision:90B",
.ollama_server = "https://ollama.example-server.de",
.temperature = 0)
llm_message("Hello World") |>
chat(my_ollama)
# Alternative Approach is to use more verbose specific functions:
llm_message("Hello World") |>
openai_chat(.model = "gpt-4o")openai(), claude(),
etc.) still work if you directly supply an LLMMessage as
arguement, but issue deprecation warnings when used directly for
chat.gemini() main API-function#Upload a file for use with gemini
upload_info <- gemini_upload_file("example.mp3")
#Make the file available during a Gemini API call
llm_message("Summarize this speech") |>
gemini(.fileid = upload_info$name)
#Delte the file from the Google servers
gemini_delete_file(upload_info$name)tidyllm_schema()gemini()-requests allow for a wide range of file types
that can be used for context in messagesgemini() file workflows:
application/pdftext/plaintext/htmltext/csstext/mdtext/csvtext/xmltext/rtfgemini() file workflows:
application/x-javascript,
text/javascriptapplication/x-python,
text/x-pythongemini() file workflows:
image/pngimage/jpegimage/webpimage/heicimage/heifgemini() file workflows:
video/mp4video/mpegvideo/movvideo/avivideo/x-flvvideo/mpgvideo/webmvideo/wmvvideo/3gppgemini() file workflows:
audio/wavaudio/mp3audio/aiffaudio/aacaudio/oggaudio/flacget_metadata() function to retrieve and format
metadata from LLMMessage objects.print method for LLMMessage
to support printing metadata, controlled via the new
tidyllm_print_metadata option or a new
.meta-arguement for the print method.conversation <- llm_message("Write a short poem about software development") |>
claude()
#Get metdata on token usage and model as tibble
get_metadata(conversation)
#or print it with the message
print(conversation,.meta=TRUE)
#Or allways print it
options(tidyllm_print_metadata=TRUE)send_openai_batch() caused by a missing
.json-arguement not being passed for messages without
schemaNew CRAN release. Largest changes compared to 0.1.0:
Major Features:
.json_schema handling in openai(),
enhancing support for well-defined JSON responses.azure_openai()
function for accessing the Azure OpenAI service, with full support for
rate-limiting and batch operations tailored to Azure’s API
structure.mistral() function
provides full support for Mistral models hosted in the EU, including
rate-limiting and streaming capabilities.pdf_page_batch()
function, which processes PDFs page by page, allowing users to define
page-specific prompts for detailed analysis..compatible argument (and flexible url and path) in
openai() to allow compatibility with third-party
OpenAI-compatible APIs.Improvements:
to_api_format() to reduce code duplication, simplify API
format generation, and improve maintainability.httr2::req_retry() in addition to the rate-limit tracking
functions in tidyllm, using 429 headers to wait for rate limit
resets.httptest2Breaking Changes:
get_reply() was split into
get_reply() for text outputs and
get_reply_data() for structured outputs, improving type
stability compared to an earlier function that had different outputs
based on a .json-arguement.chatgpt(): The chatgpt()
function has been deprecated in favor of openai() for
feature alignment and improved consistency.Minor Updates and Bug Fixes:
llm_message(): Allows
extraction of specific page ranges from PDFs, improving flexibility in
document handling.ollama_download_model() function to download models
from the Ollama API.compatible-arguement in openai() to
allow working with compatible third party APIsto_api_format():
API format generation now has much less code duplication and is more
maintainable.get_reply() was split into two type-stable functions:
get_reply() for text and get_reply_data() for
structured outputs.httr2::req_retry(): Rate limiting now uses the
right 429 headers where they come.Enhanced Input Validation: All API functions now have improved input validation, ensuring better alignment with API documentation
Improved error handling More human-readable error messages for failed requests from the API
Advanced JSON Mode in openai(): The
openai() function now supports advanced
.json_schemas, allowing structured output in JSON mode for
more precise responses.
Reasoning Models Support: Support for O1
reasoning models has been added, with better handling of system prompts
in the openai() function.
Streaming callback functions refactored: Given that the streaming callback format for Open AI, Mistral and Groq is nearly identical the three now rely on the same callback function.
chatgpt() Deprecated: The
chatgpt() function has been deprecated in favor of
openai(). Users should migrate to openai() to
take advantage of the new features and enhancements.openai(),
ollama(), and claude() functions now return
more informative error messages when API calls fail, helping with
debugging and troubleshooting.ollama_embedding() to generate embeddings using the
Ollama API.openai_embedding() to generate embeddings using the
OpenAI API.mistral_embedding() to generate embeddings using the
Mistral API.llm_message(): The
llm_message() function now supports specifying a range of
pages in a PDF by passing a list with filename,
start_page, and end_page. This allows users to
extract and process specific pages of a PDF.pdf_page_batch() function, which processes PDF files page
by page, extracting text and converting each page into an image,
allowing for a general prompt or page-specific prompts. The function
generates a list of LLMMessage objects that can be sent to
an API and work with the batch-API functions in
tidyllm.mistral() function to use Mistral Models on Le Platforme on
servers hosted in the EU, with rate-limiting and streaming support.last_user_message() pulls the last message the user
sent.get_reply() gets the assistant reply at a given index
of assistant messages.get_user_message() gets the user message at a given
index of user messages..dry_run argument, allowing users
to generate an httr2-request for easier debugging and
inspection.httptest2-based tests with mock responses for all API
functions, covering both basic functionality and rate-limiting.ollama_download_model() function to download models from
the Ollama API. It supports a streaming mode that provides live progress
bar updates on the download progress.llm_message()groq() function now supports images.llm_message().JSON Mode: JSON mode is now more widely
supported across all API functions, allowing for structured outputs when
APIs support them. The .json argument is now passed only to
API functions, specifying how the API should respond, and it is not
needed anymore in last_reply().
Improved last_reply() Behavior: The
behavior of the last_reply() function has changed. It now
automatically handles JSON replies by parsing them into structured data
and falling back to raw text in case of errors. You can still force raw
text replies even for JSON output using the .raw
argument.
last_reply(): The .json
argument is no longer used, and JSON replies are automatically parsed.
Use .raw to force raw text replies.