Package: vitals 0.3.0.9001

Simon Couch

vitals: Large Language Model Evaluation

A port of 'Inspect', a widely adopted 'Python' framework for large language model evaluation. Specifically aimed at 'ellmer' users who want to measure the effectiveness of their large language model-based products, the package supports prompt engineering, tool usage, multi-turn dialog, and model graded evaluations.

Authors:Simon Couch [aut, cre], Max Kuhn [ctb], Hadley Wickham [ctb], Mine Cetinkaya-Rundel [ctb], Posit Software, PBC [cph, fnd]

vitals_0.3.0.9001.tar.gz
vitals_0.3.0.9001.zip(r-4.7)vitals_0.3.0.9001.zip(r-4.6)vitals_0.3.0.9001.zip(r-4.5)
vitals_0.3.0.9001.tgz(r-4.6-any)vitals_0.3.0.9001.tgz(r-4.5-any)
vitals_0.3.0.9001.tar.gz(r-4.7-any)vitals_0.3.0.9001.tar.gz(r-4.6-any)
vitals_0.3.0.9001.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
vitals/json (API)

# Install 'vitals' in R:

install.packages('vitals', repos = c('https://tidyverse.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/tidyverse/vitals/issues

Pkgdown/docs site:https://vitals.tidyverse.org

Datasets:

are - An R Eval

On CRAN:

7.60 score 57 stars 77 scripts 445 downloads 16 exports 35 dependencies

Last updated from:e1ad045bba. Checks:7 NOTE, 2 OK. Indexed: yes.

Target	Result	Time
linux-devel-x86_64	NOTE	216
source / vignettes	OK	268
linux-release-x86_64	NOTE	188
macos-release-arm64	NOTE	105
macos-oldrel-arm64	NOTE	89
windows-devel	NOTE	108
windows-release	NOTE	103
windows-oldrel	NOTE	123
wasm-release	OK	144

Exports:detect_answer detect_exact detect_includes detect_match detect_pattern generate generate_structured model_graded_fact model_graded_qa Task vitals_bind vitals_bundle vitals_log_dir vitals_log_dir_set vitals_log_read vitals_view

Dependencies:askpass cli coro cpp11 curl dplyr ellmer fastmap generics glue httpuv httr2 jsonlite later lifecycle magrittr openssl otel pillar pkgconfig promises purrr R6 Rcpp rlang S7 stringi stringr sys tibble tidyr tidyselect utf8 vctrs withr

Citation

Development and contributors

Readme and manuals

Help Manual

Help page	Topics
An R Eval	are
Convert a chat to a solver function	generate
Convert a chat to a solver function with structured output	generate_structured
Scoring with string detection	detect_answer detect_exact detect_includes detect_match detect_pattern scorer_detect
Model-based scoring	model_graded_fact model_graded_qa scorer_model
Creating and evaluating tasks	Task
Concatenate task samples for analysis	vitals_bind
Prepare logs for deployment	vitals_bundle
The log directory	vitals_log_dir vitals_log_dir_set
Read an eval log back into ellmer chats	vitals_log_read
Interactively view local evaluation logs	vitals_view