Rating and Evaluating Texts Using Large Language Models
# production version
install.packages("chatRater")
# developmental version
::install_github("ShiyangZheng/chatRater") remotes
::install_github("ShiyangZheng/chatRater")
remotes
library(tidyverse)
library(chatRater)
<- 'bare your soul'
stim <- list('buy the farm', 'beat the clock')
stim_list
<-'gpt-4' # or 'deepseek-chat'
model <- 'You are a native English speaker.'
prompt <- 'A list of idioms is given below.
question To what extent do you agree with the following statement:
The figurative meaning of this idiom had a lot in common with its literal meaning.
Please rate according to the 5-point scale explained below.
1 = Completely disagree;
3 = Neither agree nor disagree;
5 = Fully agree.
Please limit your answer to numbers.'
<- 1
top_p <- 0
temp <- 5
n_iterations <- ""
api_key
set.seed(56475764)
<- generate_ratings(model, stim, prompt, question, top_p, temp, n_iterations, api_key)
res <- generate_ratings_for_all(model, stim_list, prompt, question, top_p, temp, n_iterations, api_key)
res1
# write the results in a CSV file
write.csv(res, "idiom_ratings_3.csv", row.names = FALSE)
write.csv(res1, "idiom_ratings_4.csv", row.names = FALSE)
You can also experience all the functionalities of chatRater in an interactive Rshiny app.
To cite package ‘chatRater’ in publications use:
Zheng, S. (2025). chatRater: A Tool for Rating Text Using Large Language Models (Version 1.0.0) [R package]. Retrieved from https://github.com/ShiyangZheng/chatRater
A BibTeX entry for LaTeX users is
@Manual{, title = {chatRater: A Tool for Rating Text Using Large Language Models}, author = {Shiyang Zheng}, year = {2025}, note = {R package version 1.0.0}, }