Comparison with other packages • intraclass

library(intraclass)

If you already use another R package for intraclass correlations, two questions matter before switching: does intraclass agree with the tool I trust on the problems that tool handles? and what does it do that my current tool cannot? This article answers both, on the package’s own shipped datasets, with every number computed live as the page builds. Any unfamiliar term is defined in the Glossary.

The comparison packages are psych (Revelle’s psych::ICC, the most widely used ANOVA ICC in R), irr (irr::icc, a classical inter-rater-reliability toolkit), and irrICC (Gwet’s model-based ICCs). All three are optional: the code chunks below only run when the package is installed.

Does it agree? (validation)

On a balanced design, one where every subject is rated by every rater, the whole ICC family is defined for all of these tools. So we can line them up coefficient by coefficient. The ratings dataset is six subjects each scored by the same four raters.

intraclass estimates the coefficients from variance components fitted by REML (a mixed model), whereas psych and irr derive them from classical ANOVA mean squares. Those are different computational routes to the same population quantity, and they are known to converge to each other. The table shows how close they land here:

wm <- to_wide(ratings)
# Scalar `type` and `unit` make this a one-row fit, so `[1]` is that row --
# the coefficient asked for. Elsewhere, select by `term`.
ic <- function(model, type, unit) {
  tidy(icc(ratings, subject = subject, rater = rater, score = score,
           model = model, type = type, unit = unit))$estimate[1]
}
ps <- psych::ICC(wm)$results
psv <- stats::setNames(ps$ICC, ps$type)

rows <- list(
  c("ICC(1)",   "oneway", "agreement",   "single",  "ICC1"),
  c("ICC(1,k)", "oneway", "agreement",   "average", "ICC1k"),
  c("ICC(A,1)", "twoway", "agreement",   "single",  "ICC2"),
  c("ICC(A,k)", "twoway", "agreement",   "average", "ICC2k"),
  c("ICC(C,1)", "twoway", "consistency", "single",  "ICC3"),
  c("ICC(C,k)", "twoway", "consistency", "average", "ICC3k")
)

comparison <- do.call(rbind, lapply(rows, function(r) {
  data.frame(
    coefficient = r[1],
    intraclass  = ic(r[2], r[3], r[4]),
    psych       = unname(psv[r[5]]),
    irr         = irr::icc(wm, model = r[2], type = r[3], unit = r[4])$value
  )
}))

knitr::kable(comparison, digits = 5, row.names = FALSE)

coefficient	intraclass	psych	irr
ICC(1)	0.16574	0.16574	0.16574
ICC(1,k)	0.44280	0.44280	0.44280
ICC(A,1)	0.28977	0.28976	0.28976
ICC(A,k)	0.62006	0.62005	0.62005
ICC(C,1)	0.71484	0.71484	0.71484
ICC(C,k)	0.90932	0.90932	0.90932

max_gap <- max(abs(comparison$intraclass - comparison$psych),
               abs(comparison$intraclass - comparison$irr))

Every coefficient matches to five decimal places. The largest disagreement anywhere in the table is 7.2e-06. That residual is not error in either tool. It is the small-sample gap between a REML fit and ANOVA mean squares, which vanishes as the sample grows. On the designs classical tools handle, you lose nothing by using intraclass, and the psych agreement is in fact checked on every test run of this package.

A model-based tool from a different lineage agrees too. irrICC implements Gwet’s ICCs, estimated by a moment method rather than either REML or ANOVA. Its two-way random agreement coefficient (icc2r) reproduces intraclass’s ICC(A,1):

w <- reshape(ratings, idvar = "subject", timevar = "rater", direction = "wide")
w <- w[order(as.integer(as.character(w$subject))), ]
gwet_frame <- data.frame(
  Target = as.integer(as.character(w$subject)),
  J1 = w$score.1, J2 = w$score.2, J3 = w$score.3, J4 = w$score.4
)
gwet_agree <- irrICC::icc2.inter.fn(gwet_frame)$icc2r
intraclass_a1 <- with(tidy(icc(ratings, subject = subject, rater = rater, score = score,
                              model = "twoway", type = "agreement",
                              unit = "single")), estimate[term == "ICC(A,1)"])

data.frame(
  source   = c("intraclass ICC(A,1)", "irrICC icc2r (Gwet)"),
  estimate = c(intraclass_a1, gwet_agree)
)
#>                source  estimate
#> 1 intraclass ICC(A,1) 0.2897700
#> 2 irrICC icc2r (Gwet) 0.2897638

What does it add? (differentiation)

The classical tools were built for the balanced, complete case. Real rating data are rarely so tidy, and that is where the packages diverge.

Incomplete and unbalanced data

The ratings_incomplete dataset is the same study with four ratings missing. In particular, the second rater scored only two of the six subjects:

wide_incomplete <- reshape(ratings_incomplete, idvar = "subject",
                           timevar = "rater", direction = "wide")
wide_incomplete <- wide_incomplete[order(as.integer(as.character(wide_incomplete$subject))), ]
colnames(wide_incomplete) <- c("subject", paste0("rater", 1:4))
knitr::kable(wide_incomplete, row.names = FALSE)

subject	rater1	rater2	rater3	rater4
1	9	2	5	8
2	6	1	3	2
3	8	NA	6	8
4	7	NA	2	6
5	10	NA	6	9
6	6	NA	4	7

A classical ANOVA ICC needs a complete rectangle, so psych and irr listwise-delete any subject with a missing cell. Here that discards the four subjects rater 2 skipped, leaving only two:

wm_inc <- to_wide(ratings_incomplete)
surviving <- sum(stats::complete.cases(wm_inc))
c(observed_cells = nrow(ratings_incomplete),
  possible_cells = nrow(ratings),
  subjects_after_listwise_deletion = surviving)
#>                   observed_cells                   possible_cells 
#>                               20                               24 
#> subjects_after_listwise_deletion 
#>                                2

An ICC computed from two subjects is not usable, whatever its value. intraclass instead fits the mixed model to every observed rating and reports an effective number of ratings (k_eff) that accounts for the imbalance:

fit_inc <- icc(ratings_incomplete, subject = subject, rater = rater, score = score,
               model = "twoway", type = "agreement", unit = "average")
gl_inc <- glance(fit_inc)
c(estimate = with(tidy(fit_inc), estimate[term == "ICC(A,k)"]),
  subjects_used = gl_inc$n_subjects,
  ratings_used = gl_inc$n_obs,
  k_eff = gl_inc$k_eff)
#>      estimate subjects_used  ratings_used         k_eff 
#>     0.5205561     6.0000000    20.0000000     3.2727273

All six subjects and all twenty observed ratings contribute, and nothing is thrown away. irrICC can also fit incomplete data with its own model, as the capability matrix below shows, but the mean-squares tools cannot.

The bigger picture

Agreement on balanced data and graceful handling of missing data are two entries in a wider gap. The table below summarizes what each package computes. It is a map of intent, not a scorecard: each tool is excellent at what it was designed for.

Capability	`psych`	`irr`	`irrICC`	`intraclass`
Balanced ANOVA ICC family	✅	✅	✅	✅
Incomplete / unbalanced data	no	no	✅	✅
Multilevel (subject and cluster) IRR	no	no	no	✅
Boundary-aware interval	no	no	partial	✅
Fixed vs. random rater framing	partial	partial	no	✅
Guidance on which ICC to report	no	no	no	✅

Two rows deserve a word. Model-based extractors such as performance::icc return variance components or a variance-partition coefficient. That is the raw material of an ICC, but not the inter-rater-reliability coefficient family itself, nor the error-variance framing that distinguishes agreement from consistency. intraclass’s own generalizability coefficients were validated against gtheory, agreeing to within 0.001. gtheory is a generalizability-theory package archived from CRAN in March 2025, and is not a dependency here. Those committed reference values live in the package’s reference notes. And an interval that is boundary-aware is something none of the classical tools provide. Such an interval behaves correctly when a variance component is estimated at its zero boundary, where a normal-approximation interval silently misbehaves.

intraclass earns its extra machinery on exactly these cases. For the details of each, see the companion articles:

Choosing an ICC: the selection framework the last matrix row points to.
Multilevel designs: subject- and cluster-level reliability when raters are nested.
Interval methods: the boundary-aware Monte-Carlo and bootstrap intervals.
Estimation engines: the mixed-model, SEM, and Bayesian backends behind these numbers.

When to use which

If your design is balanced and complete and you only need the classic McGraw–Wong coefficients, psych and irr are mature, familiar, and, as the table above shows, numerically identical to intraclass. Reach for intraclass when your data are incomplete or unbalanced, or when raters are nested in clusters. Reach for it too when you need an interval you can trust near the boundary, or when you want the package to help you choose and justify the coefficient in the first place.