Recommend an ICC and the call that computes it

choose_icc() walks the decision tree of the Choosing an ICC vignette and returns a recommendation object naming the coefficient(s) to report and the exact icc() call that computes them. It does not fit a model: there is no data argument. Copy the emitted call and run it on your data.

Usage

choose_icc(
  model = NULL,
  type = NULL,
  unit = NULL,
  raters = NULL,
  multilevel = NULL,
  level = NULL
)

# S3 method for class 'icc_recommendation'
format(x, ...)

# S3 method for class 'icc_recommendation'
print(x, ...)

Arguments

model: "twoway" (crossed: the same raters judge every subject) or "oneway" (raters are interchangeable across subjects). Defaults to "twoway". Under "oneway" the type and raters choices do not exist (there is no rater term), and supplying them is an error.
type: "agreement" (the value itself must match, so systematic rater offsets count as error), "consistency" (only rank order matters, so a constant per-rater offset is forgiven), or "both". Required for a two-way design.
unit: "single" (you will act on one rater's score), "average" (the mean of your raters), or "both". Required.
raters: "random" (a sample you generalize beyond, and the recommended default for interrater reliability) or "fixed" (exactly these judges, no generalization). Required for a two-way design.
multilevel: TRUE if subjects are nested in higher-level clusters (pupils in classrooms, patients in clinics), else FALSE (the default).
level: For a multilevel design, "subject" (within-cluster reliability), "cluster" (between-cluster reliability), or "both". Required when multilevel = TRUE.
x: An icc_recommendation object.
...: Unused, for method consistency.

Value

choose_icc() returns an icc_recommendation object (a list). It carries the recommended coefficient rows ($rows), the exact icc() call as a string ($call), the per-decision rationale ($rationale), and any notes ($notes).

The methods documented on this page return:

format.icc_recommendation(): a character vector holding the printed recommendation, one line per element.
print.icc_recommendation(): the icc_recommendation object invisibly, having emitted that recommendation.

Details

Supply the decisions as arguments to get advice programmatically. In an interactive session, call choose_icc() with the relevant answers omitted to be asked the outstanding questions one at a time.

The two structural facts about your design default to the common case, a crossed, non-multilevel two-way design, matching icc(). They are whether the raters are crossed (model) and whether subjects are nested in clusters (multilevel). The choices that actually select the coefficient are type, unit, raters, and level when multilevel. None of them has a silent default. In a non-interactive session, leaving one unanswered is an error naming the unanswered decision, rather than quietly picking one for you.

Examples

# Two-way absolute agreement, single rater, random raters (Shrout & Fleiss
# ICC(2,1)): the two structural defaults (crossed, non-multilevel) are implied.
choose_icc(type = "agreement", unit = "single", raters = "random")
#> ── Recommended ICC ─────────────────────────────────────────────────────────────
#> Design: two-way random, absolute agreement
#> 
#> Recommendation: ICC(A,1)
#> Shrout & Fleiss equivalent: ICC(A,1) = ICC(2,1)
#> 
#> Why:
#>   - Crossed (two-way): the same raters judge every subject.
#>   - Absolute agreement: the value itself must match; a systematic difference between raters counts as error.
#>   - Single rater: you will act on one rater's score.
#>   - Random raters: a sample you generalize beyond, to the rater universe they were drawn from.
#> 
#> Run this on your data:
#>   icc(data, score, subject, rater, type = "agreement", unit = "single")
#> 
#> Notes:
#>   - Complete vs. incomplete is automatic: icc() uses whatever ratings are present and projects ICC(*,k) to the effective number of ratings (k_eff). The design must stay connected, or icc() fails loudly.

# Consistency of the average of fixed raters -- McGraw & Wong ICC(C,k):
choose_icc(type = "consistency", unit = "average", raters = "fixed")
#> ── Recommended ICC ─────────────────────────────────────────────────────────────
#> Design: two-way mixed, consistency
#> 
#> Recommendation: ICC(C,k)
#> Shrout & Fleiss equivalent: ICC(C,k) = ICC(3,k)
#> 
#> Why:
#>   - Crossed (two-way): the same raters judge every subject.
#>   - Consistency: only the rank order must match; a constant per-rater offset is forgiven.
#>   - Average: you will act on the mean of your raters.
#>   - Fixed raters: exactly these judges; the coefficient does not generalize past them.
#> 
#> Run this on your data:
#>   icc(data, score, subject, rater, type = "consistency", raters = "fixed", unit = "average")
#> 
#> Notes:
#>   - Random raters is the recommended default for interrater reliability; use fixed only when these are the entire population of raters you will ever use.
#>   - Complete vs. incomplete is automatic: icc() uses whatever ratings are present and projects ICC(*,k) to the effective number of ratings (k_eff). The design must stay connected, or icc() fails loudly.

# Both error definitions side by side: the emitted call leaves `type` out,
# because icc() reports agreement and consistency by default.
choose_icc(type = "both", unit = "single", raters = "random")
#> ── Recommended ICC ─────────────────────────────────────────────────────────────
#> Design: two-way random, absolute agreement & consistency
#> 
#> Recommendation: ICC(A,1), ICC(C,1)
#> Shrout & Fleiss equivalent: ICC(A,1) = ICC(2,1)
#> 
#> Why:
#>   - Crossed (two-way): the same raters judge every subject.
#>   - Both error definitions: absolute agreement (the value itself must match) and consistency (a constant per-rater offset is forgiven) side by side.
#>   - Single rater: you will act on one rater's score.
#>   - Random raters: a sample you generalize beyond, to the rater universe they were drawn from.
#> 
#> Run this on your data:
#>   icc(data, score, subject, rater, unit = "single")
#> 
#> Notes:
#>   - The emitted call omits type on purpose: icc() reports both error definitions by default, so leaving the argument out is what asks for the pair.
#>   - Complete vs. incomplete is automatic: icc() uses whatever ratings are present and projects ICC(*,k) to the effective number of ratings (k_eff). The design must stay connected, or icc() fails loudly.

# A one-way design (interchangeable raters): type/raters do not apply.
choose_icc(model = "oneway", unit = "single")
#> ── Recommended ICC ─────────────────────────────────────────────────────────────
#> Design: one-way random
#> 
#> Recommendation: ICC(1)
#> Shrout & Fleiss equivalent: ICC(1) = ICC(1,1)
#> 
#> Why:
#>   - One-way: raters are interchangeable across subjects, so systematic rater differences are absorbed into error -- the most conservative ICC.
#>   - Single rater: you will act on one rater's score.
#> 
#> Run this on your data:
#>   icc(data, score, subject, rater, model = "oneway", unit = "single")
#> 
#> Notes:
#>   - Complete vs. incomplete is automatic: icc() uses whatever ratings are present and projects ICC(*,k) to the effective number of ratings (k_eff). The design must stay connected, or icc() fails loudly.

# A multilevel design, both levels:
choose_icc(type = "agreement", unit = "single", raters = "random",
  multilevel = TRUE, level = "both")
#> ── Recommended ICC ─────────────────────────────────────────────────────────────
#> Design: multilevel, two-way random, absolute agreement
#> 
#> Recommendation:
#>   subject: ICC(A,1)
#>   cluster: ICC(A,1)
#> 
#> Why:
#>   - Crossed (two-way): the same raters judge every subject.
#>   - Absolute agreement: the value itself must match; a systematic difference between raters counts as error.
#>   - Single rater: you will act on one rater's score.
#>   - Random raters: a sample you generalize beyond, to the rater universe they were drawn from.
#>   - Both levels: within-cluster (subject) and between-cluster (cluster) reliability side by side.
#> 
#> Run this on your data:
#>   icc(data, score, subject, rater, cluster, type = "agreement", unit = "single")
#> 
#> Notes:
#>   - Complete vs. incomplete is automatic: icc() uses whatever ratings are present and projects ICC(*,k) to the effective number of ratings (k_eff). The design must stay connected, or icc() fails loudly.
#>   - See vignette("multilevel-designs") for a worked multilevel example.