Multilevel Modeling

Effect Sizes & Reporting

Spring 2026 | CLAS | PSYC 894
Jeffrey M. Girard | Lecture 14a

Roadmap

1. Effect Sizes
   • Marginal vs. Conditional R²
   • StdX and StdYX Effects
2. The Reporting Checklist
   • Intro and Methods
   • Results and Discussion

Effect Sizes in MLMs

Beyond p-Values

• Statistical significance tells us if an effect exists. It does not tell us if the effect is meaningful.
• Effect sizes contextualize the magnitude of results from your model.
• They answer the question: “What is the practical significance of this result?”
• Reviewers and journal editors increasingly demand effect sizes for all key findings.

Two Types of Effect Sizes

Unstandardized Effect Sizes

• Expressed in the raw units of the dependent variable
• e.g., For every extra year of service, salary increases by $1000
• Useful when the outcome variable has an intuitive metric

Standardized Effect Sizes

• Measure magnitude without units
• e.g., The model explains 24 percent of the variance in patient satisfaction
• Useful when the original metrics are abstract

The R-squared Metric

• The most common standardized effect size is R²
• It represents the proportion of variance in the outcome variable explained by the model and all its included effects
• In single-level regression, the math is straightforward: \[R^2 = \frac{\text{Explained Variance}}{\text{Total Variance}}\]
• It yields an intuitive measure ranging from:
  0 (nothing explained) to 1 (everything explained)
• However, applying this to Multilevel Models is notoriously difficult

The Multilevel Variance Problem

Why can’t we just use a single R² for an MLM?

• Multiple Denominators:
  Total variance is partitioned. Do we want to explain total outcome variance, within-cluster variance, or between-cluster variance?
• Multiple Numerators:
  We have multiple sources of explained variance. We have fixed effects at Level-1, fixed effects at Level-2, random intercepts, and random slopes.
• Summing these up into a single neat number obscures the complexity of the data structure and can be misleading.

The Rights & Sterba Framework

• Rights & Sterba (2019) developed a complex framework for R² in MLMs.
• Because there are so many combinations of numerators and denominators, their framework yields 12 different R-squared measures for a single model.
• For example:
  • Proportion of total variance explained by Level-1 fixed effects.
  • Proportion of within-cluster variance explained by random slopes.
  • Proportion of between-cluster variance explained by random intercepts.

Practical Constraints

• This framework is academically rigorous and statistically exact.
• If your research question relies on isolating exactly how much Level-2 variance is explained by a Level-1 random slope, it is the tool to use.
• However, for most applied researchers, reporting 12 effect sizes per model is overwhelming for both the author and the reader.
• Most journals expect a simpler, two-part summary of model performance.

The Nakagawa Framework

• The widely accepted standard for applied MLMs is the framework developed by Nakagawa & Schielzeth (2013).
• It simplifies the problem by providing two metrics:
  • Marginal R²: Variance explained by only the fixed effects.
  • Conditional R²: Variance explained by the entire model
    (fixed effects plus random effects).
• This is what we will usually report in practice.

Calculating
R-squared

Setup and Data

library(tidyverse)
library(glmmTMB)
library(easystats)

hospital <- read_csv("hospital_sim.csv")
glimpse(hospital)

Rows: 2,000
Columns: 6
$ satisfaction  <dbl> -0.30518145, 0.18461383, 1.28090155, 0.35863181, 0.…
$ wait_time     <dbl> 0.35091220, -0.53187051, -0.36259725, -1.04860229, …
$ visit_type    <chr> "Emergency", "Emergency", "Routine", "Routine", "Em…
$ hospital      <chr> "H01", "H01", "H01", "H01", "H01", "H01", "H01", "H…
$ nurse_ratio   <dbl> -0.6830383, -0.6830383, -0.6830383, -0.6830383, -0.…
$ hospital_type <chr> "Non-Teaching", "Non-Teaching", "Non-Teaching", "No…

Preparing the Data

To calculate accurate R-squared values, we do not need to standardize or center our Level-2 predictors. However, we must decompose our Level-1 predictors to properly isolate the within-cluster and between-cluster variance.

l2_means <- hospital |> 
  summarize(.by = hospital, wait_b = mean(wait_time))

dat_unstd <- hospital |> 
  left_join(l2_means, by = "hospital") |> 
  mutate(wait_w = wait_time - wait_b)

Fitting the Baseline Model

Now we fit our model using our unstandardized, centered predictors.

fit_unstd <- glmmTMB(
  formula = satisfaction ~ 1 + wait_b + wait_w + nurse_ratio + 
    (1 + wait_w | hospital), 
  data = dat_unstd,
  REML = TRUE
)

Extracting R-squared

We can extract our Nakagawa R² values cleanly using the {performance} package from the {easystats} ecosystem.

r2(fit_unstd)

# R2 for Mixed Models

  Conditional R2: 0.642
     Marginal R2: 0.176

Interpreting R-squared

• Marginal R² (0.176): Fixed effects explain ~17.6% of the variance.
  • Prediction Framing: How well we predict a new observation from a new, unseen cluster. We must rely solely on the population averages.
• Conditional R² (0.642): The entire model (fixed + random) explains ~64.2% of the variance.
  • Prediction Framing: How well we predict a new observation from an existing, known cluster. We can fine-tune the prediction using that cluster’s random effects.
• The Difference: The random effects alone account for roughly 46.6% of the variance (Conditional minus Marginal).

Standardized Effects

To Standardize Y or Not?

Before we standardize our slopes, we must ask a theoretical question about our outcome variable (Y).

• Meaningful Outcomes: If Y has a well understood and intuitive metric (e.g., dollars, days, blood pressure), we leave it in its raw units. We only standardize the predictors, yielding StdX slopes.
• Arbitrary Outcomes: If Y is an abstract scale (e.g., a satisfaction survey, an anxiety inventory), raw points are less intuitive. We standardize the predictors AND the outcome, yielding StdYX slopes.

The Predictor Trap

Regardless of using StdX or StdYX, we must be incredibly careful about how we standardize our multilevel predictors (X).

• The “Global” Trap: Standard software defaults to standardizing based on the total sample’s mean and SD. Here is why that fails:
  • Mixing Levels: A Level-1 predictor can only explain within-cluster variance. Dividing its slope by the total SD (which includes between-cluster differences) unfairly penalizes it, making strong local effects look artificially weak.
  • Cluster Weighting: Level-2 predictors are attributes of the cluster. If you standardize across all data rows, a cluster with 500 observations influences the math 10x more than a cluster with 50 observations, distorting the meaning of a “standard unit” for that variable.

StdX: Data Preparation (L2)

To avoid mixing levels, we create a distinct Level-2 dataset using summarize. Standardizing within this dataset uses the correct means and SDs.

# 1. Create a distinct Level-2 dataset
l2_std <- hospital |> 
  summarize(
    wait_b = mean(wait_time),
    nurse_ratio = first(nurse_ratio),
    .by = hospital
  ) |> 
  mutate(
    # 2. Standardize across the Level-2 clusters
    wait_b_z = standardize(wait_b),
    nurse_z = standardize(nurse_ratio)
  )

StdX: Data Preparation (L1)

Next, we join those standardized L2 variables back to our L1 data and strictly standardize the within-cluster variance.

dat_stdx <- hospital |> 
  # 3. Join the Level-2 data back to the main dataset
  left_join(l2_std, by = "hospital") |> 
  mutate(
    # 4. Isolate and standardize the within-cluster variance
    wait_w = wait_time - wait_b,
    wait_w_z = standardize(wait_w)
  )

StdX: Model & Interpretation

fit_stdx <- glmmTMB(
  formula = satisfaction ~ 1 + wait_b_z + wait_w_z + nurse_z + 
    (1 + wait_w_z | hospital), 
  data = dat_stdx,
  REML = TRUE
)
model_parameters(fit_stdx, effects = "fixed")

# Fixed Effects

Parameter   | Coefficient |   SE |         95% CI |         z |      p
----------------------------------------------------------------------
(Intercept) |   -6.20e-17 | 0.11 | [-0.21,  0.21] | -5.90e-16 | > .999
wait b z    |        0.02 | 0.11 | [-0.19,  0.23] |      0.18 | 0.860 
wait w z    |       -0.37 | 0.03 | [-0.43, -0.31] |    -11.90 | < .001
nurse z     |        0.19 | 0.14 | [-0.08,  0.47] |      1.41 | 0.158 

• A one SD increase in a patient’s wait time (relative to their hospital’s average) significantly decreases their satisfaction by 0.37 raw points.

The Problem with Global StdYX

If Y is an arbitrary scale, you might be tempted to standardize Y globally before fitting your model (like you would for OLS). Do not do this for multilevel data!

• The ICC Penalty: If you standardize Y globally, the denominator includes both Level-1 and Level-2 variance.
• The Trap: A Level-1 predictor can only explain Level-1 variance. By dividing its effect by the total variance pool, you mathematically penalize it.
• The Result: If your model has high Level-2 variance (high ICC), your Level-1 StdYX effects will look artificially tiny.
• The Solution: We must use Level-Specific Standardization.

Level-Specific StdYX

There is no automated R package for this (yet). We manually take the coefficients from our StdX model (where X is standardized but Y is raw) and divide them by the level-specific standard deviations of Y.

# 1. Calculate the standard deviation of the hospital means (Level 2 SD)
sd_y_b <- hospital |> 
  summarize(sat_b = mean(satisfaction), .by = hospital) |> 
  pull(sat_b) |> sd() |> print()

[1] 0.7003727

# 2. Calculate the standard deviation of the within-cluster scores (Level 1 SD)
sd_y_w <- hospital |> 
  mutate(sat_b = mean(satisfaction), .by = hospital) |> 
  mutate(sat_w = satisfaction - sat_b) |> 
  pull(sat_w) |> sd() |> print()

[1] 0.7221508

Calculating & Interpreting StdYX

We divide the slopes from our fit_stdx model by the matching SD of Y.

# L2 StdYX Slope = (L2 StdX Slope) / (L2 SD of Y)
0.19 / 0.700 # nurse_z

[1] 0.2714286

• A one between-hospital SD increase in nurse ratio was associated with a non-significant increase in satisfaction of 0.27 between-hospital SDs.

# L1 StdYX Slope = (L1 StdX Slope) / (L1 SD of Y)
-0.37 / 0.722 # wait_w_z

[1] -0.5124654

• A one within-hospital SD increase in wait time was associated with a significant decrease in satisfaction of 0.51 within-hospital SDs.

The Reporting
Checklist

The Goal of Reporting

• Multilevel modeling involves dozens of minor analytical decisions.
• Centering, random effect structures, and estimation algorithms all drastically change your results.
• Your goal in a manuscript is not just to report the p-values, but to provide enough transparent detail that another researcher could perfectly replicate your model from your raw data.

IMRD: Introduction

The justification for an MLM should begin before the Methods section.

What to include:

• Acknowledge the nested structure of your data in your hypotheses.
• Clearly state the levels of analysis (e.g., L1 is patients, L2 is hospitals).
• Distinguish between within-cluster and between-cluster hypotheses. Do not hypothesize a generic “effect of wait time” if you specifically mean “when a patient waits longer than their hospital’s average.”

IMRD: Methods (Data)

The participants section must describe the sample sizes at all levels.

What to include:

• Total number of Level-1 observations (e.g., N = 2,000 patients).
• Total number of Level-2 clusters (e.g., J = 40 hospitals).
• The average cluster size, minimum cluster size, and maximum cluster size (e.g., an average of 50 patients per hospital, ranging from 12 to 89).

IMRD: Methods (Variables)

You must explicitly state how every variable was handled.

What to include:

• Identify which variables were measured at Level-1 and Level-2.
• Explicitly state your centering decisions. Did you grand-mean center? Did you group-mean center (cluster-mean center)?
• State why you chose that centering method. Did you want to isolate the purely within-cluster effect, or were you aiming for a blended effect?

IMRD: Methods (Modeling)

Clearly identify the mathematical tools you used.

What to include:

• Software and packages with version numbers (e.g., R version 4.3.1, glmmTMB version 1.1.8).
• The estimation method used. Did you use Restricted Maximum Likelihood (REML) or standard Maximum Likelihood (ML)?
• Mention the specific optimizer if you had to change it from the default to achieve convergence.

IMRD: Methods (Structure)

Explain how you built your equations.

What to include:

• Describe the random effects structure. Did you include random intercepts? Random slopes?
• State whether you allowed the random intercepts and slopes to correlate.
• If you had to simplify your random effects structure due to singular fits or convergence issues, document that process here. Explain what was dropped and why.

IMRD: Methods (Reliability)

You cannot simply present a correlation matrix for a multilevel dataset.

What to include:

• A table of means, standard deviations, and correlations for all Level-1 variables.
• A completely separate table of descriptives and correlations for all Level-2 variables, calculated at the cluster level.
• Evidence of reliability for your continuous predictors. Predictors measured with high error will bias your regression coefficients.

IMRD: Methods (Missing Data)

Missing data is especially dangerous in MLMs. Deleting one Level-2 cluster due to a missing covariate takes all of its Level-1 observations down with it.

What to include:

• The exact percentage of missing data reported separately for Level-1 variables and Level-2 variables.
• The method used to handle missingness (e.g., Full Information Maximum Likelihood or Multiple Imputation).
• If using listwise deletion, explicitly state the final analytical sample size and justify why deletion was acceptable.

IMRD: Results (The Null Model)

Always report the baseline before adding predictors.

What to include:

• Report the unconditional ICC from the null model.
• Provide a plain language interpretation: “The ICC was 0.22, indicating that 22 percent of the variance in patient satisfaction is attributable to differences between hospitals.”
• This justifies the need for multilevel modeling in the first place.

IMRD: Results (Fixed Effects)

Report the focal tests of your hypotheses.

What to include:

• The unstandardized coefficient estimate.
• The 95% Confidence Interval for the estimate.
• The standard error.
• The exact p-value.
• A plain language interpretation of the direction and magnitude of the effect.

IMRD: Results (Random Effects)

Do not ignore the random portion of the model.

What to include:

• The variance (or standard deviation) of the random intercepts.
• The variance (or standard deviation) of any random slopes.
• The correlation between random intercepts and random slopes.
• The residual (Level-1) variance.

IMRD: Results (Effect Sizes)

Provide the big picture summary of model performance.

What to include:

• Report both the Marginal R2 and Conditional R2.
• Cite Nakagawa & Schielzeth (2013) to clarify your framework.
• If comparing multiple models (e.g., a baseline model vs a full model), report the change in R2 (Delta R2) or use AIC/BIC to discuss relative model fit.
• Clearly state whether you are reporting raw, StdX, or StdYX slopes.

IMRD: Results (Model Building)

Reviewers want to see the sequence of models you estimated.

What to include:

• Step 1: The Null Model to establish baseline variance and the ICC.
• Step 2: The Random Coefficients Model containing only Level-1 predictors to see if those slopes vary significantly across clusters.
• Step 3: The Full Contextual Model containing Level-1 and Level-2 predictors, including any cross-level interactions.

IMRD: Discussion (Language)

Multilevel modeling uses the word “effect” constantly (e.g., fixed effects, random effects). This can create a semantic trap.

What to watch out for:

• Do not let the statistical terminology bleed into causal claims in your discussion section.
• Unless you have a rigorously controlled experimental design, your “fixed effects” are merely associations or predictions.
• Carefully edit your conclusions to ensure you are not implying a causal mechanism that your design cannot support.