Split-half factor comparability

Measures how well each factor at each level of a bass-ackwards hierarchy replicates across random split-halves of the sample. A factor is a summary variable standing in for a group of items that move together. In a split-half analysis the sample is split into two random halves and each half is analysed separately. The coefficients are Everett's (1983) factor comparability coefficients. They revive the split-half replication gate of the research program that produced the bass-ackwards method. Saucier (1997) screened factor solutions by their split-half stability, and Saucier, Georgiades, Tsaousis, and Goldberg (2005) chose the optimal hierarchical level by requiring split-half replication above a .90 threshold. This is also the direct instrument for the overextraction caution in suggest_k(), because non-replicable structure concentrates in the deeper levels of an overextracted hierarchy (Forbes, 2023).

Usage

comparability(
  data,
  k_max,
  engine = "pca",
  cor = "pearson",
  fm = "minres",
  n_splits = 10L,
  seed = NULL,
  ...
)

Arguments

data: A data frame or numeric matrix of observed variables (items in columns, observations in rows). Raw data only, because splitting needs rows, so a correlation matrix is not accepted. Missing values are handled pairwise throughout (as in ackwards()'s default).
k_max: Maximum number of factors or components to evaluate. Normally this is the same value (or one or two above it) you intend to pass to ackwards(). Required.
engine: Extraction engine: "pca" (default) or "efa". "esem" is not yet supported here, because fitting 2 * n_splits lavaan hierarchies needs its own performance treatment. For ESEM (exploratory structural equation modeling) workflows, run comparability() with engine = "efa" as a structural screen.
cor: Correlation basis: "pearson" (default) or "spearman". As with suggest_k(), "polychoric" is not supported, because estimating polychoric matrices in every half-sample is slow and unstable. Users analysing ordinal items (a few ordered categories, such as a 1 to 5 rating) should screen replicability on the Pearson basis. They should then fit the final model with cor = "polychoric" in ackwards(), which estimates the correlation between the continuous traits assumed to underlie the ordered responses.
fm: Factor extraction method passed to psych::fa(). It is only used when engine = "efa". One of "minres" (default), "ml", or "pa".
n_splits: Number of random split-half replicates. Default 10L. The published precedents used a single split (Saucier et al., 2005). Repeating the split guards against the luck of one draw, so the coefficients are summarised over replicates. Each replicate fits 2 * k_max solutions, so the default costs 20 hierarchy fits. PCA (principal component analysis) and EFA (exploratory factor analysis) are fast enough that this is typically a few seconds.
seed: Integer seed for reproducible splits. NULL (default) uses the current RNG state.
...: Reserved for future arguments.

Value

An object of class "comparability". Print it for a per-level summary, or call autoplot() on it for a diagnostic plot. The list contains:

coefficients: Data frame with one row per split x level x factor. Its columns are split, level, factor (full-sample m{k}f{j} label), r (score comparability), and phi (Tucker's congruence of the matched loading columns). A value is NA when the level did not converge in one of the halves.
summary: Data frame with one row per level x factor. Its columns are level, factor, r_median, r_min, phi_median, phi_min (across splits), and n_splits_ok (splits in which both halves converged).
k_max: Deepest level evaluated. It can be lower than the k_max you asked for when the full-sample fit truncated, that is, when deep levels did not converge. The original request is kept in k_requested.
k_requested, n_splits, n_half, engine, cor, fm, n_obs, n_vars, seed: Metadata.

Details

For each of n_splits random half-splits, solutions at every level from 1 to k_max are fit independently in each half. Each half-solution's factors are matched to the full-sample solution's factors, so coefficients are reported under the same m{k}f{j} labels you get from ackwards(). The comparability coefficient for a factor is the correlation between its two matched half-solution scores. It is computed on the pooled correlation matrix via the same W'RW algebra used for between-level edges, applying both halves' scoring weights to the full sample, exactly Everett's procedure. Tucker's congruence coefficient (phi) between the matched half-solution loading columns is reported alongside. A loading is the correlation between an item and a factor, and congruence is a 0 to 1 index of how similar two loading patterns are. So comparability asks whether the two halves' scores agree, and phi asks whether their loading patterns agree.

comparability() always uses the varimax rotation, the ackwards() default, for the full sample and for every half. It has no rotation argument, so its coefficients describe the varimax hierarchy even when you fit ackwards() with an oblique rotation.

Interpreting the output

Coefficients near 1 mean the factor re-emerges in independent half-samples. A factor whose comparability is low is sample-idiosyncratic and should not anchor substantive interpretation. Two benchmarks are conventional. The first, .90, is the split-half replication threshold of Goldberg's lexical research program (Saucier et al., 2005). It follows from Everett's (1983) rationale, under which split-half factors share at least 81% of their variance. The second, .95, is the stricter bound at which two factors are conventionally treated as interchangeable (Lorenzo-Seva & ten Berge, 2006). These are conventions, not tests, so comparability() reports every coefficient and flags nothing. The deepest level at which all factors replicate is a natural hierarchy floor for ackwards()'s k_max. See vignette("ackwards-girard") for the full workflow.

A level that fails to converge in a half-sample yields NA coefficients for that split, because convergence is data, not an error. The number of usable splits per factor is reported in summary$n_splits_ok and a message summarises any shortfall.

References

Everett, J. E. (1983). Factor comparability as a means of determining the number of factors and their rotation. Multivariate Behavioral Research, 18(2), 197–218. doi:10.1207/s15327906mbr1802_5

Saucier, G. (1997). Effects of variable selection on the factor structure of person descriptors. Journal of Personality and Social Psychology, 73(6), 1296–1312. doi:10.1037/0022-3514.73.6.1296

Saucier, G., Georgiades, S., Tsaousis, I., & Goldberg, L. R. (2005). The factor structure of Greek personality adjectives. Journal of Personality and Social Psychology, 88(5), 856–875. doi:10.1037/0022-3514.88.5.856

Lorenzo-Seva, U., & ten Berge, J. M. F. (2006). Tucker's congruence coefficient as a meaningful index of factor similarity. Methodology, 2(2), 57–64. doi:10.1027/1614-2241.2.2.57

Forbes, M. K. (2023). Improving hierarchical models of individual differences: An extension of Goldberg's bass-ackward method. Psychological Methods. doi:10.1037/met0000546

Examples

# \donttest{
cmp <- comparability(sim16, k_max = 5, n_splits = 5, seed = 1)
#> ℹ Fitting 5 split-half replicates (pca, k = 1-5)...
#> ✔ Fitting 5 split-half replicates (pca, k = 1-5)... [140ms]
#> 
cmp
#> 
#> ── Split-Half Factor Comparability (ackwards) ──────────────────────────────────
#> Engine: pca
#> Basis: pearson
#> n: 1,000 (500 per half)
#> Splits: 5
#> Levels: 1-5
#> 
#> ── Comparability by level (median across splits) ──
#> 
#> k = 1: median r 1.00, min r 1.00 (m1f1)
#> k = 2: median r 1.00, min r 1.00 (m2f1)
#> k = 3: median r .79, min r .54 (m3f2)
#> k = 4: median r 1.00, min r 1.00 (m4f1)
#> k = 5: median r 1.00, min r .09 (m5f5)
#> ────────────────────────────────────────────────────────────────────────────────
#> Per-factor detail (incl. Tucker's φ) in `$summary`; per-split values in
#> `$coefficients`.
#> Conventional benchmarks: ≥ .90 replication floor (Everett, 1983; Saucier et
#> al., 2005), ≥ .95 factors interchangeable (Lorenzo-Seva & ten Berge, 2006) --
#> conventions, not tests. Interpret levels whose factors all replicate.
cmp$summary
#>    level factor   r_median      r_min phi_median    phi_min n_splits_ok
#> 1      1   m1f1 0.99897441 0.99601208 0.99649052 0.99229173           5
#> 2      2   m2f1 0.99812229 0.99717568 0.99456494 0.99096434           5
#> 3      2   m2f2 0.99850742 0.99600301 0.99601594 0.99407301           5
#> 4      3   m3f1 0.78659814 0.73422936 0.85231449 0.82714548           5
#> 5      3   m3f2 0.54081241 0.28312435 0.59080882 0.37403301           5
#> 6      3   m3f3 0.82233169 0.74886533 0.87304335 0.81227471           5
#> 7      4   m4f1 0.99751505 0.99459289 0.99356134 0.98733427           5
#> 8      4   m4f2 0.99756135 0.99455678 0.99197260 0.99050142           5
#> 9      4   m4f3 0.99753490 0.99620300 0.99384481 0.99266075           5
#> 10     4   m4f4 0.99795705 0.99723533 0.99492648 0.99367497           5
#> 11     5   m5f1 0.99742903 0.99288874 0.99363079 0.98729318           5
#> 12     5   m5f2 0.99604617 0.97938857 0.99122516 0.98620797           5
#> 13     5   m5f3 0.99596034 0.97771195 0.99181348 0.98687112           5
#> 14     5   m5f4 0.99748783 0.99463997 0.99493919 0.99365708           5
#> 15     5   m5f5 0.09432914 0.07241386 0.05763385 0.03246931           5
# }