Split-half factor comparability

Measures how well each factor at each level of a bass-ackwards hierarchy replicates across random split-halves of the sample. A factor is a summary variable standing in for a group of items that move together. In a split-half analysis the sample is split into two random halves and each half is analysed separately. The coefficients are Everett's (1983) factor comparability coefficients. They revive the split-half replication gate of the research program that produced the bass-ackwards method. Saucier (1997) screened factor solutions by their split-half stability, and Saucier, Georgiades, Tsaousis, and Goldberg (2005) chose the optimal hierarchical level by requiring split-half replication above a .90 threshold. This is also the direct instrument for the overextraction caution in suggest_k(), because non-replicable structure concentrates in the deeper levels of an overextracted hierarchy (Forbes, 2023).

Usage

comparability(
  data,
  k_max,
  engine = "pca",
  cor = "pearson",
  fm = "minres",
  n_splits = 10L,
  seed = NULL,
  ...
)

Arguments

data: A data frame or numeric matrix of observed variables (items in columns, observations in rows). Raw data only, because splitting needs rows, so a correlation matrix is not accepted. Missing values are handled pairwise throughout (as in ackwards()'s default).
k_max: Maximum number of factors or components to evaluate. Normally this is the same value (or one or two above it) you intend to pass to ackwards(). Required.
engine: Extraction engine: "pca" (default) or "efa". "esem" is not yet supported here, because fitting 2 * n_splits lavaan hierarchies needs its own performance treatment. For ESEM (exploratory structural equation modeling) workflows, run comparability() with engine = "efa" as a structural screen.
cor: Correlation basis: "pearson" (default) or "spearman". As with suggest_k(), "polychoric" is not supported, because estimating polychoric matrices in every half-sample is slow and unstable. Users analysing ordinal items (a few ordered categories, such as a 1 to 5 rating) should screen replicability on the Pearson basis. They should then fit the final model with cor = "polychoric" in ackwards(), which estimates the correlation between the continuous traits assumed to underlie the ordered responses.
fm: Factor extraction method passed to psych::fa(). It is only used when engine = "efa". One of "minres" (default), "ml", or "pa".
n_splits: Number of random split-half replicates. Default 10L. The published precedents used a single split (Saucier et al., 2005). Repeating the split guards against the luck of one draw, so the coefficients are summarised over replicates. Each replicate fits 2 * k_max solutions, so the default costs 20 hierarchy fits. PCA (principal component analysis) and EFA (exploratory factor analysis) are fast enough that this is typically a few seconds.
seed: Integer seed for reproducible splits. NULL (default) uses the current RNG state.
...: Reserved for future arguments.

Value

An object of class "comparability". Print it for a per-level summary, or call autoplot() on it for a diagnostic plot. The list contains:

coefficients: Data frame with one row per split x level x factor. Its columns are split, level, factor (full-sample m{k}f{j} label), r (score comparability), and phi (Tucker's congruence of the matched loading columns). A value is NA when the level did not converge in one of the halves.
summary: Data frame with one row per level x factor. Its columns are level, factor, r_median, r_min, phi_median, phi_min (across splits), and n_splits_ok (splits in which both halves converged).
k_max: Deepest level evaluated. It can be lower than the k_max you asked for when the full-sample fit truncated, that is, when deep levels did not converge. The original request is kept in k_requested.
k_requested, n_splits, n_half, engine, cor, fm, n_obs, n_vars, seed: Metadata.

Details

For each of n_splits random half-splits, solutions at every level from 1 to k_max are fit independently in each half. Each half-solution's factors are matched to the full-sample solution's factors, so coefficients are reported under the same m{k}f{j} labels you get from ackwards(). The comparability coefficient for a factor is the correlation between its two matched half-solution scores. It is computed on the pooled correlation matrix via the same W'RW algebra used for between-level edges, applying both halves' scoring weights to the full sample, exactly Everett's procedure. Tucker's congruence coefficient (phi) between the matched half-solution loading columns is reported alongside. A loading is the correlation between an item and a factor, and congruence is a 0 to 1 index of how similar two loading patterns are. So comparability asks whether the two halves' scores agree, and phi asks whether their loading patterns agree.

Interpreting the output

Coefficients near 1 mean the factor re-emerges in independent half-samples. A factor whose comparability is low is sample-idiosyncratic and should not anchor substantive interpretation. Two benchmarks are conventional. The first, .90, is the split-half replication threshold of Goldberg's lexical research program (Saucier et al., 2005). It follows from Everett's (1983) rationale, under which split-half factors share at least 81% of their variance. The second, .95, is the stricter bound at which two factors are conventionally treated as interchangeable (Lorenzo-Seva & ten Berge, 2006). These are conventions, not tests, so comparability() reports every coefficient and flags nothing. The deepest level at which all factors replicate is a natural hierarchy floor for ackwards()'s k_max. See vignette("ackwards-girard") for the full workflow.

A level that fails to converge in a half-sample yields NA coefficients for that split, because convergence is data, not an error. The number of usable splits per factor is reported in summary$n_splits_ok and a message summarises any shortfall.

References

Everett, J. E. (1983). Factor comparability as a means of determining the number of factors and their rotation. Multivariate Behavioral Research, 18(2), 197–218. doi:10.1207/s15327906mbr1802_5

Saucier, G. (1997). Effects of variable selection on the factor structure of person descriptors. Journal of Personality and Social Psychology, 73(6), 1296–1312. doi:10.1037/0022-3514.73.6.1296

Saucier, G., Georgiades, S., Tsaousis, I., & Goldberg, L. R. (2005). The factor structure of Greek personality adjectives. Journal of Personality and Social Psychology, 88(5), 856–875. doi:10.1037/0022-3514.88.5.856

Lorenzo-Seva, U., & ten Berge, J. M. F. (2006). Tucker's congruence coefficient as a meaningful index of factor similarity. Methodology, 2(2), 57–64. doi:10.1027/1614-2241.2.2.57

Forbes, M. K. (2023). Improving hierarchical models of individual differences: An extension of Goldberg's bass-ackward method. Psychological Methods. doi:10.1037/met0000546

Examples

# \donttest{
cmp <- comparability(sim16, k_max = 5, n_splits = 5, seed = 1)
#> ℹ Fitting 5 split-half replicates (pca, k = 1-5)...
#> ✔ Fitting 5 split-half replicates (pca, k = 1-5)... [244ms]
#> 
cmp
#> 
#> ── Split-Half Factor Comparability (ackwards) ──────────────────────────────────
#> Engine: pca
#> Basis: pearson
#> n: 1,000 (500 per half)
#> Splits: 5
#> Levels: 1-5
#> 
#> ── Comparability by level (median across splits) ──
#> 
#> k = 1: median r 1.00, min r 1.00 (m1f1)
#> k = 2: median r 1.00, min r 1.00 (m2f1)
#> k = 3: median r .79, min r .54 (m3f2)
#> k = 4: median r 1.00, min r 1.00 (m4f1)
#> k = 5: median r 1.00, min r .09 (m5f5)
#> ────────────────────────────────────────────────────────────────────────────────
#> Per-factor detail (incl. Tucker's φ) in `$summary`; per-split values in
#> `$coefficients`.
#> Conventional benchmarks: ≥ .90 replication floor (Everett, 1983; Saucier et
#> al., 2005), ≥ .95 factors interchangeable (Lorenzo-Seva & ten Berge, 2006) --
#> conventions, not tests. Interpret levels whose factors all replicate.
cmp$summary
#>    level factor   r_median      r_min phi_median    phi_min n_splits_ok
#> 1      1   m1f1 0.99897441 0.99601208 0.99649052 0.99229173           5
#> 2      2   m2f1 0.99812229 0.99717568 0.99456494 0.99096434           5
#> 3      2   m2f2 0.99850742 0.99600301 0.99601594 0.99407301           5
#> 4      3   m3f1 0.78659814 0.73422936 0.85231449 0.82714548           5
#> 5      3   m3f2 0.54081241 0.28312435 0.59080882 0.37403301           5
#> 6      3   m3f3 0.82233169 0.74886533 0.87304335 0.81227471           5
#> 7      4   m4f1 0.99751505 0.99459289 0.99356134 0.98733427           5
#> 8      4   m4f2 0.99756135 0.99455678 0.99197260 0.99050142           5
#> 9      4   m4f3 0.99753490 0.99620300 0.99384481 0.99266075           5
#> 10     4   m4f4 0.99795705 0.99723533 0.99492648 0.99367497           5
#> 11     5   m5f1 0.99742903 0.99288874 0.99363079 0.98729318           5
#> 12     5   m5f2 0.99604617 0.97938857 0.99122516 0.98620797           5
#> 13     5   m5f3 0.99596034 0.97771195 0.99181348 0.98687112           5
#> 14     5   m5f4 0.99748783 0.99463997 0.99493919 0.99365708           5
#> 15     5   m5f5 0.09432914 0.07241386 0.05763385 0.03246931           5
# }