Choosing the Right Correlation Method: Theory and Rationale

Introduction

The Pearson product-moment correlation coefficient is the most widely used measure of linear association. However, it is only optimal when both variables are continuous and approximately normally distributed. In practice, researchers work with binary variables (yes/no, male/female), ordinal variables (Likert scales, education levels), and nominal categorical variables (colour, country), each requiring a different correlation method to avoid bias, attenuation, or outright meaninglessness.

smartcor automates method selection based on detected variable types. This vignette explains the statistical theory behind each method, why it is appropriate for a given variable-type combination, and what can go wrong when the wrong method is used.

Variable Types

Before selecting a correlation method, we must classify each variable. smartcor recognises five types:

Type	Description	Examples
Continuous	Numeric with many distinct values	Income, temperature, test scores
Count	Non-negative integers with many distinct values	Number of citations, visits, or events
Binary	Exactly two distinct values	Male/female, yes/no, 0/1
Ordinal	Ordered categories, discrete	Likert scales (1–5), education level
Categorical	Unordered categories	Colour, country, blood type

The detect_type() function applies the following heuristic, in order:

1. Ordered factors are always ordinal.
2. Unordered factors and character vectors are categorical, unless they have exactly two levels, in which case they are binary.
3. Logical vectors are binary.
4. Numeric vectors with exactly 2 unique values are binary.
5. Numeric vectors with $\leq 10$ unique values are ordinal (configurable via ordinal_threshold).
6. Remaining non-negative, integer-valued numeric vectors are count (set detect_count = FALSE to skip this step).
7. All other numeric vectors are continuous.

This heuristic is deliberately conservative: a 7-point Likert scale stored as integers (1, 2, …, 7) will be detected as ordinal, which is the correct statistical treatment. Count variables are treated exactly like continuous variables when a correlation is computed; the separate label is purely descriptive.

Correlation Methods by Variable-Type Combination

Continuous + Continuous: Pearson Correlation

The Pearson correlation coefficient between continuous variables $X$ and $Y$ is defined as:

$$r = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2 \cdot \sum_{i=1}^n (y_i - \bar{y})^2}}$$

Why Pearson? For two continuous variables, Pearson is the maximum-likelihood estimator of the population correlation $\rho$ under bivariate normality. It captures linear association and has well-understood sampling properties (exact $t$-test for $H_0: \rho = 0$, Fisher’s $z$-transformation for confidence intervals).

When Pearson is suboptimal: Pearson is sensitive to outliers and non-linear relationships. When the relationship is monotonic but not linear, or when distributions are heavily skewed, Spearman or Kendall may be more appropriate. smartcor offers these as alternatives via method = "spearman" or method = "kendall".

Continuous + Binary: Point-Biserial Correlation

When one variable is continuous and the other is binary (coded 0/1), the point-biserial correlation is used. It is defined as:

$$r_{pb} = \frac{\bar{X}_1 - \bar{X}_0}{s_X} \cdot \sqrt{\frac{n_1 n_0}{n^2}}$$

where $\bar{X}_1$ and $\bar{X}_0$ are the means of the continuous variable in each binary group, $s_X$ is the pooled standard deviation, and $n_1$, $n_0$ are the group sizes.

Key insight: The point-biserial correlation is mathematically identical to the Pearson correlation when the binary variable is coded as 0 and 1. This is not an approximation; it is an algebraic equivalence. The proof follows from substituting $Y \in \{0, 1\}$ into the Pearson formula and simplifying (see Appendix in Harshvardhan and Ranjan, 2026).

Monte Carlo simulations confirm this: the maximum absolute difference between Pearson and point-biserial across 500 replications, multiple sample sizes, and multiple true correlations is exactly 0.

Implication: There is no advantage to using a specialised point-biserial function in software. The Pearson correlation with 0/1 coding gives the identical result. smartcor uses this equivalence internally.

Binary + Binary: Phi Coefficient

When both variables are binary, the phi coefficient ($\phi$) measures their association. For a $2 \times 2$ contingency table:

$$\phi = \frac{ad - bc}{\sqrt{(a+b)(c+d)(a+c)(b+d)}}$$

where $a, b, c, d$ are the cell counts.

Phi = Pearson for 0/1 coding. Like the point-biserial case, $\phi$ is algebraically identical to the Pearson correlation when both variables are coded 0/1. Simulations confirm that the maximum difference is on the order of $10^{-14}$ (floating-point precision).

The attenuation problem. Phi measures the observed association between two binary variables. But what if each binary variable is a dichotomised version of an underlying continuous (latent) variable? For example, “pass/fail” is a binary reduction of a continuous test score.

In this case, phi underestimates the latent correlation $\rho$. The attenuation depends on the marginal proportions (how far the cut points are from the median):

• When $\rho = 0.7$ and variables are dichotomised at the median, phi is approximately 0.45–0.50.
• The attenuation increases as cut points move away from the median.

This is why tetrachoric correlation exists.

Binary + Binary (Latent Normal): Tetrachoric Correlation

The tetrachoric correlation assumes that each observed binary variable arises from dichotomising an underlying continuous normal variable at some threshold. It estimates the correlation between the latent continuous variables, not the observed binary ones.

The model is:

$$(X^*, Y^*) \sim \text{Bivariate Normal}(\mu, \Sigma) \quad \text{with} \quad \Sigma = \begin{pmatrix} 1 & \rho \\ \rho & 1 \end{pmatrix}$$

The observed binary variables are $X = \mathbf{1}(X^* > \tau_x)$ and $Y = \mathbf{1}(Y^* > \tau_y)$, where $\tau_x, \tau_y$ are the thresholds estimated from the marginal proportions.

When to use tetrachoric:

• When the binary variables are genuinely dichotomised continuous variables (e.g., pass/fail from test scores).
• When you want to recover the latent correlation for use in structural equation modelling or factor analysis.

When NOT to use tetrachoric:

• When the binary variable is inherently binary (e.g., male/female, alive/dead) with no meaningful continuous latent variable.
• When the latent normality assumption is implausible.

A 2 x 2 table is always saturated (df = 0), so the LR test that drives assume_latent_normal = "auto" cannot run and the data cannot support the latent-normality assumption. smartcor therefore defaults a binary-binary pair to phi; tetrachoric requires an explicit assume_latent_normal = TRUE or method = "tetrachoric".

Ordinal + Ordinal: Polychoric Correlation

The polychoric correlation generalises the tetrachoric to ordinal variables with more than two categories. It assumes each observed ordinal variable is a discretisation of a latent normal variable:

$$X^* \sim N(0, 1), \quad X = k \iff \tau_{k-1} < X^* \leq \tau_k$$

The polychoric correlation $\hat{\rho}$ is the maximum-likelihood estimate of $\rho$ in the bivariate normal model, given the observed contingency table and estimated thresholds.

Advantages:

• Recovers the latent correlation without attenuation.
• Simulations show that polychoric is essentially unbiased (bias < 0.003) when the latent normality assumption holds, even with as few as 100 observations.
• More ordinal categories improve precision: the biggest improvement is from 2 to 3 categories, with diminishing returns beyond 5.

Disadvantages:

• Sensitive to violated normality. When the latent distribution is bimodal or heavily skewed, polychoric shows positive bias (up to +0.06 in simulations). The direction of bias is towards overestimation.
• Requires iterative optimisation; can fail to converge with very small samples or sparse tables.

Alternative: Kendall’s tau-b ($\tau_b$).

Kendall’s tau is defined in terms of concordant and discordant pairs:

$$\tau_b = \frac{C - D}{\sqrt{(C + D + T_X)(C + D + T_Y)}}$$

where $C$ = concordant pairs, $D$ = discordant pairs, $T_X$ = pairs tied on $X$ only, $T_Y$ = pairs tied on $Y$ only.

Simulations show that Kendall’s tau has a consistent negative bias relative to the true $\rho$ (it underestimates), but this bias is stable across all distribution types: normal, skewed, heavy-tailed, bimodal, and uniform. This makes it the most robust choice when the latent normality assumption is uncertain.

smartcor defaults to polychoric when latent normality is assumed, and Kendall’s tau otherwise.

Continuous + Ordinal: Polyserial Correlation

The polyserial correlation is the continuous-ordinal analogue of polychoric. It assumes the ordinal variable arises from discretising a latent normal variable, while the continuous variable is observed directly.

The model estimates $\rho$ between the continuous variable and the latent continuous variable underlying the ordinal one.

Simulations show polyserial is essentially unbiased (bias < 0.005) when the latent normality assumption holds. When the assumption is violated, Spearman’s rank correlation is a robust alternative.

Binary + Ordinal: Rank-Biserial Correlation

When one variable is binary and the other is ordinal, the rank-biserial correlation (Glass, 1966) is appropriate. It compares the mean ranks of the ordinal variable in each binary group:

$$r_{rb} = \frac{2(\bar{R}_1 - \bar{R}_0)}{n}$$

This is a nonparametric measure that respects the ordinal nature of one variable and the dichotomous nature of the other. Kendall’s tau is available as an alternative.

Categorical + Categorical: Cramer’s V

When both variables are nominal (unordered categories), there is no concept of “direction” in the association. Cramer’s V is the standard symmetric measure:

$$V = \sqrt{\frac{\chi^2}{n \cdot (k - 1)}}$$

where $\chi^2$ is the Pearson chi-squared statistic and $k = \min(r, c)$ is the smaller of the number of rows and columns in the contingency table.

Cramer’s V ranges from 0 (no association) to 1 (perfect association) and is corrected for table dimensions.

Alternatives:

• Theil’s U (uncertainty coefficient): An asymmetric measure based on information theory. $U(Y|X)$ measures the proportion of entropy in $Y$ that is explained by knowing $X$.
• Tschuprow’s T: Similar to Cramer’s V but uses the geometric mean of $(r-1)$ and $(c-1)$ instead of the minimum, making it stricter for non-square tables.

Other Combinations

• Binary + Categorical: Treated as Cramer’s V (the binary variable is a special case of a 2-level nominal variable).
• Ordinal + Categorical: Cramer’s V is used, but with a warning that ordinal ordering information is lost.
• Continuous + Categorical: No standard correlation measure exists. smartcor bins the continuous variable into quantiles and computes Cramer’s V, with a warning.

Why Not Always Use Pearson?

A common question is: “Why not just use Pearson for everything?” The answer depends on what you mean by correlation and what you intend to do with it.

1. Attenuation bias. When continuous variables are observed only as ordinal categories (e.g., Likert scales), Pearson/phi/point-biserial measures the association between the observed (discretised) values. This systematically underestimates the latent association. For example, with a true $\rho = 0.7$ and binary dichotomisation, phi gives approximately 0.45–0.50. Polychoric/tetrachoric recovers the correct 0.70.

This matters for factor analysis and structural equation modelling, where underestimated correlations lead to underestimated factor loadings and distorted model fit.

2. Scale appropriateness. Pearson assumes interval-scale data. Ordinal variables violate this: the “distance” between categories 1 and 2 is not necessarily the same as between 4 and 5. Rank-based measures (Kendall, Spearman) and latent-variable measures (polychoric) respect the ordinal scale.

3. Meaninglessness for nominal data. Computing Pearson on arbitrarily coded nominal variables (e.g., red = 1, blue = 2, green = 3) is meaningless because the numbers have no inherent order. Cramer’s V and related measures correctly capture nominal association.

The Latent Normality Question

The choice between latent-variable methods (polychoric, polyserial, tetrachoric) and distribution-free methods (Kendall, Spearman) hinges on whether the underlying latent distribution is approximately normal.

When latent normality is reasonable:

• Psychological constructs measured on Likert scales (e.g., agree/disagree).
• Test scores dichotomised into pass/fail.
• Any variable that is conceptually continuous but observed in discrete categories.

When latent normality is doubtful:

• Inherently discrete variables (e.g., number of children).
• Variables with known multimodal or heavily skewed latent distributions.
• When you have no theoretical reason to assume normality.

Practical guidance:

• If latent normality holds, polychoric/polyserial are essentially unbiased and more efficient.
• If normality is violated, Kendall’s tau is the safest choice: it consistently underestimates the true $\rho$ but by a stable, predictable amount across all distribution types.
• When in doubt, compute both and compare. If the estimates differ substantially, the normality assumption may be violated.

Summary of Method Selection Logic

The following table summarises smartcor’s decision matrix:

Variable types	Default method	Alternative
Correlation methods (signed, from -1 to +1)
Continuous + continuous	Pearson	Spearman (suggested, especially if nonlinear)
Continuous + binary	Pearson = point-biserial	—
Binary + binary	Pearson = phi	Tetrachoric (preferred, if latent normality holds)
Continuous + ordinal	Spearman/Kendall	Polyserial (preferred, if latent normality holds)
Ordinal + ordinal	Kendall’s tau	Polychoric (preferred, if latent normality holds)
Binary + ordinal	Rank-biserial	Spearman
Association measures (unsigned, from 0 to 1)
Continuous + categorical	Cramer’s V (binned)	—
Binary + categorical	Cramer’s V	—
Ordinal + categorical	Cramer’s V	—
Categorical + categorical	Cramer’s V	Theil’s U, Tschuprow’s T

Count variables are treated as numeric continuous variables, so each count combination follows the corresponding continuous row.

By default (assume_latent_normal = "auto"), smartcor runs the likelihood-ratio chi-square test of test_bivariate_normality() on each pair where the latent-normality question matters and follows the verdict: polychoric or polyserial when the bivariate normal threshold model is not rejected, Kendall or Spearman when it is. The binary-binary pair is decided by the saturated-table rule above: the 2 x 2 table gives the test zero degrees of freedom, so phi is reported unless assume_latent_normal = TRUE is set. Passing NULL asks interactively; in non-interactive sessions that falls back to TRUE.

The smart_cor_df() convenience function uses assume_latent_normal = "auto" and never prompts, which makes it suitable for scripts and pipelines.

References

• Glass, G. V. (1966). Note on rank biserial correlation. Educational and Psychological Measurement, 26(3), 623–631.
• Olsson, U. (1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44(4), 443–460.
• Pearson, K. (1900). Mathematical contributions to the theory of evolution. VII. On the correlation of characters not quantitatively measurable. Philosophical Transactions of the Royal Society A, 195, 1–47.
• Kendall, M. G. (1938). A new measure of rank correlation. Biometrika, 30(1/2), 81–93.
• Cramer, H. (1946). Mathematical Methods of Statistics. Princeton University Press.
• Harshvardhan, M. and Ranjan, P. (2026). smartcor: Intelligent Correlation Method Selection for Mixed Variable Types. arXiv preprint arXiv:2607.22285. https://doi.org/10.48550/arXiv.2607.22285