Skip to contents

Changelog

Source: NEWS.md

smartcor 1.2.0

Version 1.2.0 aligns the R and Python packages with the accompanying paper (Harshvardhan and Ranjan, 2026). The changes below apply to both languages unless a section says otherwise.

New selection rules

Method selection now follows Table 3 and Figure 4 of the paper. Every pair starts from the Step 0 diagnostics described below. Default selections change for eight of the ten type pairs:

  • Continuous + continuous: Spearman is selected when the outer-fence rule flags either variable. Otherwise test_linearity() decides between Pearson and Spearman as before. When Pearson is selected and Mardia’s test rejects bivariate normality, the rationale says that the interval and p-value are approximate.
  • Continuous + binary: was always point-biserial. Now rank-biserial when the within-group residuals have outliers or fail the Jarque-Bera test; otherwise biserial when the LR test does not reject latent normality, and point-biserial when it does.
  • Binary + binary: "auto" still gives phi and TRUE tetrachoric. assume_latent_normal = FALSE now gives Yule’s Q instead of phi.
  • Continuous + ordinal: the continuous variable must pass the outer-fence rule and the Jarque-Bera test before polyserial is considered. The fallback is Kendall’s tau when the ordinal variable has 5 or fewer categories and Spearman otherwise (previously always Spearman).
  • Ordinal + ordinal: when the LR test rejects, the fallback is Kendall’s tau if either variable has 5 or fewer categories and Spearman otherwise (previously always Kendall’s tau).
  • Binary + ordinal: was always rank-biserial. Now polychoric when the LR test does not reject latent normality, and rank-biserial otherwise.
  • Continuous + categorical: was Cramer’s V on the binned continuous variable. Now the correlation ratio eta-squared, or Kruskal-Wallis eta-squared (H) when the within-group residuals have outliers or fail the Jarque-Bera test.
  • Ordinal + categorical: was Cramer’s V. Now Freeman’s theta, which keeps the order of the ordinal variable.

Binary + categorical and categorical + categorical still use Cramer’s V. Count variables follow the continuous rules everywhere, so count pairs change in the same way as the continuous ones.

assume_latent_normal = TRUE or FALSE now fixes only the latent-normality node; the outlier and Jarque-Bera checks still run. In "auto" mode a failed LR test resolves to “not assumed”.

Diagnostics

  • New detect_outliers() applies the outer-fence rule (values below Q1 - 3 IQR or above Q3 + 3 IQR, type-7 quartiles). It returns a smartcor_outliers object with the fences, quartiles, flagged values and their positions, skewness, and excess kurtosis. When the IQR is 0 the rule is marked as degenerate and flags nothing.
  • test_bivariate_normality() now chooses the test from the variable types: Mardia’s test (via MVN::mardia()) for two continuous variables, the Jarque-Bera test (via moments::jarque.test()) plus the LR chi-square test for continuous-ordinal and continuous-binary pairs, and the LR test alone for ordinal and binary pairs. Categorical variables give an error. A contingency table passed as x still runs the LR test. The result gains test, components, null_hypothesis, and interpretation; with several components the overall p.value is the smallest one, and the test rejects if any component rejects.
  • test_linearity() now returns a smartcor_lintest object with a print method. New fields: p_y_on_x and p_x_on_y, each max(min(p_bp, p_lb, p_bds), p_reset) for one direction; p.value, the larger of the two, which is below alpha exactly when linearity is rejected; direction, the direction whose component p-values are reported; null_hypothesis; and interpretation.
  • The BDS p-value in test_linearity() is now the minimum over the distance thresholds. It was the mean, which is not a p-value. Some linearity decisions change as a result.
  • New argument alpha = 0.05 on smart_cor(), smart_cormat(), smart_cor_df(), and compare_methods(), passed to every diagnostic test.

Linearity diagnostics and method selection

  • Added test_linearity() in both languages. It removes incomplete pairs by default, regresses each variable on the other, and orders the residuals by the regressor, averaging residuals at tied values. Diagnostics include Box-Pierce, Ljung-Box, BDS, Breusch-Pagan, and RESET, alongside Pearson, Spearman, and Kendall correlations.
  • Automatic smart_cor() calls for continuous/count pairs without outliers run these diagnostics at alpha. A direction rejects linearity when a residual structure test and RESET both reject. Spearman is selected only when both directions reject; otherwise the rule retains Pearson, with the inference recommendation taken from a direction that does not reject. The model behind the test assumes errors independent of the regressor, which can hold in only one direction, so requiring both directions keeps skewed margins from being read as nonlinearity. Explicit method arguments bypass the diagnostic rule.
  • Results include linearity_test. Inconclusive diagnostics retain Pearson and explain why. Non-rejection does not establish linearity. The decision does not depend on argument order or row order.
  • When the rule recommends robust/bootstrap Pearson inference, bootstrap = "auto" now uses a paired percentile-bootstrap confidence interval. The p-value remains the standard Pearson test; it is not a robust p-value. Set bootstrap = FALSE in R or bootstrap=False in Python to disable resampling.

New estimators

Four estimators bring the total to 18 methods:

  • biserial: the biserial correlation, fitted as a polyserial correlation with a two-category variable (see the polyserial note below). Wald p-value and interval on the Fisher-z scale. If the fit fails, point-biserial is reported and named in the result.
  • eta_squared: the correlation ratio, SS_between / SS_total. One-way ANOVA F p-value and a noncentral-F interval, mapped to eta-squared as in effectsize::eta_squared().
  • kw_eta_squared: Kruskal-Wallis eta-squared, (H - k + 1) / (n - k). Kruskal-Wallis p-value and a percentile bootstrap interval.
  • freemans_theta: Freeman’s theta for a nominal and an ordinal variable. Permutation p-value (n_perm shuffles of the ordinal variable) and a percentile bootstrap interval.

Polyserial and biserial now default to polycor::polyserial()’s own two-step estimator, with std.err = TRUE for the interval and p-value. New argument polyserial_ml = FALSE on smart_cor(), smart_cormat(), smart_cor_df(), and compare_methods(); TRUE, which we recommend, fits the full maximum-likelihood estimate with a strict tolerance and a safe-start retry, as earlier versions did.

available_methods() and compare_methods() now list the methods of Table 1 of the paper for each pair. The duplicate Pearson row for continuous-binary pairs is gone. For continuous-categorical pairs, compare_methods() bins the continuous variable only for Cramer’s V.

The two-letter codes used in the method heatmaps are Bs (biserial), E2 (eta-squared), EH (Kruskal-Wallis eta-squared), and Ft (Freeman’s theta). The clash between tetrachoric and Tschuprow’s T is fixed: they are now Tc and Ts.

Results

  • smart_cor() results gain diagnostics, a list with outliers, normality (Mardia or Jarque-Bera), latent (the LR test), and linearity. Entries that did not run are NULL. normality_test and linearity_test are kept for compatibility.
  • print.smartcor() shows one line per diagnostic that ran, under a bold green Diagnostics header.
  • tidy() gains four columns, outlier_flag, normality_p, latent_p, and linearity_p (24 columns in all). They are NA when a test did not run.
  • smart_cormat() keeps its return structure. Its stored pairs drop diagnostics but keep the four summary fields. Diagonal method labels follow the type and assume_latent_normal.
  • detect_type() classifies any variable with exactly two distinct values as binary, including a two-level ordered factor.

Optional positive-definite matrix adjustment

  • Added nearPD to smart_cormat() and smart_cor_df(). It remains off by default (FALSE in R, False in Python).
  • Explicit enablement adjusts the matrix while retaining a unit diagonal. R uses Matrix::nearPD(..., corr = TRUE); Python implements the corresponding Higham/Dykstra adjustment.
  • smart_cormat() retains original_correlations and adjustment diagnostics. Methods, pairwise details, and tidy/long output still describe the original estimates and inference. The adjustment does not refit those estimates.
  • The adjustment works with the new measures in the matrix (eta-squared and Freeman’s theta lie in [0, 1]).
  • Missing or infinite matrix entries must be resolved before adjustment. Use smart_cormat() rather than the two-table smart_cor_df() wrapper when you need the original matrix and adjustment diagnostics.

Python

  • Polychoric, polyserial, and biserial estimates are now full joint maximum-likelihood fits in Python, as polycor computes them in R; R restarts the polycor fit from its own solution with a tight tolerance so both languages reach the same optimum. Polychoric estimates at the boundary (|rho| >= 0.995, a table that separates the variables) fall back to Kendall’s tau in both languages.
  • Permutation p-values (Theil’s U, Freeman’s theta) are computed even with bootstrap=False in Python, as in R.
  • Python implements the same rules, diagnostics, estimators, method codes, and output fields. detect_outliers() returns an OutlierResult with 0-based index. Mardia’s test uses the formulas of MVN::mardia() and the Jarque-Bera test uses scipy.stats.jarque_bera. TIDY_COLUMNS lists the same 24 columns in the same order as R.
  • _select_method() is now a function of the diagnostics and follows the same criteria as R.
  • Fixed leftovers from the package rename: doctests use files("pysmartcor"), the install hint is pip install pysmartcor[viz], the license-files path in pyproject.toml points to src/pysmartcor/LICENSES/, and docstrings refer to pysmartcor.results.

Dependencies and validation

  • Added R dependencies fNonlinear, lmtest, Matrix, MVN (>= 6.0), and moments to Imports, and effectsize and rstatix to Suggests (used in tests). Added Python dependency statsmodels.
  • Added tests for type detection, the outer-fence rule, Mardia and Jarque-Bera against their reference packages, the classical estimators against cor.test() and polycor, the type-aware normality test, the linearity fields, every selection branch with its TRUE/FALSE overrides, the four new estimators against reference implementations, compare_methods() rows, tidy columns, and matrix adjustment with mixed measures. Cross-language comparisons cover the diagnostics and adjustment on the tested inputs; they do not establish parity for every package function. Seeded bootstrap and permutation results can differ across languages.

smartcor 1.0.1

CRAN release: 2026-08-09

CRAN resubmission after reviewer feedback. No statistical results change.

  • The Description field now cites the accompanying paper: Harshvardhan and Ranjan (2026) doi:10.48550/arXiv.2607.22285.
  • The ggcor_heatmap() / ggcor_method_heatmap() example is no longer wrapped in \dontrun{}; it uses only bundled data and runs during checks.
  • The internal seed helper no longer touches the global environment directly; it delegates to withr::with_seed() (withr added to Imports). Seeded results are unchanged, and a seeded call still restores the caller’s .Random.seed.
  • graphics and utils are now declared in Imports (both were already imported in NAMESPACE).
  • inst/CITATION, the README, and the documentation now cite the published arXiv preprint (arXiv:2607.22285) instead of “work in progress”.

smartcor 1.0.0

First public release.

  • smart_cor() detects the types of two variables and selects an appropriate correlation method (Pearson, Spearman, Kendall’s tau-b, polychoric, polyserial, tetrachoric, point-biserial, rank-biserial, Goodman-Kruskal gamma, or phi), with analytic inference on the Fisher-z scale and optional bootstrap.
  • smart_cormat() and smart_cor_df() extend automatic selection to full mixed-type data frames and correlation matrices.
  • compare_methods() reports all applicable estimators for a pair side by side.
  • test_bivariate_normality() implements the likelihood-ratio chi-square test for bivariate normality (Joreskog 2005); smart_cor() runs it automatically before reporting polychoric/polyserial estimates and falls back to Kendall’s tau-b when it rejects.
  • detect_type() and available_methods() expose the type-detection and method-lookup machinery.
  • Heatmap visualizations: ggcor_heatmap(), ggcor_method_heatmap(), and plot_cor_heatmap().
  • tidy() method for pipeline-friendly output.