Python SciPy t-test, Chi-square, and Correlation

When you need to run a t-test, a chi-square test, or a correlation analysis in Python, scipy.stats provides the core functions. The python scipy t test chi square and correlation workflow is a common pattern in data analysis and hypothesis testing. This article shows how to use scipy.stats for these three families of statistical tests, explains their assumptions, and helps you decide which test fits your data.

Running a t-test with scipy.stats

The t-test compares means between groups or against a known value. scipy.stats offers three main functions:

• ttest_1samp for a single sample against a population mean.
• ttest_ind for two independent samples.
• ttest_rel for paired samples.

Here is a minimal example using ttest_ind:

import numpy as np
from scipy import stats

group_a = np.array([2.3, 2.8, 3.1, 2.9, 3.4])
group_b = np.array([3.5, 3.8, 4.1, 3.9, 4.2])

t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"t-statistic: {t_stat:.3f}, p-value: {p_value:.4f}")

The function returns the t-statistic and the two-tailed p-value. For a one-tailed test, divide the p-value by 2 and check the sign of the t-statistic.

For paired data, use ttest_rel:

before = np.array([5.1, 5.4, 5.2, 5.6])
after = np.array([5.8, 6.1, 5.9, 6.3])

t_stat, p_value = stats.ttest_rel(before, after)

And for a one-sample test:

sample = np.array([12.2, 12.8, 13.1, 12.9, 13.4])
t_stat, p_value = stats.ttest_1samp(sample, popmean=12.0)

The popmean argument is the value you compare the sample mean against.

Chi-square test for categorical data

The chi-square test works on categorical data. scipy.stats provides two common functions:

• chisquare for a goodness-of-fit test against expected frequencies.
• chi2_contingency for a test of independence between two categorical variables.

For a goodness-of-fit test, you provide observed and expected frequencies:

observed = [25, 35, 20, 20]
expected = [25, 25, 25, 25]

chi2_stat, p_value = stats.chisquare(observed, f_exp=expected)
print(f"Chi-square: {chi2_stat:.3f}, p-value: {p_value:.4f}")

For a contingency table, use chi2_contingency:

table = np.array([[30, 10], [20, 40]])

chi2_stat, p_value, dof, expected = stats.chi2_contingency(table)
print(f"Chi-square: {chi2_stat:.3f}, p-value: {p_value:.4f}, dof: {dof}")

The function returns the test statistic, p-value, degrees of freedom, and the expected frequency table. The expected table is useful for checking whether any cell has an expected count below 5, which can invalidate the test.

Correlation analysis: Pearson and Spearman

Correlation measures the strength and direction of a relationship between two continuous variables. scipy.stats offers pearsonr for linear correlation and spearmanr for rank-based (monotonic) correlation.

x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 4, 5, 4, 5])

pearson_r, pearson_p = stats.pearsonr(x, y)
spearman_rho, spearman_p = stats.spearmanr(x, y)

print(f"Pearson r: {pearson_r:.3f}, p: {pearson_p:.4f}")
print(f"Spearman rho: {spearman_rho:.3f}, p: {spearman_p:.4f}")

pearsonr assumes a linear relationship and normally distributed residuals. spearmanr works on ranks and detects monotonic relationships, making it more robust to outliers and non-linear monotonic patterns.

Assumptions and when each test applies

Each test has assumptions you must verify before trusting the p-value.

The t-test assumes:

• The data are continuous and approximately normally distributed, especially for small samples.
• Observations are independent.
• For ttest_ind, the two groups have similar variances (Welch's t-test is used by default, which relaxes this assumption).

The chi-square test assumes:

• Observations are independent.
• Expected frequencies are not too small; a common rule is that no more than 20% of cells have expected count below 5, and no cell has expected count below 1.
• The data are categorical, not continuous.

Correlation tests assume:

• For Pearson, a linear relationship and no significant outliers.
• For Spearman, a monotonic relationship (not necessarily linear).
• Both require paired observations.

If your data violate these assumptions, the p-value may be misleading. Consider transformations, non-parametric alternatives, or exact tests.

Choosing the right test for your data

The choice depends on your data type and research question.

Use ttest_ind when you have two separate groups. Use ttest_rel when the same subjects are measured twice. Use chisquare when you have observed counts and a theoretical expected distribution. Use chi2_contingency when you have a contingency table of counts.

For correlation, use pearsonr when the relationship is linear and residuals are normal. Use spearmanr when you suspect a monotonic but non-linear relationship, or when outliers are present.

Common pitfalls and practical considerations

A p-value below 0.05 does not prove an effect is meaningful. Always consider effect size and confidence intervals. scipy.stats does not return effect sizes directly, so you may need to compute them separately.

Multiple testing inflates the chance of false positives. If you run many tests, apply a correction like Bonferroni or use an FDR-controlling method.

For chi2_contingency, the expected frequency table is returned. Check it for small expected counts. If any expected count is below 1, or many are below 5, consider Fisher's exact test (available as scipy.stats.fisher_exact for 2x2 tables).

When working with large datasets, scipy.stats functions are vectorized and efficient, but they still require the data to be in memory. For very large arrays, consider using numpy operations to compute test statistics manually if memory is a constraint, but that is rarely necessary for typical data analysis tasks.

Also note that pearsonr and spearmanr return NaN if either input has constant values. Check for zero variance before calling the function.

Handling missing data and edge cases

scipy.stats functions do not automatically drop missing values. You must remove NaN values before calling them. Use numpy.isnan or pandas.dropna to clean the data.

For paired tests, ensure the two arrays have the same length and are aligned. For ttest_rel, the order matters because it pairs elements by position.

When your data are not normally distributed, the t-test may still be acceptable for large samples due to the Central Limit Theorem. For small samples, consider the Mann-Whitney U test (mannwhitneyu) or Wilcoxon signed-rank test (wilcoxon) as non-parametric alternatives.

For correlation, if you have ordinal data or non-normal continuous data, spearmanr is a safer default.

These practical details will help you avoid the most common mistakes when applying python scipy t test chi square and correlation in real projects.