Glossary /

T-Test

A t-test is a type of hypothesis test which assumes the test statistic follows the t-distribution. It can be used to determine if there is a statistically significant difference between two groups. 

Technical Definition

The t-test is the hypothesis test of the t-distribution. The t-distribution is a particular kind of probability distribution, similar to the normal distribution but the variance is estimated rather than known. There are various different types of t-tests; any hypothesis test which relies on the assumption that the parameter of interest follows a t-distribution falls under the t-test family. A t-test results in a t-score, which can then be translated to a p-value for easier interpretation and to determine statistical significance. 

Different Types of T-Tests

There are different versions of the t-test that can be used in different scenarios, the three main types are:

Independent samples t-test

This is the type most commonly used in online experimentation. It compares the means for two independent groups. For example, when you randomly assign all visitors to a website into one of two groups, you are creating two separate samples of visitors who are independent from each other. The independent samples t-test can be used to test for differences between the average behavior of users in those two groups.

Paired samples t-test

This type of test is used for paired data, when each measurement in a sample is paired with a measurement from the other sample. For example in a repeated-measures design, each pair may contain measurements for the same unit before and after a treatment, or in a matched-pairs design each unit may be matched with a similar unit from another sample. 

One sample t-test

The one-sample t-test can be used to determine whether the mean of a single sample differs from a particular value. For example, it could be used to determine whether the average exam score for a class of students differs from a particular target.