Technical Definition

The t-test is the hypothesis test of the t-distribution. The t-distribution is a particular kind of probability distribution, similar to the normal distribution but the variance is estimated rather than known. There are various different types of t-tests; any hypothesis test which relies on the assumption that the parameter of interest follows a t-distribution falls under the t-test family. A t-test results in a t-score, which can then be translated to a p-value for easier interpretation and to determine statistical significance.

Different Types of T-Tests

There are different versions of the t-test that can be used in different scenarios, the three main types are:

Independent Samples T-Test

This is the type most commonly used in online experimentation. It compares the means for two independent groups. For example, when you randomly assign all visitors to a website into one of two groups, you are creating two separate samples of visitors who are independent from each other. The independent samples t-test can be used to test for differences between the average behavior of users in those two groups.

Paired Samples T-Test

This type of test is used for paired data, when each measurement in a sample is paired with a measurement from the other sample. For example in a repeated-measures design, each pair may contain measurements for the same unit before and after a treatment, or in a matched-pairs design each unit may be matched with a similar unit from another sample.

One Sample T-Test

The one-sample t-test can be used to determine whether the mean of a single sample differs from a particular value. For example, it could be used to determine whether the average exam score for a class of students differs from a particular target.