Fisher and Fisher-Freeman-Halton Exact Test
Introduction
Perhaps the most commonly used test when you have two binary variables is the Fisher (Exact) Test. It tests if "the relative proportions of one variable are independent of the second variable; in other words, the proportions at one variable are the same for different values of the second variable" (McDonald, 2014, p. 77). The hypothesis is also sometimes stated in terms of the Odds Ratio, that it would be 1. Slightly less formal: It tests if the two variables have an association (no relation / no influence on each other).
For larger tables the test is also known as the Fisher-Freeman-Halton Exact Test (Freeman & Halton, 1951).
Its important to note that the test assumes the margins are fixed (i.e. the row and column totals don't change), so only use this test if this assumptions is valid for your data. See InfluentalPoints.com for more critique on this test.
As Hitchcock (2009, pp. 3–4) points out, the history is a bit murky. Some refer to Fisher (1922) who does seem to mention the exact distribution in a footnote on page 339, but the test is supposedly first fully discussed by Fisher in the fifth edition of his book (1934). Irwin (1935) notes that his paper was concluded already in 1933, but publication was delayed. He also refers to Yates (1934) who also discuss the test, and refers to personal communication with Fisher. Another paper from Fisher (1935b) is also sometimes referred to. Fisher (1935a, pp. 24–29) described an experiment and the exact test to use, which is commonly known as the Lady Tasting Tea Experiment.
Performing the Test
Click here to see how to perform a Fisher Exact test (for 2x2 tables)
with Flowgorithm
A flowgorithm for Fisher's Exact Test in Figure 1.
It takes as input paramaters four counts (one for each cell).
It uses a function the binomial coefficient.
Flowgorithm file: FL-TSfisher2x2.fprg.
with SPSS
manually (formula and example)
If we are given a 2 by 2 table, and label each cell as shown in Table 1.
| Col. 1 | Col. 2 | Total | |
|---|---|---|---|
| Row 1 | a |
b |
R1 = a + b |
| Row 2 | c |
d |
R2 = c + d |
| Total | C1 = a + c |
C2 = b + d |
n = R1 + R2 |
We can then use the following algorithm:
Step 1: Determine the probability of the sample table.
\(p_{sample} = \frac{\binom{R1}{a}\times\binom{R2}{c}}{\binom{n}{C1}} = \frac{\binom{R1}{b}\times\binom{R2}{d}}{\binom{n}{C2}}\)
This formula uses the binomial coefficient, defined as:
\(\binom{x}{y}=\frac{x!}{y!\times\left(x-y\right)!}\)
Which in turn uses the factorial operator (!), defined as:
\(x! = \prod_{i=1}^x i\)
Step 2: Determine the minimum and maximum of the top-left cell.
\(a_{min} = \text{MAX}\left(0, C1 + R1 - n\right)\)
\(a_{max} = \text{MIN}\left(R1, C1\right)\)
The reason for the minimum value of 'a', is first that it cannot be negative, since these are counts. So 0 would be the lowest ever possible. However, once 'a' is set, and the totals are fixed, all other values should also be positive (or zero). The value for 'b' will be if 'a' is 0, it will simply be R1 - a. The value for 'c' is also no issue, this is simply C1 - a. However 'd' might be negative, even if a = 0. The value for 'd' is n - R1 - c. Since c = C1 - a, we get d = n - R1 - C1 + a. But this could be negative if R1 + C1 > n. So, 'a' must be at least C1 + R1 - n.
The maximum for 'a' is simply the minimum of either it's row total, or column total.
Step 3: Go over all possible values of the top-left cell and calculate the probability. Add it to the p-value if it is less or equal to the one from the sample.
Worked out example
We'll use the example used earlier, shown in Table 2.
| Col. 1 | Col. 2 | Total | |
|---|---|---|---|
| Row 1 | a = 8 |
b = 16 |
R1 = 24 |
| Row 2 | c = 3 |
d = 15 |
R2 = 18 |
| Total | C1 = 11 |
C2 = 31 |
n = 42 |
Step 1: Determine the probability of the sample table.
Filling out the formula we get:
\(p_{sample} = \frac{\binom{24}{8}\times\binom{18}{15}}{\binom{42}{11}}\)
Using the formula for the binomial coefficient:
\(p_{sample} = \frac{\frac{24!}{8!\times\left(24-8\right)!}\times\frac{18!}{15!\times\left(18-15\right)!}}{\frac{42!}{11!\times\left(42-11\right)!}}\)
\( = \frac{\frac{24!}{8!\times16!}\times\frac{18!}{15!\times3!}}{\frac{42!}{11!\times31!}}\)
\( = \frac{\frac{24\times23\times\dots\times17}{8!} \times \frac{18\times17\times16}{3!}}{\frac{42\times41\times\dots\times32}{11!}}\)
\( = \frac{\frac{24\times23\times\dots\times17}{8\times7\times\dots\times1} \times \frac{18\times17\times16}{3\times2\times1}}{\frac{42\times41\times\dots\times32}{11\times10\times\dots\times1}}\)
\( = \frac{\frac{29654190720}{40320} \times \frac{4896}{6}}{\frac{42\times41\times\dots\times33\times32}{11\times10\times\dots\times1}}\)
To simplify the denominator we could expand the factorials:
\( \frac{42\times41\times\dots\times33\times32}{11\times10\times\dots\times1} = \frac{42\times41\times40\times39\times38\times37\times36\times35\times34\times33\times32}{11\times10\times9\times8\times7\times6\times5\times4\times3\times2\times1}\)
We can make a lot of simplifications.
\(= \frac{\left(7\times6\right)\times41\times\left(10\times4\right)\times39\times\left(2\times19\right)\times37\times\left(9\times4\right)\times\left(5\times7\right)\times34\times\left(11\times3\right)\times\left(8\times4\right)}{11\times10\times9\times8\times7\times6\times5\times4\times3\times2\times1}\)
Crossing out factors in both numerator and denominator, the denominator disappears (reduces to 1), and we are left with:
\( = 41\times39\times19\times37\times4\times7\times34\times4 = 4280561376\)
Plugging this back in we get:
\( p_{sample} = \frac{\frac{29654190720}{40320} \times \frac{4896}{6}}{4280561376}\)
The two fractions in the numerator can also be simplified to:
\( p_{sample} = \frac{735471 \times 816}{4280561376}\)
\( = \frac{600144336}{4280561376}\)
Numerator and denominator are both divisible by 15504 so we get:
\(p_{sample} = \frac{38709}{276094} \approx 0.1402\)
Step 2: Determine the minimum and maximum of the top-left cell.
\(a_{min} = \text{MAX}\left(0, C1 + R1 - n\right)\)
\(= \text{MAX}\left(0, 11 + 24 - 42\right)\)
\(= \text{MAX}\left(0, -7\right)\)
\(= 0\)
For the maximum value of a:
\(a_{max} = \text{MIN}\left(R1, C1\right)\)
\(= \text{MIN}\left(24, 11\right)\)
\(= 11\)
Step 3: Go over all possible values of the top-left cell and calculate the probability. Add it to the p-value if it is less or equal to the one from the sample.
We start with the minimum option of \(a=0\). In that case:
\(b = R1 - a = 24 - 0 = 24\)
\(c = C1 - a = 11 - 0 = 11\)
\(d = R2 - c = 18 - 11 = 7\)
Now we calculate the probability of this arrangement:
\(p_{table} = \frac{\binom{24}{0}\times\binom{18}{11}}{\binom{42}{11}}\)
Note that the denominator is the same, and will be the same for all tables we are going to calculate, so we get:
\(p_{table} = \frac{\binom{24}{0}\times\binom{18}{11}}{4280561376}\)
\( = \frac{\frac{24!}{0!\times\left(24-0\right)!}\times\frac{18!}{11!\times\left(18-11\right)!}}{4280561376}\)
\( = \frac{\frac{24!}{0!\times24!}\times\frac{18!}{11!\times7!}}{4280561376}\)
\( = \frac{\frac{18\times17\times\dots\times12}{7!}}{4280561376}\)
\( = \frac{\frac{18\times17\times16\times15\times14\times13\times12}{7\times6\times5\times4\times3\times2}}{4280561376}\)
\( = \frac{\frac{\left(6\times3\right)\times17\times\left(4\times4\right)\times\left(5\times3\right)\times\left(7\times2\right)\times13\times\left(2\times6\right)}{7\times6\times5\times4\times3\times2}}{4280561376}\)
\( = \frac{17\times4\times3\times2\times13\times6}{4280561376}\)
\( = \frac{31824}{4280561376}\)
\( = \frac{3}{403522} \approx 0.000\)
The next option is \(a=1\). In that case:
\(b = R1 - a = 24 - 1 = 23\)
\(c = C1 - a = 11 - 1 = 10\)
\(d = R2 - c = 18 - 10 = 8\)
Now we calculate the probability of this arrangement:
\(p_{table} = \frac{\binom{24}{1}\times\binom{18}{10}}{\binom{42}{11}}\)
Note that the denominator is the same, and will be the same for all tables we are going to calculate, so we get:
\(p_{table} = \frac{\binom{24}{1}\times\binom{18}{10}}{4280561376}\)
\( = \frac{24\times43758}{4280561376}\)
\( = \frac{99}{403522} \approx 0.0002\)
The next option is \(a=2\). In that case:
\(b = R1 - a = 24 - 2 = 22\)
\(c = C1 - a = 11 - 2 = 9\)
\(d = R2 - c = 18 - 9 = 9\)
Now we calculate the probability of this arrangement:
\(p_{table} = \frac{\binom{24}{2}\times\binom{18}{9}}{\binom{42}{11}}\)
\( = \frac{\binom{24}{2}\times\binom{18}{9}}{4280561376}\)
\( = \frac{276\times48620}{4280561376}\)
\( = \frac{1265}{403522} \approx 0.0031\)
The next option is \(a=3\). In that case:
\(b = R1 - a = 24 - 3 = 21\)
\(c = C1 - a = 11 - 3 = 8\)
\(d = R2 - c = 18 - 8 = 10\)
Now we calculate the probability of this arrangement:
\(p_{table} = \frac{\binom{24}{3}\times\binom{18}{8}}{\binom{42}{11}}\)
\( = \frac{\binom{24}{3}\times\binom{18}{8}}{4280561376}\)
\( = \frac{2024\times43758}{4280561376}\)
\( = \frac{8349}{403522} \approx 0.0207\)
We keep doing this for all values of a until we reach the maximum of 11
a = 4 gives \(\frac{2277}{28823}\approx0.0790\)
a = 5 gives \(\frac{5313}{28823}\approx0.1843\)
a = 6 gives \(\frac{5313}{19721}\approx0.2694\)
a = 7 gives \(\frac{34155}{138047}\approx0.2474\)
a = 8 gives \(\frac{38709}{276094}\approx0.1402\)
a = 9 gives \(\frac{12903}{276094}\approx0.0467\)
a = 10 gives \(\frac{2277}{276094}\approx0.082\)
a = 11 gives \(\frac{23}{39442}\approx0.0006\)
The last thing to do, is add all the found probabilities, for those that were less or equal to the sample. In this example when 'a' is 0, 1, 2, 3, 4, 8, 9, 10 and 11.
\(\frac{3}{403522} + \frac{99}{403522} + \frac{1265}{403522} + \frac{8349}{403522} + \frac{2277}{28823} + \frac{38709}{276094} + \frac{12903}{276094} + \frac{2277}{276094} + \frac{23}{39442} \)
\(=\frac{39}{5245786} + \frac{1287}{5245786} + \frac{16445}{5245786} + \frac{108537}{5245786} + \frac{414414}{5245786} + \frac{735471}{5245786} + \frac{245157}{5245786} + \frac{43263}{5245786} + \frac{3059}{5245786} \)
\(=\frac{1567672}{5245786}=\frac{783836}{2622893} \approx 0.2988 \)
Interpreting the Result
The assumption about the population for this test (the null hypothesis) is that the two variables are indpendent (i.e. no influence on each other).
The test provides a p-value, which is the probability of a test statistic as from the sample, or even more extreme, if the assumption about the population would be true. If this p-value (significance) is below a pre-defined threshold (the significance level \(\alpha\) ), the assumption about the population is rejected. We then speak of a (statistically) significant result. The threshold is usually set at 0.05. Anything below is then considered low.
If the assumption is rejected, we conclude that the two variables are dependent in the population.
Note that if we do not reject the assumption, it does not mean we accept it, we simply state that there is insufficient evidence to reject it.
Writing the results
APA (2019) does not specify how to write the results of a Fisher test. It shows for most tests to report the test-statistic, the degrees of freedom (if applicable) and the p-value. However with an exact test as the Fisher test, there isn't a test-statistic. So we can only report the p-value.
So for example:
A Fisher exact test, indicated that the two variables had a significant dependency, p < .001.
The p-value is shown with three decimal places, and no 0 before the decimal sign. If the p-value is below .0005, it can be reported as p < .001.
APA (2019, p. 88) states to also report an effect size measure.
Next...
After the test an effect size measure could be determined. There too many to list. They are discussed on the Binary-Binary Effect Sizes page.
Alternatives
For a 2x2 table there are quite a lot of different tests. Upton (1982) discusses 24 of them, many of those are approximations.
The Fisher Exact Test is quite computational heavy, and for large values this could become a problem. In those cases a Chi-Square test is often used (Pearson or G). The chi-square test can also be used with more than two categories. There are also specific alternatives for the Fisher test. For example the Barnard test, Boschloo's test, Santner and Snell's test, and Suissa and Shuster's test.
Google adds