Frans Coenen

Department of Computer Science

The University of Liverpool

13 February 2004


1. Overview.
2. Example.


Chi-squared tersting is used to determine whether two variables are independent of one another. In Chi-squared testing we compare a set of observed values (O) against a set of expected values (E) --- values that would be expected if there were no association between the variables. We calculate a value, chi2, using the identity:

chi2= sum of ((0-E)^2/E)

If the result is above a given critical threshold value then we can say that there is a relationship between the varaiables, otherwise there is no relation.

Given a classification rule we can determine whether the rule is surprising (i.e. unexpected) or not by determining whether there exists some special relationship between the attributes and the classifier, or that the rule is simply one that we might expect assuming a normal (chi-squared) distribution.


Given a rule A -> c such that:

support(A)         =  8  
support(c)         = 12 
support(A + c)    =   6 
N (num of records) = 32

We can produce a contingency table (referred to by some authors as a confusion matrix) of observed values as follows:

c !c
A sup(A+c) --- The number of true positives sup(A)-sup(A+c) --- The number of false negatives sup(A) --- The number of +ve records
!A sup(c)-sup(A+c) --- The number of false positives N-sup(A)-sup(c)-sup(Ac) --- The number of true negatives N-sup(A) --- The number of -ve records
sup(c) --- The number of records covered by the ruleN-sup(c) --- The number of records NOT covered by the ruleN

A 6 2 8
!A 61824

The above contingency table iis a generalisation of a confusion matrix

The expected values are then calculated thus:

A (12*8)/32=3 (20*8)/32=5 8

The Chi-squared value can then be calculated as follows:

O EO-E(O-E)^2((O-E)^2)/E
6 3 393.0
6 9 -391.0
2 5 -391.8
1815 390.6

To determine whether this Chi-squared value is significant or not we must know a critical value for Chi-squared. These are usually published in tables with signifcance level along the X-axis (expressed as a precentage) and the degrees of freedom (DF) along the Y-axis. DF is calculated as follows:

DF = (num rows in O or E table-1) *
	(num columns in O or E table-1)

In the case of classification rule mining this is always going to be:

DF (2-1)*(2-1)=1

Some critical values for DF=1 and a range of signifcance levels are given below. Usually a significance level of 5% is chosen, in which case the Chi-squared value for the above rule of 6.4 can be said to be significant (interesting or surprising).

10% 5% 2.5% 1% 0.5%

Created and maintained by Frans Coenen. Last updated 09 February 2001.