shklov, powell (2001) a frequency table analytical approach to observational data

8/2/2019 Shklov, Powell (2001) a Frequency Table Analytical Approach to Observational Data

1/10

A Frequency Table Analytical Approach to Observational Data

byNorm Shklov, Ph. D.

Department of Mathematics and StatisticsUniversity of Windsor

Windsor, Ontario, Canada(Retired)

andJay C. Powell, Ph.D.

Faculty of EducationUniversity of Windsor

(Retired)

Running Head: Frequency table data analysisSeptember 2001.


2/10

ABSTRACTHaving established clear evidence for a form of learning that is transformational and

discontinuous, the authors herewith present a way to consider frequency data in theeducational and social-sciences setting. This paper describes the procedure in detail and givesseveral instances of its application.

2


3/10

The ProblemThese two researchers have, for some time, been investigating non-traditional

approaches to observational data. The initial problem arose because they were trying to findsome way to examine, in multiple-choice tests, the patterns in which students' answers(whether right or wrong) change with learning. Their presumption has bel'that there may be adiscontinuous component to learning in addition to the linear cumulative one commonlyassumed.

As background to this problem there are two possibilities. The first possibility is thatall wrong answers are blind guesses and that students will change from any wrong answer tothe right one with appropriate instruction. In this case the frequent current practice ofconsidering only right answers when observing student behavior for quality control would beentirely appropriate. There would be no meaningful information to be found among thewrong answers.

The second possibility, which comes from Powell's teaching experience and fromPiaget's many years of clinical observation (see: Flavell, 1963), is that answers change insome sort of sequential order that is dependent upon how students interpret the questionsbeing asked. Their interpretations, in tum, could be dependent upon their current style ofthinking and their current depth of intellectual maturity.

In this case, learning would display discontinuous transformational properties(systematic answer changes) in addition to cumulative properties (increasing numbers ofcorrect answers). There might be considerable meaningful information in all answers, notmerely the right ones.

To look at change within the testing context, it is necessary to give the same test morethan once. When this is done, it becomes possible to compare the answers selected upon the

3


4/10

first administration to those selected on the second administration and build frequency tablesthat summarize these selections. There are two possibilities. First, the students may select thesame answer on both administrations. Second, they may change from one answer to anotherwithin each item.

If the first possibility (that students guess blindly until they know the answer) iscorrect, the typical observation should be changes to the right answer from any wrong onefollowed by systematic repetition of the right answer over time. No other combination ofselection should be statistically significant beyond a chance level.

If the second possibility is correct, then a much more complex pattern of answerstability and answer change should become evident as students transform their thinking withincreased maturity.

-


5/10

1) The number of times the members of the sample chose both members of the pairof responses being considered (the observed frequency),

2) The number of times this group chose the first member of the pair on the firstadministration (row or column sums),

3) The number of times this group of students chose the second member of this pairon the second administration (column or row sums), and

4) The total nwnber of students in the entire frequency table (N).In mathematical terms, the values being considered are the cell frequency, its

associated row and column sums, and the table sum.Unlike typical contingency tables, the concern here is with each cell instead of the

overall departure of the entire matrix from a l or some other generalized mathematicaldistribution. The problem being considered here is trying to decide how large (or how small)an observed frequency in any cell must be in order to indicate a meaningfl1l (statisticallysignificant) joint (or mutually exclusive) event.

Insert Table 1 about here

Suppose, as illustrated in Table l.a, that we have a joint choice of 11, a first variablefrequency of 15, and a second variable frequency of 19. If the group size is 23, the expectedjoint choice frequency is (15 x 19)/23 = 12.39. In this case, 11 is a lesser value than thisexpectation but may not be enough below this expected value to conclude that choosing oneoption implies the rejection of the other. If the group size is 90, as illustrated in Table l.b, theexpected frequency is (15 x 19)/90 = 3.17. Are 11 observations enough larger than thisexpectation to indicate a systematic joint choice?

.5


6/10

-tb' ::. investigation of the literature The nearest procedure that seemed to havecredibility was the proposal of Fuchs and Kennett (1980). Using a simulation of this problemwith 10,000 cases on two by two tables, the standard deviation from their proposed procedurewas consistently greater that 1.00. The observation indicates that their proposal has aconservative bias in which frequencies that should be accepted would be rejected.

In this same simulation for finding a way to put a probability value on one cell withknow marginal sums and any total frequency were employed. Shklov and Powell (1988)summarized the most important of these attempts, in which they compared, in simulatedconditions; the binomial, multinomial, hypergeometric and uniform distributions. Only themultinomial distribution gave a statistically significant fit with constrained marginal totals.

Historically, this problem ar-ose from the observation that, on highly discriminatingtest questions, par-ticular "wrong" answers tend to cluster at specific narrow ranges of thetotal scores. These answer tend to cluster around their modes of selection across an entiretest. This property is well known from item response theory (IRT) but the interpretation ofthese answers has been problematic.

To address this interpretation issue, two approaches are possible. First, interpretationscan be assigned to distracters based upon errors in information or logic. Powell (1970) andPowell and Isbister (1974) attempted this approach with mixed results. These classificationswere less stable, using linear statistical procedmes, than would be hoped fo If a majorimprovement in testing technology were to be achieved.

The second possibility would be to cluster these answers statistically and then attemptto provide interpretations from written reports ofreasoning or from interviews. Powell (1968)showed that, with adults, answers in the "wrong answer" set could be predicted fromreasoning reports about two thirds of the time. Using interviews with students from the third

6


7/10

through the eighth grades Powell (1977) found a Piaget-like sequence of interpretations ofthese answer clusters with a consistency ofmore than 50 % but less than 65 %.. This studyalso showed, as would be expected because these subsets clustered around their modes ofselection, a strong age-dependent sequence in their order.

There were unanswered question remaining from this latter study. First, how muchinfluence does the thought processes leading to the "wrong" answers have upon the "right"answers being observed. By predicting the total scores on another test from all of thesubscores on the test showing the Piaget-like sequence, Powell (1976) showed that the

"wrong" answer subscores were first in o r d e ~ frequently and explained more of thevariability than the "right" answer subscores (both concrete and abstract).

Second, ifthere is truly a Piaget-like developmental progression among theseanswers, then this lea.';ing sequence must be discontinuous and the changes of answers fromko n ~ n t e l l e c t u a l phase to the next should be appropriate, both in direction and by age level.,.Giving the test twice across the age range from less than eight to more than nineteen andusing the multinomial procedure (described in detail in this present paper) to establish whichcells were statistically significant, Powell and Shklov (1992) showed that both theseconditions were met. Of additional interest is the observation that although nearly every agelevel was highly statistically significant for the repeated selection of the "right" answer, Thistest-retest value accounted for only 23 % of all the answers pairs on this test. When the sumof the frequencies in all the significant cells was tabulated, 75 % of all the answer pairs wereexplained as meaningful (P ""7'0.945).

Discussions with other psychometricians suggested that some of the more recenteconometric techniques might serve this same purpose. Keswick (2001) undertook thisattempt. This is what he reported:

7


8/10

I applied a logistic regression, an ANOVA (using the categorizations - twoby two and four by four transition matrices), a Chi Square (again using thecategorization), and since the categories are ordinals I applied a tobit (which is just anordered probit analysis). No procedure explained more variance. Of greaterconcern, none really showed a direct detectable relationship.It would seem, therefore, that to detect discontinuities in learning that may influence

the scores we obtain for assessing student performance, some other approach than the typicallinear statistical analysis would seem to be required. One possible approach is the application

paper to provide the details of this procedure, with examples from different types ofof the adaptation we have developed of the multinomial procedures. It is the purpose of this

qualitative research.L. . - - -

/IThe Procedure

Step 1.This procedure begins with an m x n matrix of frequencies 0u as follows:

011 0 12 0 1" R10 21 0 22 O 2,, R2Oml Om2 Om" RmC1 C2 C" N

nwhere: R='L.oI IJ ')=1

m~ ='L. 0 IJ ,

i= ]

(1)

(2)

8


9/10

and:

Step 2

In nN= 2. 2. 0u.

i=lj=1(3)

The next step is to collapse this matrix around any cell 0u . This step produces a 2 x 2matrix, with the observed frequency of 0u being 0, of the sort:0 j R' , where: j=R ' j -o ,g h R'2 R'2 = N - R'I ,C'I C'2 N g= C', - 0 ,

C'2=N-C' I ,

and h = N - (0 + j+ g).In words,jis the frequency of the remainder of the row, g is the frequency of the remainderof the column and h is the residual frequency of the total table. Tables 2 and 3 give anumerical example of these first two steps in this procedure, begilming with the 4 x 4frequency matrix, such as:


By choosing to look at 0 22 (with a frequency of 0 = 33; the repeated choice of thecorrect answer as shown in the box in Table 2) the collapsed matrix becomes:


In order to determine the probability that 0 would be 33 by chance alone, it isnecessary to find two cumulative probabilities. First, it is necessary to find the cumulative

9


10/10

probability for the range of all possible values in that cell with the tlu'ee marginal values keptconstant. Second, there is the need to find the cumulative probability for all single tables thathave a value of 33 or less.

To achieve this end, it is necessary to find the smallest possible value and the largestpossible value that 0 can achieve within these marginal constraints.

Since these values are all frequencies, and the least possible frequency is zero (0), theleast possible value that 0 can possess is when either the frequency value of0 or ofh is zero(0). Similarly, the greatest possible value that cell 0 can contain occurs when either thefi'equency of f or of g is zero (0).Mathematically:O(min) = 0, if 0 :::; h; otherwise O(min) = 0 - ho(max) = the lesser ofR' I and C' I

(4)(5)

In the present case, the minimum value is 0 (zero) and the maximum value is 46.

Step 3The multinomial equation is applied to the range of all possible values from the

minimum (0) to the maximum (46), adding these partial probabilities to find the totalpossible probability (PI )o f events within these constrained conditions. The equationbecomes:

(6)O(min)

[0

shklov, powell (2001) a frequency table analytical approach to observational data

Documents