Measuring the results of teaching

Front matter

mm

(ilass .

Book

Preface

This book is written for the teacher in the elementary school. As such it is not intended to be a fundamental treatise upon educational measurements, but rather a text which will help teachers to use standardized tests to the greatest advantage. Only certain ones of the available standardized tests are described. This was thought to be a more helpful plan than to include all of the available tests, because the elementary teacher seldom has at hand the information necessary for an intelligent selection of standardized tests. The value of a number of the tests described in this text has been demonstrated by wide usage. Others are just being made available for use. In these cases it has been necessary for the author to exercise his judgment based upon four years' experience in supplying tests to teachers and superintendents through the Bureau of Educational Measurements and Standards of the Kansas State Normal School, Emporia, Kansas. Some worthy tests have been omitted partly because of the limitations of space and partly because of other considerations. Other worthy tests will doubtless be devised in the future, some of them replacing certain of the tests chosen for description in this text.

The feasibility of the test being used by teachers who have not had special training in the field of educational measurements has been kept constantly in mind. Detailed directions for the use of a test are generally not reproduced, for they are furnished with the test when purchased for class use. Only those general features which are necessary for understanding the tests and the method of handling the x PREFACE

results have been given. It is believed that any teacher who studies carefully the descriptions given in this text will have no difficulty in using any of the tests described.

It is the contention of the author that the use of a standardized test is justified only when the teacher can use the resulting measures as a basis for improving instruction. Consequently much space is given to the interpretation of scores or measures and the corrective instruction which should be given to correct unsatisfactory scores. Unfortunately little is known about corrective measures for certain school subjects. This, however, is a condition which time will remedy.

The author is aware that in a sense he has done little more than bring together the results of a number of workers in this field and he realizes his indebtedness to them. He is particularly indebted to Dean F. J. Kelly and Captain J. C. DeVoss who kindly permitted him to use portions of their chapters in Educational Tests and Measurements.

Walter S. Monroe Bloomington, Indiana

Contents

I. The Inaccuracy of Present School Marks . • 1 XL TnE Measurement of Ability in Reading ... 22

III. The Meaning of Scores and Correcting Defects

in Reading 43

Chapter I The Inaccuracy Of Present School Marks

The measurement of results not new in education. Educational measurements are not new in school work, although this name has not been applied to them until very recently. Since schools have existed, teachers and other school officials have attempted to measure the abilities of pupils by estimating daily recitations and by examinations. The measures of the abilities of pupils obtained in these ways are thought to possess a high degree of precision and are considered very important.

The promotion of pupils depends upon the " grades" they receive. The ability of a pupil in each of the subjects is measured by the teacher's estimate and by examination, and if the resulting measures show the pupil to be a few points, or in some instances a fraction of a point below the "passing mark," the pupil is classified as a failure. If the resulting measures equal or are above the "passing mark," the pupil is promoted.

The "grades" or school marks are entered upon the monthly or quarterly report cards. Parents, as well as teachers and pupils, take these school marks very seriously. If Johnnie's "grades" for a given month are below those of the preceding months, or, worse still, if they are below those of neighbor Smith's Mary, an explanation is demanded. A permanent record is kept of at least the yearly "grades," and the awarding of school honors is based upon it.

Until recently, practically all admission to college was determined jj by examination. Except in the universities and colleges of the Central and Western American States, the custom still maintains generally throughout the world. This practice is based on the assumption that the examining committee can determine thereby the effectiveness of the candidate's college preparatory work. The civil service, from its inception in China centuries ago until the present day, has employed the examination as a means for measuring the ability of persons who desire positions operated under this system.

The use of scientific tests and standards is new. Although the measurement of the results of instruction is not new, it should be recognized that the use of tests which have been scientifically constructed and the interpretation of the resulting measures by comparison with standards is one of the most recent educational developments. Thorndike, " the father of this movement," has discovered what is probably the earliest record of this new type of educational measurement. The date of this publication, which was by an English schoolmaster, is 1864. In our own country, Rice's report on spelling in 1897 marked the beginning of the movement, but except for this and a few other pioneer efforts, the development has been confined to the last ten years. Within this period those who ridiculed the work of Rice have been converted to educational measurements, and standardized tests are now generally recognized as one of the most helpful instruments at the command of the teacher and the supervisor.

Recent investigations have shown school marks to be inaccurate. One of the most important factors contributing to our present use of standardized tests has been a number of investigations made to ascertain the accuracy or reliability of measures obtained by means of teachers' estimates and by means of examinations. In the world of physical things we measure distance by means of the yardstick, mass by means of scales, the volume of liquids by means of gallon measures. Measurements of these magnitudes, when made carefully with accurate instruments, possess a high degree of reliability. By a high degree of reliability we mean, for example, that if two persons measure the length of the same room by means of the same yardstick or any other yardstick, the two measurements will be approximately equal. If they differ by more than one or two inches, we doubt the accuracy of both, and we demand that the room be measured again. Similarly, in the case of school-children, if we find that, when the same children are measured in the same subjects by two different teachers, the two sets of measures do not agree rather closely, we have reason to doubt the accuracy of both sets of measures. On the other hand, if the two sets of measures ("grades") agree closely, we have reason to believe them accurate or reliable.

In this chapter we present evidence from three types of investigations which show that marks given by teachers under ordinary conditions are not accurate measures of the abilities of their pupils: (1) Kelly's investigation based upon the final "grades" given to pupils in two successive years by different teachers; (2) Johnson's investigation based upon the distribution of "grades"; (3) the marking of examination papers.

(1) Kelly's investigation. In 1913, Kelly ! made an investigation of the marks given to the sixth-grade pupils in four ward schools in Hackensack, New Jersey, and the marks given to the same pupils when they went to a common

1 Kelly, F. J., Teachers' Marks. (Teachers College Contributions to Education, no. 66, p. 7.)

departmental school for seventh-grade work. This will be recognized as a case where the abilities of the same pupils were measured by two different sets of teachers, the sixth-grade teachers in the ward schools and the seventh-grade teachers in the departmental school. Since in the departmental school all of the pupils were taught arithmetic by one teacher, there was an opportunity to compare the "grades" given in arithmetic by the sixth-grade teachers in the different ward schools. If these teachers were accurate in their "grading," we would expect to find that all of the pupils who received a mark of "G" (good) in arithmetic in the sixth grade would receive approximately the same mark in the seventh grade. If, however, the sixth-grade teachers were inaccurate in their marking, — that is, some of them marked too high or too low, — we would expect to find that pupils having the mark of " G" in the sixth grade, but coming from different schools, would, on the average, receive different marks in the seventh grade. This condition was found to exist.

Kelly states his conclusions as follows:

This means that for work which the teacher in school " C " (one of the ward schools) would give a mark of "G" (good) in language, penmanship, or history, the teacher in school "D" (another ward school) would give less than a mark "F" (fair).

{2) Johnson's investigation. Another type of investigation has been made by Johnson, I Principal of the University High School of the University of Chicago. It is based upon the fact that when accurate measurements are made of any ability of a large group of pupils, the resulting measures are distributed; that is, arranged along the scale of measurement, in a certain definite way. For example, in Fig. 1 there are*

1 Johnson, F. W., "A Study of High School Grades"; in School Review, vol. 19, pp. 13-24. See also Kelly, F. J., Teachers Marks, p. 11, and following, for reports of similar investigations.

represented graphically four distributions of the measures of silent reading ability secured by giving the Kansas Silent Reading Tests. The number of measures represented in each grade is over 5000. The base line of the curve in each rttnvi t h ids fa a

4>* <&**/»»

7* iwwfc.

M *ljS ^' S g^t

Fig. 1. Showing the Distribution op Measures of Silent Reading Ability as measured by the Kansas Silent Reading Tests.

case represents the scale of the test, 0, 1, 2, 3, 4, 5, and so on. At any point of this base line the height of the broken line curve above the base line represents the number of pupils having the measure represented. The general shape of these four broken line curves is the same. A few pupils received very low measures and a few very high ones. The great majority of the measures are grouped near the middle where the curve is highest. A curve which, beginning with the low measures, rises gradually and then falls gradually, as do those shown in Fig. 1, is called a " normal curve" and represents the shape of the distribution when accurate measurements have been made. If the shape of the curve representing the distribution of a particular set of measures differs materially from the general shape of the curves in Fig. 1, there is reason for questioning the accuracy of the measures.

In the University High School, "F" denotes failure, and the four successive ranks above failure are indicated by "D," "C," "B," and "A." For the several departments of the school, Johnson tabulated the number of times each mark was given during the years 1907-08 and 1908-09. The conditions which he found to exist may be illustrated by Fig. 2. The upper figure shows the distributions of marks in English (left) and history (right) . It will be noted that in the case of English a much larger proportion of low marks ("F" and "D") were given than in history. For the high marks ("A" and "B") just the reverse is true. Both curves fail to conform closely to the normal curve described above which suggests that the marks may not represent accurate measures.

However, the most striking part of the figure is the lower which represents the distributions of the marks of two teachers in the same department. The distribution for teacher A conforms reasonably close to the normal curve, but that for teacher B departs from it in a very conspicuous fashion. It is obvious that teacher B is accustomed to give "high grades." In so doing he has furnished evidence that his marks are probably inaccurate.

(3) Marking examination papers. The written examination is the most common means of measuring the abilities of pupils, although many teachers and school patrons oppose its use. They contend that pupils working under pressure frequently become nervous and confused and consequently cannot do themselves justice, while other pupils, who have no real grasp of the subject, are able by cramming to write

£n*lisK

History

Teach erA

Teacher B

Fig. 2. (Upper.) Showing Distribution Marks in University of Chicago High School in English and History. (Lower.) Showing Distribution of Marks of Two Teachers in the Same Department. (After Johnson.)

excellent papers. It is also contended that the questions are frequently not well selected and do not pertain to the essentials of the subject.

There is probably some truth in the above assertions, but within the past few years there have been a number of investigations to ascertain if teachers mark examination papers accurately, assuming that what appears on the papers is a true record of the abilities of the pupils. Starch and Elliott l investigated the accuracy with which teachers marked papers in English, geometry, and history. Their method and the facts revealed may be illustrated by the case of geometry.

A facsimile reproduction was made of an actual examination paper in plane geometry. A copy of this reproduction was sent to each of the high schools included in the North Central Association of Colleges and Secondary Schools, with the request that it be marked on the scale of one hundred per cent by the teacher of geometry. The teacher was asked to mark the paper by the method he was accustomed to use. Papers were returned from 116 schools, and the results tabulated. When we consider that the subject-matter of geometry is quite definite, and that the papers were marked by teachers who were thoroughly acquainted with the subject, it would seem that we might expect the marks or "grades" placed upon this examination paper to be in close agreement. However, exactly the opposite was the case.

Distribution of marks. The distribution of the marks is shown in Fig. 3. The scale is marked on the base line and the number of dots above any point indicates the number of teachers who gave the indicated "grade." Thus the "grade" of 75 was given by thirteen teachers, the "grade" of 76 by three teachers, and so on. Of the 116 marks, two were above 90, while one was below 30. Twenty were 80 or above, while twenty other marks were below 60. Fortyseven teachers assigned a mark passing or above, while sixty-nine teachers thought the paper not worthy of a passing mark.

1 Starch and Elliott, "Reliability of Grading High-School Work in English"; in School Review, vol. 20, pp. 442-57; "Reliability of Grading Work in Mathematics"; in School Review, vol. 21, pp. 254-59; "Reliability or Grading Work in History"; in School Review, vol. 21, pp. 676-81.

Not only were similar results obtained by Starch and Elliott in English and in history, but other investigators l have verified them many times. In the face of such facts only one conclusion is possible; namely, that under ordinary conditions the marks assigned to examination papers by teachers are very unreliable. Such marks can represent only very crude and very inaccurate measures of the abilities of pupils. It is not too much to say that the mark which a

• •••• ••••

Fig. 3. Distribution op Marks assigned to one Geometry Paper by 116 Teachers.

Passing grade 75. Range 28 to 92. Marks assigned by schools whose passing grade was 70 were weighted by 3 points. Median 70. Probable 7.5.

pupil receives on an examination paper depends upon the teacher who "grades" the paper, as well as upon what the pupil places upon the paper.

It has also been shown that the same teacher is not consistent in his own marking. If a set of papers are marked a second time, the two sets of marks will vary widely. 2

Summary. We have now presented an illustration of each of three types of evidence that teachers' marks, both final "grades" and examination "grades," are inaccurate. In each case the illustration is typical of a number of similar ones which might be mentioned. We have, therefore, a

1 See Kelly, F. J., Teachers* Marks, p. 51, and following, for accounts of other investigations.

2 See Starch, Daniel, Educational Measurements, p. 9.

large amount of evidence that teachers* marks are not accurate. We shall next consider two causes for the errors in marking examination papers.

Conditions which contribute to the inaccuracy in marking examination papers. (1) Error due to unequal value of questions. A critical study of examinations and of the manner of giving them reveals certain conditions which contribute to the inaccuracy in teachers' marks. In the first place, the questions are generally considered equal in value, but if we judge the value of questions on the basis of their difficulty as shown by the responses of the pupils, it is seldom that the same credit should be given for answering correctly two different questions. As evidence of this consider the following questions taken from an examination in United States history. 1 The number following the question is the per cent of pupils who answered the question correctly.

To what religious body did most of the settlers of Pennsylvania belong? 62 . 3

What critical problem arose during Buchanan's administration? 7.0

What is the main purpose of the Monroe Doctrine? %5.5

These differences in the per cent of correct answers are merely typical of what is very likely to be the case in any examination prepared by the teacher. The questions will not be equally difficult, and it is the general practice to base the credit given for a correct answer upon the difficulty of the question : that is, less credit is given for answering correctly an "easy" question than for a "hard" one.

It is easy to understand how a serious element of error is introduced when each question is considered to have a value of ten points and the questions are not equal in difficulty.

1 Buckingham, B. R., "Survey of the Gary and Prevocational Schools," Seventeenth Annual Report of the City Superintendent of Schools (New York City), 1914-15.

The situation is much the same as we should have in measuring distances if yardsticks of different lengths were used, but were considered to be equal. Under such circumstances a yard would have no definite length, and to say that a certain distance was 21.42 yards would convey no definite information about it. For this reason the Federal Government has standardized all weights and measures by establishing definite units, and before we can obtain definite measures of the abilities of children, it will be necessary to devise tests consisting of standard units: that is, the questions or exercises composing the test must be evaluated.

A teacher's estimate of the difficulty of questions is unreliable. Can a teacher judge of the difficulty of a problem or even arrange a list of problems in order of difficulty? One investigator l studied this question by submitting the following list of twenty- three problems to twenty teachers who were asked to estimate the per cent of pupils who would solve each problem correctly if given ten minutes for each. From this information it was possible to determine which problem each teacher considered easiest, which second in difficulty, and so on. The results of these teachers' judgments are given in Table I.

1. How much change should I expect from $5, after paying for 5 pounds of coffee at 38 cents a pound?

2. If $1991 a day is paid to 724 men who each earn the same wages, how much does each man receive?

3. A boy had 210 marbles. He lost 1/3 of them. How many were left?

4. A grocer had a tank holding 44 3/16 gallons of oil. One day he drew out 15 3/4 gallons and the next day 9 1/8 gallons. How many gallons were left in the tank?

5. There are 550 pupils on the roll. If 5/8 of them are here to-day, how many are absent?

1 Comin, Robert, "Teachers' Estimates of the Ability of Pupils"; in School and Society, vol. 3, p. 67, January 8, 1916.

6. If 3/4 of a pound of cheese is sold for 45 cents, how much can be bought for $1?

7. A storekeeper sold 12 yards of cloth, which was 4/15 of the whole piece. How many yards in the whole piece?

8. A baseball team played 160 games during the season and won 100 of them. What part of the whole number of games did the team win?

10. A man bought a house for $7250. After spending $321.50 for repairs, he sold it for $9125. How much did he gain?

11. A reader has 29 lines on a page and in all 10,034 lines. How many pages in the book?

12. A boy lost one fourth of his kite string in a tree, one third in some wire, and one fifth in a hedge. What part of his string was left?

13. How much will 8 3/4 dozen pencils cost at the rate of $1/4 for half a dozen?

14. If it takes a train three quarters of an hour to reach a certain station, what fraction of an hour will it take the train to go 3/5 of the distance?

15. A man has a salary of $125 a month. He saves 20 per cent of his salary. How much will he save in a year?

16. A workman pays $22 a month for board, which is 20 per cent of his wages. What are his wages?

17. Mr. Marshall receives a salary of $2500 a year. His rent costs him 1 /3 of this and his other expenses are $1500. He saves the rest. What per cent of his salary does he save?

18. John had $1.20 Monday. He earned 30 cents each day on Tuesday, Wednesday, Thursday, and Friday. Saturday morning he spent one third of what he had earned in the four days. Saturday afternoon his father gave John half as much as John then had. How much did his father give John?

19. A boy had $3. He paid it all for four articles, which we will call A, B, C, and D. B cost as much as D. A cost as much as B, C, and D together. The boy sold A and B for 1 1 /2 times what he paid for them. He sold C and D f or 1 1 /4 times what he paid for them. How much did he get for the four articles?

20. A party of children went from a school to a woods to gather nuts. The number found was but 205, so they bought 1955 nuts more from a farmer. The nuts were shared equally by the children and each received 45. How many children were there in the party?

21. One summer a farmer hired 43 boys to work in an apple orchard. There were 35 trees loaded with fruit and in 57 minutes each boy had picked 49 apples. If in the beginning the total number of apples on the trees was 19,677, how many were there still to be picked?

22. A girl found that by careful counting there were 87 letters more on a page of her history than oa a page of her reader. She read 31 pages in each book in the first 29 days of school. How many more letters each day did she read in one book than in the other?

23. The children of a school made small boxes to be filled with candy and given as presents at a school party. Six hundred boxes were needed. In 4 days grades 3 to 7 made 20, 25, 83, 150, and 150 boxes. The eight grade agreed to make the rest. How many did the eighth grade make?

Table I. Showing the Rank of Problems by Teachers' Judgments

Bank

Problems

4 5

2 2 1 2 2

'4 3 4 3 2 1

i

12 4

'i

l l l

*2

i l

3 2

3 2

1 1 1

'i

i

3 1

2 3 1

1 1 i

"i

3 1 4 4 1 1 3

i

*2

"i

'i

l

1 1

i s i

3 2

'4

2 1 1 1 2

1 2 1 2 3 2 2 1

"i l l

i

'i

l

'4

3 2

'2

'4

'i

'4

2 4 2 3

'i

'i

i

'5 2 3

i 3 i

1 2 1 4 2 3

'3

'2

i

'2 1

2 1 1

'2 2 i

'i

3 3 2 3 2 1 1

'2

'i

1 2 2 1 1 1 1 1 1 i

'i

'4 2 2 is

"i

i

'3 2 3 3 5

'i i i 2

i

'3 2 1

"<6 1 1

i

6 6 1

5 5 4 3 1

1 2 3

. 18

'i

•2

i

1 1 1

'2

4 3

i

Table H. Problems ranked according to Real Difficulty and Teachers' Estimates

. hf

Bank

Average rani.

OOlCT/h

(real difficulty)

teachers' estim

16.5

15.5

16.5

15.5

Table I shows that four teachers judged Problem 1 to be the easiest, five judged it to be second in difficulty, two judged it to be third in difficulty and so on. The remarkable thing about this table is the lack of agreement in the judgments of the teachers. The problems are not unusual. The list or a portion would make a typical examination. It should be clear that if a teacher giving such a list attempts to assign values to the several questions, the values thus assigned are likely to be inaccurate. It is doubtful, except in extreme cases such as Problem 19, whether the values will be more accurate than if the questions are considered to be equal in value.

The problems were given to the pupils in the fifth, sixth, seventh, and eighth grades in one school. The total number of pupils was about 1500. By doing this it was possible to determine what problem actually was easiest, and the order of difficulty for the entire list. The average rank of each problem as determined by the pupils' scores, and the rank as determined by the average of teachers' opinions, are given in Table II. This table presents some interesting facts. Twelve of the twenty teachers agreed that Problem 3 was easiest and no one ranked it above seventh in difficulty. Its real rank was found to be tenth. Similar discrepancies can be pointed out for other problems, although in the case of certain problems the average of the teachers' estimates approximates the real rank. Thus even the average judgment of twenty teachers on the relative difficulty of problems is not reliable and the judgment of a single teacher is much less reliable. (#) Rate of doing work neglected. In the second place, it is customary in giving an examination to allow sufficient time for all pupils to answer all of the questions, or if this is not done, the papers are graded on the basis of what each pupil has done. This manner of giving an examination fails to take into account the rate at which a pupil is able to answer the questions. Only the quality of the answers is considered, and the pupil who answers the questions with difficulty, and who barely finishes in the time allowed, receives exactly the same "grade" as the more capable pupil who is able to answer the questions easily and who finishes in one half or one third of the time, providing the two sets of answers are equivalent. It is clear that when this is done, the "grade" or mark which the pupil receives is not a true measure of his ability, because the rate at which he is able to do work is a "dimension" of his ability as well as the quality of what he does. In certain cases the rate may be a relatively unimportant dimension. Neglecting it in measuring the ability of a pupil is much like neglecting the width in measuring a rectangle to determine its area.

Some may insist that it is unfair to the slow-working pupil not to allow sufficient time for him to answer all of the questions. However this may be, it certainly is unjust to the more capable pupil to deprive him of the opportunity to demonstrate what he is able to do. This is exactly the case when the work asked of him is sufficient to keep him employed only a half or a third of the period allowed for the examination. This practice of ignoring the rate of working probably tends to cause desultory and careless school work.

Investigation has shown that rapid work and a high degree of quality or accuracy are not incompatible in arithmetic. The same statement can be made with reference to reading. Investigation has indicated that a considerable per cent of pupils can be made more accurate in arithmetic by forcing them to work more rapidly. It has also been shown that about three pupils out of four make progress in rate of work and accuracy at the same time. In view of these facts, it appears that good instruction requires that the teacher give attention to the rate of doing work as well as to the quality of the work done. The rate at which a pupil is able to do work of a given quality is as much a factor of his ability as is the quality of the work which he does.