How to measure in education

Preface

The science of measurement is in its infancy and the art of measurement is younger still. Yet both have developed at such a phenomenal rate in the last few years as to make this movement for the mental measurement of children the most dramatic tendency in modern education. Educators returning from a few years' sojourn in foreign missionary fields testify that no recent change in educational practices compares with that effected by the growth of scientific educational and intelligence testing. During the war, and since, it was my privilege to confer with educational or military commissions sent here by several foreign governments to study our schools. These representatives testified that the extensive use of scientific mental measurement was one of the most distinctive features of American education. This book describes rather fully the meaning and methods of this movement.

The art of measurement is younger than the science of measurement because the abler workers have, of necessity, devoted their energies almost exclusively to the origination of foundational techniques. But the whole movement was so promising in the way of concrete assistance in meeting educational problems that practical educators have irresistibly demanded that the science of measurement be turned into the art of measurement almost overnight. Hence the last two or three years has witnessed a feverish effort to meet these demands. The result has been numerous mistakes and remarkable successes.

This book has a fourfold aim. First, it aims to present these successes and warn against a repetition of the mistakes.

Secondly, it aims to help forward the movement for mak-

^j Preface

im teaching a genuine profession. Effective teaching requleTa skll and a command of refined P--dure m ^^ of that needed by the physician or surgeon. The medical profession is a genuine profession just because its members are experts in refined procedure. This book is one of the two which I have planned to show that tangible learnable, refined techniques are possible in teaching and that even a ^^born" teacher will not, in a few years, be able to compete with a professionally trained teacher.

In the third place this book aims to meet the needs ot the educators who are interested in both the How and the Why. Extremely elementary books in this field have done an admirable service in diffusing an interest in mental measurement. The more intelligent teachers are now asking, however, for a book which not only brings the art of measurement up to date, but which, in addition, goes sufficiently into the science of it that they may be able to use tests less blindly than heretofore.

The final aim of this book is to bring together in one convenient volume most of the techniques needed by those engaged in mental measurement. At present the worker in this field must go to one book to learn how to construct a mental test, to another book to learn how to give and use the results of the test, to another book to learn how to apply statistical methods, and to still another book to discover methods for graphic and tabular presentation. The expert will and should continue to consult these special treatises. Others do not like to give the necessary time. This book then, is really several small books in one. Furthermore, it has been so arranged that the reader can confine himself to the earlier and less technical portions or he can omit this and study the more technical chapters placed toward the end.

As I think through the contents of this book and reflect upon the influences which have made it, I am keenly conscious that it is both autobiography and biography. It is autobiography because several years of my own thought and

Preface vii

labor and much experimentation in the world's most stimulating educational laboratory, Teachers College and New York City, have gone into it.

This book is a biography because from cover to cover I see the tangible and relatively intangible evidences of my many teachers. My pupils have been my teachers. My contemporaries in mental measurement have been my teachers. In so far as my mention can accomplish it, I wish, however, to honor particularly five individuals. The first is, and has been, an elementary school teacher in the mountains of Kentucky and Tennessee. The second is President Emeritus of Cumberland College, Williamsburg, Ky. The third is President of Lincoln Memorial University, Cumberland Gap, Tennessee. The influence of the fourth is responsible for my interest in realms which transcend those represented in this book. The ideas of the fifth are so completely a part of me that it might be said that he wrote the book through an imperfect medium. They are Jesse Worley, E. E. Wood, George A. Hubbell, Frank M. McMurry, and Edward L. Thorndike.

Wm. a. McCall

PART ONE How TO Use Measurement

CHAPTER

PAGE

I Place of Measurement in Education ... 3

II Measurement in Classifying Pupils ... 19

III Measurement in Diagnosis 67

IV Measurement in Teaching 112

V Measurement in Evaluating Efficiency of

Instruction 149

VI Measurement in Vocational Guidance . . . 169

Part Two

How to Construct and Standardize Tests VII Preparation and Validation of Test Material 195 VIII Organization of Test Material and Preparation OF Instructions 227

IX Scaling the Test 249

X Scaling the Test — T Scale 272

XI Determination of Reliability, Objectivity,

AND Norms 307

PART THREE Tabular, Graphic, and Statistical Methods XII Tabular Methods 321

XIII Graphic Methods 331

ix

X.

Contents

CHAPTER PAGE

XIV Statistical Methods — Mass Measures . . . 354 XV Statistical Methods — Point Measures . . . 365 XVI Statistical Methods — Variability Measures . 378 XVII Statistical Methods — Relationship and Reliability Measures 388

Appendix

How TO Secure Tests and Directions for Their Use . 409

Index 411

LIST OF TABLES

TABLE PAGE

1. Retardation and acceleration in towns and cities 23

2. Classification of pupils on the basis of educational age and E. Q. 26

3. For converting pupils' composite scores on educational tests into

educational ages . . 34

4. Reclassification and placement table 47

5. Distribution of changes made in reclassifying a school , . .51

6. Distribution of changes made in reclassifying another school . 52

7. Effect of specially promoting pupils 53

8. Data needed to guide instruction in reading 67

9. For transmuting number of questions correct into T scores . . 72

10. Grade norms on the Thorndike-McCall Reading Scale ... 72

11. For transmuting T scores into reading ages 73

12. Sample tabulation form for Thorndike-McCall Reading Scale . 74

13. Scores made on an informal silent reading test 137

14. Efficiency measurement with several standard tests 158

15. Reading efficiency of Baltimore white and colored schools . . 161

16. Interpretation of efficiency by means of grade unit and relative

position 162

17. Construction of percentile table 254

18. Computation of percentile score 255

19. Computation of age score and E. Q 257

20. For converting per cents correct into P. E. distances .... 259

21. Illustrating conversion of per cents correct intO' P. E. distances . 259

22. Conversion of per cents of better judgment into P. E. differences

in merit 266

23. For converting per cents into S. D. distances 274

24. Shows how to scale total scores 279

25. Age distributions for the Thorndike-McCall Reading Scale . . 280

26. Comparison of equal-step and unequal-step scales 302

27. Simple-total vs. cumulative-total method of combining units . . 303 28 Illustrates proper construction of a table 326

29, Illustrates an effective method of tabular presentation . . . 329

30, Spelling scores made by 64 fourth grade pupils . . . . . . 354

31, Shows frequency distributions of the same data grouped in step

intervals of 2 and step intervals of 4 361

32 Computation of mean for scores ungrouped and grouped in step

intervals of i 357

32 Computation of mean for scores ungrouped and grouped in step

intervals of 10 369

xi xii List of Tables

TABLE PAGE

34. Computation of Qi, median, and Qa for scores ungrouped and

grouped in step intervals of i 371

35. Computation of Qi, median, and Q3 for scores ungrouped and

grouped in step intervals of 10 373

36. Computation of Qi, median, and Qs when step intervals are unequal 375

37. Computation of Qi, median, and Qs when there are zeros in the

frequency column 376

38. Computation of mean deviation for scores ungrouped and grouped

in step intervals of i ' 381

39. Computation of mean deviation for scores ungrouped and grouped

in step intervals of 10 383

40. Computation of S. D. for scores ungrouped and grouped in step

intervals of i 384

41. Computation of S. D. for scores ungrouped and grouped in step

intervals of 10 386

42. Computation of / 390

43. Computation of i? 392

45. How to interpret r 394

46. Summary of statistical results 400

47. Conversion of experimental coefficient into statement of chances 405

48. Statistical problems and answers 407

LIST OF DIAGRAMS

FIGURE PAGB

1. Comparison of E. Q.'s and I. Q.'s 41

2. Overlapping of educational ages for two adjoining grades . . 43

3. Estimation of true age means . . 284

4. How to fill out individual record card for one of Courtis' Standard

Supervisory Tests 321

5-21. Illustrating standard methods for graphic presentation by means

of bar and curve diagrams 334

22. A sector diagram 345

23. A sectioned-bar diagram 34^

24. A combination bar-and-sectioned-bar diagram 347

25. Frequency surface showing identification of components . . . 349

26. An approximately normal frequency surface 355

27. A normal frequency surface 356

28. Minus skewed frequency surface 357

29. Plus skewed frequency surface 357

30. Multi-modal frequency surface 359

31. A frequency surface with step intervals of i 360

32. A frequency surface with step intervals of 2 . . . . . . 360

33. Rectilinear relationship with an r of .303 395

34. Rectilinear relationship with an r of .8 395

35. Rectilinear and curvilinear relationships 39^

Chapter I Place Of Measurement In Education Thesis I. "whatever Exists At All, Exists In Some Amount'^

It is possible to become so immersed in the details of the measurement of pupil achievement as to lose sight of its fundamental significance. It is this absence of perspective which is responsible for two educational afflictions-|-the lop-sided enthusiast for the scientific measurement of education, and the^qually unbalanced opponent of the movement. Educational measurement has a sort of philosophy. I have attempted to condense the main elements of this philosophy into a series of theses which may help those who have not had much time to think along these lines to appreciate the true place which measurement should have in education. The first of these theses is stated above.

Since all sane persons accept this thesis it needs no qualification, but a qualified thesis will suffice for our purpose, namely, whatever change the teacher makes in a pupil must be a change in an amount of something. We teachers will scarcely insist that our effort makes no change in amount. Even though such were the result of our effort it would not so much disprove the thesis but rather prove our own inefficiency.

There is an ever-dwindling group who strenuously oppose

^ E. L. Thorndike, The Seventeenth Year Book of the National Society for the Study of Education, Part II, p. i6; Public School Publishing Co., Bloomington, 111.

the practical implications of the above thesis. They claim to be interested in the emancipation of education from the quantitative idea. Their effort is directed toward the qualitative in education. According to them there is in every person a non-quantative quality — a

"... something far more deeply interfused, Whose dwelling is the light of setting suns."

Did they truly "see into the life of things" they would realize that there is never a quantity which does not measure some quality, and never an existing quality that is nonquantitative. Even our halos vary in diameter.

THESIS 2. ANYTHING THAT EXISTS IN AMOUNT CAN BE MEASURED

At least half a dozen scales now exist by which it would have been possible to measure the quality of the Handwriting on the Wall. Faust said:

"What she reveals not to thy mental sight Thou wilt not wrest from her with levers and with screws."

But science has enormously increased the subtlety of levers and screws, and our mental sight is obtuse compared to some of our present-day mental tests. Jesus tacitly accepted and practiced mental measurement when He estimated the quantity of faith on a mustard-seed scale.

It is possible to measure, at least crudely, an individual's love of a sunset or appreciation of opera. Theoretically the thesis is sound but whether practically we shall ever possess sufficient ingenuity to discover all the things that exist in amount and then measure them with any great accuracy, is a question. All that is necessary to accept for the present is that all the abilities and virtues for which education is consciously striving can be measured and be measured better than they ever have been. The measurement of initiative, judgment of relative values, leadership, appreciation of good literature and the like is entirely possible. We already have a scientific scale for the measurement of poetic appreciation. The measurements may not be as exact as we might wish, but they would have value.

THESIS 3. MEASUREMENT IN EDUCATION IS IN GEN- ERAL THE SAME AS MEASUREMENT IN THE PHYSI- CAL SCIENCES.

The two types of measurement are fundamentally alike because both measure physical manifestation. Neither adding ability, nor good intentions can be measured by plunging a thermometer into a pupiPs spiritual medium, but they can be by measuring his behavior and judging his inner condition therefrom. Unless the witness is a habitual liar, psychologists can, with considerable success, determine by means of a breathing curve, when a witness is not telling the truth.

In a still invisible future it may be possible to secure a "movie" of a pupiFs mental machinery when in operation and thus secure the desired information but for the present it is necessary to measure the product produced and, if desired, infer the inner condition of the pupil.

Measurement must frequently meet the objection of being too mater iahstic. Listen to Gilder in "The Poet's Protest."

"O man with your rule and measure, Your tests and analyses!

You may take your empty pleasure, May kill the pine, if you please, You may count the rings and the seasons, May hold the sap to the sun.

You may guess at the ways and the reasons Till your httle day is done."

To parody Wagner in "The Better Way," one would think that it was the purpose to measure human worth by the ell, the value of a life by the number of its years, the painter's canvas by the yard, or the work of the poet by the pound or bushel. A student writes: "Measurement should not be applied where spiritual factors and ideal values are involved." Those educators who protest most violently against any such measurement of the pupil are daily probing his mental activity by methods which are comparable to the surgical operations of bygone ages. They find themselves in a position of disapproving the lover who estimates his lady's affection by the radius of the pupil of her eye under standardized lighting, and of approving the scientific father who soothes the mother for his punishment of their infant by saying: "I am not slapping an innocent soul but spanking a physiological reaction."

THESIS 4. ALL MEASUREMENTS IN THE PHYSICAL SCIENCES ARE NOT PERFECT

Physical measurements are, in general, more exact than educational measurements but education has no monopoly upon imperfect tests. There are tests which are now the rule in physical sciences for which an expert in educational measurements would blush. The general superiority of physical measurements is not due to the fact that they are radically different in kind. Physical measurements are subject to practically all the errors which trouble educational measurement. It is not that they do not exist in the former, but that they usually exist in such small amounts that the average person fails to see them. They are large enough to be the despair of experts in the various sciences. Thorndike ^ has given us an excellent statement of this point:

'^Nobody need be disturbed at these unfavorable contrasts between measurements of educational products and measurements of mass, density, velocity, temperature, quantity of electricity, and the like. The zero of temperature was located only a few years ago, and the equality of the units of the temperature-scale rests upon rather intricate and subtle presuppositions. At least, I venture to assert that not one in four of, say, the judges of the Supreme Court, bishops of our churches, and governors of our states could tell clearly and adequately what these presuppositions are. Our measurements of educational products would not at present be entirely safe grounds on which to extol or condemn a system of teaching reading or arithmetic, but many of them are far superior to measurements whereby our courts of law decide that one trade-mark is an infringement on another."

But the imperfections of educational measurements are, in general, far more glaring than the majority of those made in physics, chemistry and like sciences. Some may have gotten the impression from certain emotional and quixotic radicals that standard tests are perfect instruments. This is far from the truth. They have numerous and decided limitations. The recent somewhat unbalanced, but doubtless necessary, propaganda is justified not because of the perfection of the tests recommended, but the much greater imperfection of the tests or lack of tests now in use.

A common criticism of educational measurement is that the tests measure a narrow, limited segment of a pupil's totahty. Physical measurements tend to be more handicapped in this respect than educational measurements. Most of their measurements, such as measurements of length, width, weight, and temperature are exceedingly narrow abstractions and they are exceedingly useful too. A totality test for a pupil would certainly be useful but if we possessed one we would proceed immediately to construct tests for the detailed measurement of pupil abilities. Scales for the measurement of composition are useful, but scales for the measurement of the elements which go to make up composition are also useful. Teachers not only teach children ''all over"; they teach them in detail. If tests are to aid instruction effectively, there is as much need for them to measure in detail as in totality.

THESIS 5. MEASUREMENT IS INDISPENSABLE TO THE GROWTH OF SCIENTIFIC EDUCATION

Exact measurement has made possible the rapid progress in the natural sciences. It has been stated that the amount of soap used is an index of the civilization of a country. The exactness of measurement is a good index of the status of a science. Consider where science would be without its meter, gram, ampere, volt, ohm, watt, henry and the like. More than anything else it has been the absence of exact measurement which has kept education from the rank of a science. This plea for the development of those instruments which will make possible the progress of education as a science is made with knowledge of a recent statement by a prominent educator: 'T think it would be disastrous if education were reduced to an exact science."

Richards,^ in his presidential address before the American Association for the Advancement of Science, said: ^Tlato recognized, long ago, in an often-quoted epigram, that when weights and measures are left out, little remains of any art. Modern science echoes this dictum in its insistence on quantitative data; science becomes more scientific as it becomes more exactly quantitative."

In fact, measurement and education are like the twin girls whose hair the mother of many children braided together. Neither of the twins could move unless they both moved together.

Foote gives the above quotation in a nutshell when he says: ^'The day of guesswork must give way to definite facts supported by undebatable evidence."

There are those who tremble lest the development of education as a science will squeeze out of life its emotions and delicate perceptions. As well fear that woman suffrage or the '^ female" in industry will destroy gallantry among men. The roots of these fine things go too deep into human nature. In an especially unhappy mood Amiel writes: ''Philosophy will clip an angel's wings," and again, "Science is a lucid madness engaged in tabulating its own necessary hallucinations." The basic function of Science is to help us to attain our objectives in the quickest and most economical way, whether the objectives be material or

8 "The Problem of Radioactive Lead"; Science, Jan. 3, 1919.

spiritual. Science is frequently looked upon as materialistic chiefly because only those persons who seek material objectives have had the good sense to secure the aid of Science. Haeckel, who has just drawn the line under his life's work, must have had in mind the unnecessary inefficiency of idealism when he wrote: ''No cosmic problem was ever solved or even advanced by that cerebral function we call emotion." For centuries education has been like an emotional dog chasing a frantic tail. We have had a long line of great educational thinkers from Plato through Pestalozzi, and Froebel ... to Dewey and beyond. "The old order changeth, yielding place to new," but no one seems to know whether the old or the new is better. In fact, there is grave suspicion that we move in an orbit whose form is the circle. These educational leaders are not answering ques^ tions. They are asking questions which do not occur to others. They are proposing problems for experimentation. The final answer to every educational question, except one, must be left to the educational measurer and must await the development of education as a science.

THESIS 6. MEASUREMENT IN EDUCATION IS BROADER THAN EDUCATIONAL TESTS

This book is not entitled Educational Tests because there are other methods of measuring pedagogical products. As previously stated, some estimate the quality of instruction by investigating the material equipment of libraries and laboratories and class rooms, or by the academic or professional training of the teacher. Others measure instruction by observing the teacher's method and by forming an opinion on the basis of these observations. Others base their judgments upon detailed observations of the behavior of pupils. Still others test the pupils by means of examinations. This book attempts to discuss the basic principles of measurement which apply not only to educational tests but to any sort of educational measurement. Some of the above methods of measuring educational results we are likely to have with us for some time to come. In our zeal for improving tests proper, we should not neglect the refinement of these methods. The main emphasis should be, and in this book will be, upon tests, because they offer the best promise for exact measurement. For if we can trust the experience in other fields, measurement by means of some sort of instruments will gradually replace all other forms. Finally, the book is not entitled Educational Measurement because education is deeply concerned with measurements which are not exactly products of instruction but about which educators need to be critical.

THESIS 7. THERE ARE OTHER THINGS IN EDUCATION BESIDES MEASUREMENT

It will doubtless come as a surprise to the reader to hear one who is interested in the scientific measurement of education make such an admission. There are at least three other important factors in education, namely, pupil, methods and material, and goals. The teacher needs to know the psychology of the pupil, the proper goals to which the pupil is to be developed, and the methods and material which should be employed to develop the pupil from his initial ability to the desired goal. What does measurement have to do with this process?

THESIS 8. TO THE EXTENT THAT THE PUPIL'S INITIAL ABILITIES OR CAPABILITIES ARE UN- MEASURABLE A KNOWLEDGE OF HIM IS IMPOS- SIBLE

It has just been stated that a teacher needs the most intimate possible knowledge of a pupil in order to know what methods and materials to employ in order to help him most quickly to attain a desired goal. We partly know a pupil when we know the abilities and capabihties which he possesses. To determine the mere existence of an ability involves a crude measurement. But if w^e know no more than this we cannot tell whether a pupil has these abilities

Place of Measurement in Education ii

in sufficient quantities to permit him to matriculate for a Ph.D. in the university or just enough to enter the kindergarten. We must know not only what qualities exist, but also in what amount they exist, and the more exactly we know this amount the better. Measurement is essential to a practical knowledge of psychology.

THESIS p. ''TO THE EXTENT THAT ANY GOAL OF EDUCATION IS INTANGIBLE IT IS WORTHLESS''*

We want to be able to answer at least three things about any goal: (i) What is the worth of the goal? (2) What is the location of the goal? (3) Is the pupil moving toward or from the goal? Measurement is necessary to answer each one of these absolutely vital questions. Suppose it be said that one goal of instruction is to produce in the pupil an ability to write. The worth of this goal depends upon an exact or crude measurement of how much penmanship contributes to the efficiency of a number of other superior activities. The goal has advanced little beyond perfect intangibility until it is located. How much ability to write? What speed? What quality? Even the worth of the goal cannot be answered until this location is made, since the worth varies with the quantity. The very words how much imply and in fact require measurement. Finally, it is necessary to answer the question: Is the pupil moving toward or from the goal? Without measurement the question is unanswerable.

THESIS 10. THE WORTH OF THE METHODS AND MATERIALS OF INSTRUCTION IS UNKNOWN UNTIL THEIR EFFECT IS MEASURED

The purpose of certain methods and materials is to help the pupil grow toward a certain goal. Do the methods employed accomplish their purpose? We cannot tell without employing measurement. For aught we know, the methods may be actually vicious. They may be forming

* I am indebted to F. M. McMurry for this thesis.

habits which not only do not lead toward the goal, but which may be building up difficulties for another method by a subsequent teacher. It is equally true that the comparative worth of different methods and materials is unknown until their effect upon the pupil is measurable. This means that measurement is indispensable to the experimental selection of the most economical educational conditions.

Thus, measurement is everywhere in education and in our daily lives. Measurement is no rare freak. It gets up with us in the morning and goes to bed with us at night. The mile stone, the hand of the watch, the humble cup in the kitchen, the lengthening shadows of the trees on the grass, the spacing of the year into seasons, all indicate how ubiquitous measurement is. And measurement is just as immanent in the whole educational process as in life in general. There are other things in education besides measurement but they have no value so long as they are dissociated from it.

THESIS II. MEASUREMENT OF ACHIEVEMENT SHOULD PRECEDE SUPERVISION OF TEACHING METHOD

Education is now being measured in two ways. When a child, I watched two coal miners lift a derailed car. Their efforts illustrate these two methods of measurement. A lever and fulcrum were brought, but the lever broke. A stronger lever was secured, but the fulcrum was too far from the car. Finally the proper adjustments were made and the car was lifted. Whether or not the car was lifted could be determined in two ways; (i) by measuring the length of the lever, the resistance of the fulcrum and the ground under the fulcrum, the weight of the men, the point of application of their weight, the distance of this point to the fulcrum, the distance from the fulcrum to the car, the weight of the car; or (2) simply by determining whether the car was actually lifted.

It is a fair assumption that the crucial purpose of elementary education is to make certain changes in children. To this end we have surrounded them with levers and fulcra in the shape of books, pictures, maps, tools, playthings, pedagogical methods and with teachers who will utilize these instruments as leverages to produce the desired changes.

Again it is a fair assumption that the schools should know whether their levers, fulcra, etc., are really producing the changes desired. As in the case of the derailed car, there are two methods of measuring these changes; (i) strength of lever, length of leverage, etc., become the number and nature of the books in the libraries, map facihties, blackboard space, and such, and the weight of the men becomes the number of diplomas possessed by the teacher or else the amount of her skill in making provision for motive, initiative and such on the part of her pupils. (2) Whether the car is actually lifted is comparable to measuring directly the changes in the pupils.

Doubtless our relatively primitive ancestors held conferences to discuss the advisability of such and such arrangements of lever and fulcrum in lifting a weight. Of course such possible discussions never were and never could be settled until the crucial measurement — the direct measurement was made. It would be of inestimable value to know whether the presence of certain books in the schoolroom, or the possession of a certain amount of professional training on the part of the teacher and the like are prerequisites of certain defined changes in pupils. Without such ancillary measurements by teachers and supervisors, the conditions for pupils' growth cannot be arranged in advance with certainty. But we shall not arrive at such knowledge except through direct measurement. We certainly cannot claim to know the exact casual relation between defined changes in pupils, and most of the paraphernalia with which the pupil is now surrounded. In spite of our ignorance of these causal relations, the chief method of supervision at present is to attempt to judge the presence or absence or amount of presence of these levers and fulcra.