|
|
|
|
|
Assessment and Diversity
in a Digital Library
John Haberin New York City
. . . Through columbia earthscape
[As noted in this article's formal About the Author (not to mention my whole dirty history elsewhere), I work in online publishing. There I try my best to contribute to research, multimedia education, and public debate on Earth's future. Hey, this is a big planet, and someone has to do it. Now, I know you must have turned to my Web site, of primarily art reviews, with other, equally pressing interests. Perhaps an artist you admire caught your eye, perhaps one of the three big words in the title. However, I want to take a moment to imagine what sustains creativity in any of these fields.
I often argue for parallels between art and science—and not the tired metaphors of popular science either. Art has become big business, of course, and caught the eye of politicians, just as education became a presidential mantra. People demand "standards," just when artists, scientists, and teachers crave funding and space to find their own way. This article explores some of the implications. It appeared in The Journal of Geoscience Education (2001)—obviously without this note—and may not be reprinted without permission.]
ABSTRACT: As the focus changes from sharing classroom experiments to testing them, one form of misplaced optimism can easily replace another. Reliable assessment of Web-based resources faces obstacles, including deciding on standards, picking out what we are testing, seeing past instructor strengths, separating interface from content, and keeping up with instructor (and student) expectations. At columbia earthscape, a large digital library in Earth-systems science, we encourage experiment. We believe that technology use will sort itself out most quickly if we focus on existing classroom aims.
We promote interdisciplinary science, connect research and hands-on education, encourage critical thinking about the nature of science, and motivate students through the impact of science on their lives. We can then initiate a six-step process of assessment, based on the achievement of those aims. The process includes editorial selection and development, often of already reviewed materials, advisory-board review, class testing, focus groups, hit counts, and a school user survey. In the end, we know that those who hope to test the ground rules are also shaping them for others.
Introduction
New media make the case for assessment more pressing than ever. One can introduce them into the classroom way too soon. They can come with inadequate preparation, unrealistic guidelines, and unearned pats on the back, whether for the technology or for the instructor.
So, on all counts, can assessment. The story may shed light on what to expect from much-needed assessment efforts now.
A nice surprise
Not so very long ago, of course, publishers were not exactly scrambling to create Web sites for college-level science textbooks. None had yet slipped a single CD-ROM into the back covers. William J. Kaufmann, then sole author for Universe, the best-selling astronomy textbook (now Kaufmann/Freedman, 1999), went to his publisher and insisted.
One could argue that software goes naturally with precisely his field. Astronomers turn to planetarium simulations. In urban and suburban schools, they face obstacles to ordinary field work, starting with light pollution. However, Kaufmann was really reacting to something else that we now take for granted (e.g., Thomas, 2000) as "the wave of the present" (Picard, 1999) or even deplore (e.g., Reisberg, 2000)—changing study habits. He believed, too, that a CD could change student conceptions of science. His ideals still apply to us in geology education today.
Publishers are conservative. They do not throw money at new media without some proof that classes can use the help—indeed, that professors expect and demand the addition. As one of Kaufmann's editors, I sent the plans out for review. He also sought word-of-mouth through the publisher's sales representatives, who obtain feedback the hard way.
It would be an overstatement to say that nobody out there cared. Still, few professors had looked at new media, few planned to integrate such a CD into course requirements, and few could agree on what it must hold. No wonder budget was tight.
Sales on an already popular textbook rose by half in that edition, which proved to be the late author's last. The CD became a necessity for future editions and for other publishers as well—and we have no firm assurance that it helped classes. The puzzle reinforces both the difficulty and the necessity of new-media assessment now.
Assessment made difficult
Assessment of new media is not just necessary if science education is to become a priority (Hall-Wallace, 2000), but even inevitable, given the calls for science-education standards (Shea, 1997). Yet as today's focus changes from sharing classroom experiments to testing them, one form of misplaced optimism can easily replace another. Even the most perspicacious presentations on assessment—in particular, at the most recent sessions of the Geological Society of America (Suiter and Hall-Wallace, 2000) and the American Geophysical Union (Buhr and Weaver, 2000)—helped bring into focus the obstacles to clarity. If we do not single out individual authors by name, we only wish to make clear that we are far from thus accusing them of failure. Rather, we have learned from their insights. Consider just a few.
1. Changing standards. All too often, an innovator has had to write new course goals in order to see whether new media can help meet them. Old standards might not even exist. Or they might prescribe, say, labs and field traps, with no room for virtual data. Either way, the room for test bias is obvious. So is the added likelihood of "teaching to the test," a practice that has tainted cries for standards in many school systems.
2. Changing test focus. Similarly, one must decide between testing an entire unit or the actual new media. The first alternative could fail to isolate the variable, apart from the quality of the textbook, class, and labs. The second approach could well come with unrealistic expectations. Students may well not learn from a simulation alone, apart from thorough background, but no one says they should. Good teaching never meant demonstrations apart from the rest of the lecture or images in a book apart from the words.
3. Changing instructors. Often, too, one person handles the control section, another the test section. In practical terms, someone with a proven course keeps going. Meanwhile, another gets to try out courseware into which the instructor poured heart and soul. Both do best with the tools they know and love the best. Both bring a level of comfort and enthusiasm few can match. No wonder students benefit from both.
4. Changing libraries. As editors for columbia earthscape (Haber et al., 1999), we are having to test the first of the extended digital libraries in Earth-systems science education. If we are doing our job of connecting research and education with a large digital archive, that only adds to the confusion. When class testers respond, they may be responding to the quality of individual experiments in new media or to the interface that they must navigate to locate suitable materials.
5. Changing expectations. When one tries assessment, one is asking for more than just the results this term. One is also predicting the value of these classroom practices for the future. Yet the ante really does go up. Four-color textbooks may not do a better job than texts years ago, texts from which we ourselves learned. Yet those texts might not work as well today. Students have changed, and the community has learned to use colors to convey real differences. In light of the multimedia classroom and digital libraries in education, students, teachers, and their suppliers are changing once again.
Above all, all of us are creating change. As Kaufmann showed, the tester will have an impact on classes everywhere. With each new-media experiment and every attempt to assess it, one creates new expectations of future success.
A resource for change
In drawing tentative lessons, we want to show how we are implementing them here at columbia earthscape, a digital library for Earth-systems science. If creating change is an obligation and a necessity, we hope to provide a useful resource.
First, however, a word on project structure, for those unfamiliar with its aims and scope. columbia earthscape presents four sections, much as a professor or teacher wears multiple hats.
One section gathers active research, such as legislation, key reports of nongovernmental institutions, conference proceedings, books, journal text, and abstracts. A second section focuses on education, including classroom models, sample syllabi, such sources of "Quick Answers" as FAQs, and papers placing "Teaching Issues in Focus." A third section, an online-only magazine called Earth Affairs, concerns public debate and public outreach.
We reprint materials, so as to create a permanent, stable, searchable archive. However, a fourth section contains additional, annotated hyperlinks. There we also include Web addresses of geoscience departments worldwide. Finally, we make available a rundown of the environmental headlines, in conjunction with ABC News, the American Museum of Natural History, and others. Here we include video clips, a conference calendar, and links to further "Analysis & Perspective" from the popular press.
The project launched at the end of 1999, following more than a year of development. It began with support from Columbia University, the Scholarly Publishing and Academic Resource Coalition (SPARC), and the National Science Foundation. However, it also has a self-sustaining, cost-recovery model, chiefly through subscriptions to libraries and to other school and campus networks. We also offer free trial subscriptions, with heartfelt thanks for any and all feedback, partners, and suggested contributions.
So how can the community assess change while the ground shifts beneath our feet? We suggest a threefold answer. The community of educators must (1) encourage experiment, (2) see new media as a tool for existing aims, and (3) keep on testing.
Encourage experiment
Science textbooks go through extensive peer review because curricula tend to converge, even as they always evolve. The multimedia classroom, in contrast, may not converge for some time to come. Moreover, when it does, it will be author driven as much as audience driven, exactly like that first CD. Peer review that invokes a uniform standard for new media may be premature.
First, classes will continue to ask for a wide range of options. At columbia earthscape, we therefore share innovative curricula widely. That includes lectures, video clips, simulations, Web-based exercises, and labs.
Second, classes will continue to ask how existing options apply to them. We therefore create a narrative context in familiar classroom topics. We also excerpt and further develop the materials as needed. Our "mini-courses" can serve as standalone entities or source of enrichment, analogous to textbook sidebars. They encourage not just borrowing but experiment, from classes that otherwise might not know where to begin.
Last, more varied uses for models allow more varied and useful feedback. We therefore always include links to the source.
See new media as a tool
Paradoxically enough, one can put too much stress on whether and how to use new media. With a different set of questions, the integration of new media into the classroom may take care of itself.
We mean questions that have generated classroom reform for years. One can then look at new media more modestly, not as an alternative to existing classroom aims, but as ordinary tools for organizing and implementing those aims. With luck, better experiments and more structured assessment can then fall into place.
For one, entirely independent of technology, classes are striving to connect research and education. By showing examples of science and its implications for society, they aim to motivate students, to foreground the nature of science, and to permit more active learning. At columbia earthscape, we therefore incorporate entire sections on research, education, and outreach. At the same time, we use hyperlinks between sections, so as to help bring data and other research tools into the classroom.
Look at it another way. A problem with assessment has been knowing just what we are testing—the course, the instructor, or the media. To create a sufficiently precise but flexible set of standards, try relying instead on familiar, well-defined aims. Once we do, it should hardly be surprising if evidence quickly supports the value of computer-assisted instruction to developing critical-thinking skills (Renshaw et al., 1998).
Keep on testing
Just as we should not expect too narrow a standard for peer review, we should never abandon the process of assessment. At columbia earthscape, we have no "open" section alongside reviewed material. A single process of assessment applies to the whole.
Actually, much like the entire community, we are just now shifting more fully into our assessment phase. However, selectivity began over a year ago, before launch.
1. Editorial review. Naturally the editorial process contributes several layers of selectivity. We actively seek out, examine, and give context to material, offering our editorial advice and development as needed, as well as an ear to the ground of valid educational trends.
2. Scholarly review. Our editorial advisory board serves as a check. The full board meets twice each year, and board committees have the final okay on new features and new "mini-courses."
3. Class testing. During the first summer of operations, students class tested the site design. Their ideas have led to improvements in the interface.
4. Focus groups. We have begun a series of focus groups, the first in November 2000, the second in January 2001. If feedback at conventions draws heavily on the education-reform community, the focus groups assemble first-year college instructors from mainstream courses, often with little or now previous knowledge of Web-based teaching.
5. Usage statistics. We are just now creating the routines to collect statistics of site usage. We hope to see more accurately what content users select first and where our interface creates obstacles to locating solutions quickly.
6. Classroom surveys. Last, a consortium of high-schools has been assessing the value of site content. Its final report, in the form of a user survey, is due in January 2001.
For examples of the first step, we have learned about potential classroom models from the Journal of Geoscience Education, from the pioneering series of columns by Warren Huff (1998–2000), and from other peer-reviewed sources, as well as from conference reports. Other ideas have come to us from a less-formal popularity contest—the numerous course Web sites with suggested links, none finer than those compiled by John Butler (2001). Our classroom pages also integrate research from elsewhere in the columbia earthscape archives, such as books and journals already subject to peer review.
We expect to present further details, based on all these parts of the assessment process, in a future paper. However, perhaps it is not premature in the present context to note that one result already: when it comes to new media, instructors are actively seeking direction.
The end of digital libraries?
In a sense, our digital archive departs from the library metaphor. To encourage experiment, we hope to mold an interface that allows browsing, a bit unlike traditional card catalogs. To connect research and education, we hope again to refine the card-catalog model. Finally, unlike a typical print library, we do not allow unlimited growth, apart from content assessment. Yet we insist on an open, flexible, multi-part process of assessment.
Assessment is hard, and the future is uncertain. Perhaps the most stubbornly productive contributors of all to the new-media classroom, Butler and Huff (2000), have asked point-blank whether all efforts to date have failed. He calls the outcome "no significant difference."
If this paper is right, that should be no surprise. Ultimately we are all trying the same thing. We simply reach for the tools we find at hand—tools that make us and our students most comfortable. Yet if I am right, the same result is a to our success. In time, the question becomes not whether to join the experiment, but how the experiment can do the old job better. That same question should guide assessment.
And remember one essential lesson of the past. We are not just assessing the future. We are also shaping it. But then teachers should be used to that!
References cited
Buhr, S. M., and Weaver, A., 2000, Measuring success: evaluating geoscience education programs, 2000 Fall Meeting: San Francisco, California, American Geophysical Union, sessions ED11B, ED12A. back
Butler, J., and Huff, W., 2000, The use of multimedia in geoscience instruction: yet another example of the no significant difference phenomena?," 2000 Annual Meeting: Reno, Nevada, Geological Society of America, session 61. back
Butler, J., 2001, The virtual geoscience professor: Houston, University of Houston. back
Haber, J., Luby, M., and Wittenberg, K., 1999, columbia earthscape: New York, Columbia University Press. back
Hall-Wallace, M., 2000, Why science education became one university's priority: Geotimes, v. 45. back
Huff, Warren, 1998–2000, Geotimes, v. 43–v. 45, numerous issues. back
Kaufmann, W. J., and Freedman, R., 1999, Universe (Fifth edition): New York, W. H. Freeman & Company, Publishers, 640 pp. back
Picard, M. D., 1999, Computers and email, the wave of the present: Journal of Geoscience Education, v. 47, pp. 62–63. back
Reisberg, L., 2000, 10% of students may spend too much time online: Chronicle of Higher Education, June 16, p. A43. back
Renshaw, C. E., Taylor, H. A., and Reynolds, C. H., 1998, Impact of computer-assisted instruction in hydrogeology on critical-thinking skills: Journal of Geoscience Education, v. 46, pp. 274–279. back
Shea, J. H., 1997, Progress on science-education standards (editorial): Journal of Geoscience Education, v. 45, p. 102. back
Suiter, M. J., and Hall-Wallace, M., 2000, Research on teaching and learning in geoscience, 2000 Annual Meeting: Reno, Nevada, Geological Society of America, session 120. back
Thomas, K., 2000, Laptops no longer a luxury for students: USA Today, October 31. back
About the Author: Up through its launch, John Haber acted as sole editor for columbia earthscape, 510 Butler Library, Columbia University, New York, NY 10027. For 20 years he had developed college textbooks in the sciences. Michael Luby brought to the project his experience acquiring scholarly, peer-reviews journals. top