National Academies Press: OpenBook

Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines (1984)

Chapter: APPENDIX B DESIGNING AND BUILDING THE BRIDGE

« Previous: APPENDIX A BACKGROUND PAPERS
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 149
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 150
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 151
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 152
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 153
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 154
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 155
Suggested Citation:"APPENDIX B DESIGNING AND BUILDING THE BRIDGE." National Research Council. 1984. Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. Washington, DC: The National Academies Press. doi: 10.17226/930.
×
Page 156

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

APPENDIX B DESIGNING AND BUILDING THE BRIDGE The body of this report and Appendix A cover the substantive aspects of the CASH project. This appendix describes the process: the organization and structure of the June 1983 seminar at St. Michaels, the January 1984 follow-up meeting in Baltimore, and related project activities. It can be thought of as a case study of an effort to foster interdisciplinary collaboration. Readers who have found the products of the seminar to be Of some value may want to consider some of the same approaches for simi lar undertakings. PREPARING FOR THE SEMINAR Plans for the CASH seminar were developed by the chair, Judith Tanur, with assistance from staff of the Committee on National Statistics (CNSTAT), based on a proposal developed earlier by Stephen Fienberg and Hiron Straf. Three specific objectives were set for the St. Michaels meeting: (~) to review recent work in the cognitive sciences and its potential applications to survey research; (2) to propose specific research and experimentation that might lead to improvements in the questionnaire and interviewing procedures used in the National Health Interview Survey (NHIS) and other surveys; and (3) to generate ideas for basic research, including research in the cognitive sciences using surveys as vehicles for experimentation and the collection of relevant data. In planning the seminar, the organizers made three basic decisions. First, because the number of participants would be relatively Mall--about 20--and they would meet for approximately one week, it was important to find an informal and isolated setting where the participants' full attention could be given to the work of the seminar. A suitable location was round at St. Michaels, Maryland, on the Eastern Shore of the Chesapeake Bay, and the seminar was scheduled to convene there for six days, from June 15 to June 21 (Wednesday through Mbaday), 1983. Second, while the scope of the seminar was to be broader than that of the 1980 Bureau of Social Science Resarch Workshop, experience from the 149

150 earlier meeting showed the value of structuring the discussions around specific problems encountered in one or more complex, large-scale surveys. The NHIS, which is conducted for the National Center for Health Statistics (NCHS) by the Census Bureau, was selected for primary focus at St. Michaels for several reasons. First, the subject matter of the NHIS is complex, requiring performance of difficult cognitive tasks by interviewers and respondents and Shus providing a rich source of examples for analysis. Second, the error structure of the survey results is reasonably well understood as the result of extensive and continuing methodological research that began Just prior to the inception of the NHIS in 1957. Last--and very important--the two agencies responsible for the NHIS, the NCHS and the Census Bureau, expressed a strong interest in the objectives of the seminar and a willingness to assist in familiarizing participants with the NHIS. In addition, the organizers selected two other surveys for less intensive concentration: the General Social Survey, conducted by the National Opinion Research Center, and the National Election Survey, conducted by the Institute for Social Research of the University of Michigan. third. it was derided that the oartictinants would be asked not only the pro'] ect _ _, ~ ~ _ _ ~ ~ ~ to attend the seminar, but also to give some of their time to both before and after the seminar. Prior to the seminar, all the participants would be asked to read relevant background materials and to be interviewed in the NHIS. Some participants would also be asked to provide background materials for distribution in advance of briefings at the seminar. Following the seminar, all participants would be urged to submit specific research proposals and would be asked to attend a shorter follow-up meeting if the group decided that such a meeting would be valuable. Preparation for the St. Michaels meeting included--in addition to the obvious (though crucial) aspects such as participant selection, agenda development, and logistics--four special undertakings detailed below. ( 1~ I nrQnaration Id distribution to the DarticiD~ta of a set b~r~nrn~ Laterals. The complete set of background materials, listed _ in Appendix C, was mailed to all participants early in May 1983. Of special significance were two papers prepared expressly for the CASH project: Cognitive Science and Survey Methodic by Roger Tourangeau and "Potential Contributions of Cognitive Sciences to Survey Questionnaire Desigan by Norman Bradburn and Catalina Danis (presented in Appendix A). 80th papers are about existing and potential links between the cognitive sciences arid Burner research, but the authors approach" the subject in different ways. The 16 other background materials fell into several different categories: the relationships between cognitive science and survey research (items 1, 2, and 11~; the NHIS content and procedures and related methodological research (items 8-10~; other surveys to be discussed at the seminar (items 12-14~; important concepts and research in the cognitive sciences (items 3-5~; two pertinent tools used in survey research--computer-assisted telephone interviewing (CATI) and validity checks (items 6, 7, and ~5) ; and a list of selected additional readings in cognitive science and survey methods (item 167.

151 (2) ~ NHIS. Thanks to excels ent cooperation by the Census Bureau, nearly all of the participants were interviewed by regular Census Bureau interviewers for the NHIS. The experience of being respondents in NHIS personal interviews gave the seminar participants a first-hand understanding of the nature and difficulty of the cognitive tasks required of respondents to that survey. Although these interviews were not part of the regular NHIS sample and were not included in the survey estimates, standard interview procedures were used except for a requirement that the seminar participant be one of the respondents to the core questions and that he or she be selected to respond to the supplemental questions on use of alcohol and tobacco. After being reviewed for interviewer errors at the Census Bureau, the completed questionnaires were returned to the participants for their use. (3) the! aemi~ar. Interviews with two volunteer respondents were conducted by experienced Census Bureau interviewers in the respondents' homes and were videotaped by a specialist in documentary videography. The interviewers were paid by the Committee on National Statistics to conduct the interviews; they were not acting as Census Bureau employees. To protect the rights of the interviewers and respondents, informed consent procedures were developed and were reviewed and approved by the National Academy of Sciences' Committee to Review Human Studies, and the videotapes were copyrighted. (4) ~ ~ . The adaptation of the questionnaire was done by Albert Madansky, one of the semi nar participants, using a CATI system that he had developed . The objective of demonstrating a CATI system was to familiarize participants with a new interviewing technique that in likely to be increasingly used in surveys and that may involve significant changes in the cognitive tasks to be performed by interviewers and respondents. THE ST. MICHAELS MEETING The agenda for the St. Michaels meeting was divided into four principal phases, intended to proceed more or less in sequence with some overlap. The first phase was intended to facilitate getting acquainted. (line first step had been taken before the meeting by circulating curriculum vitae of all participants. ~ At the initial group cession, each participant was asked Deco descri be his or her research interests and put on the table rough ideas for research relevant to the CASH project ob] ectives . The second phase consisted of background presentations and discussions. These sessions were intended to give all participants a common information base from which they could work together to develop

152 research proposals. The topics of these discussions were essentially the same an those covered by the background materials; cognitive sciences for survey researchers; survey methods (including CATI) for cognitive scientists; specific surveys, with mayor focus on the NHIS; and reactions to participant and videotaped interviews. The background presentations and discussion occupied most of the second and third days of the seminar. The third phase was devoted to working group meetings. In order to allow substantial time for informal, intense ve discussions in smaller groups, two sets of. three working groups were established. The first set of the three groups covered the major cognitive aspects of survey interviews ~ comprehension, retrieval , and Judgment and response ), and the second set covered the principal topics included in the core portion of the NHIS questionnaire (utiliza~cion of health services, health conditions, and restricted activity). Prior to the seminar, participants had been asked to rank their choices for the topics in each set of working groups. Assignments were then made on the basis of several criteria: participant choices (virtually all designations were either first or second choices within each set); formation of heterogeneous groups, with more or leas proportional representation of cognitive scientists, survey researchers, agency representatives, and pro ject staff.; and minimization of overlap between working groups in the two sets, so that each participant would work with the largest possible number of the other participants in the working group setting. For each working group, the chair appointed a convener and a member of the project staff to serve as rapporteur. In response to suggestions by several participants during the seminar, a third whet of working groups--which came to be known as the brainstorming groups--was organized. These brainstorming groups had two objectives: (~) to allow and encourage all participants to present and get reactions to additional proposals for research and (2) to allow participants to work in a small group setting with come of the seminar participants with whom they had not been associated in either of. the other two sets of. working groups. The designation of these groups took into account requests from some participants to be in the same group with specific individuals. The fourth and S.inal phase was feedback arid integration. General feedback sessions were scheduled for the evenings of the third and fifth days in order to permit discussion of possible agenda changes and improvements in procedures. Plenary sessions were scheduled toward the close of the seminar for working group reports. In addition to the plenary sessions, a final session was scheduled at which the participants were asked to summarize their thoughts about relevant research that they would like to undertake or participate in. With minor exceptions, al' seminar participants attended the entire seminar. Several other people were invited to be guests at the seminar: they included representatives from the National Science Foundation, the National Center for Health Statistics, the Bureau of the Census, the Committee on National Statistics, and a university (see pp. v-vi). Most of the visitors came to the seminar site on Saturday or Sunday (the fourth and fifth days or the seminar) and stayed through noon on Monday. This allowed them to attend the plenary sessions at which the reports of

153 the first two sets of working groups were presented and to view one of the videotaped NHIS interviews. In addition, each guest was invited, at a plenary session, to react to the conclusions and research ideas presented by the working groups. All plenary sessions were tape recorded, and most of them were transcribed after the seminar. Members of the project staff took detailed notes at each session. These materials, plus the written working group reports, form the basis of the proceedings of the CASH project presented in Chapter ~ of this report. At the final cession, participants agreed that a report of their observations, conclusions, and research ideas should be prepared and that the CASH participants should reconvene for about two days in late 1983 or early ~ 984 to review a draft report and discuss ideas and plans for f urther research. FOLLOW-UP: THE BALTIMORE MEETING After the St. Michaels seminar, the CASM chair and staff developed the agenda for a two-day meeting in Baltimore in January 1984 and prepared a partial draft of the project report, which was mailed to participants for review about five weeks before that meeting. More important, initial results from the St. Michaels seminar began to take shape. Several of the participants, working individually or in small groups, began new research activities or developed proposals for research along the lines that had been discussed at the seminar. It soon became clear that part of the Baltimore meeting could be usefully devoted to presentation and discussion of these activities and proposals. Brief write-ups of several of them were included in the draft report Rent out before the meeting. In order to involve other researchers, some steps were talcen to begin publicizing the interim results of the CASM project. One of the videotaped NHIS interviews was shown to and discussed with a small group of Census Bureau employees in November 1983. Plans were made for a cession on the CASM project at the annual meeting of the American Association for Public Opinion Research in May ~ 984. The CASM chair, Judith Tanur, in response to a request, organized an invited paper session on cognitive aspects of survey methodology for the annual meeting of the American Statistical Association in August 1984. All of the regular participants in the St. Michaels seminar returned for the January 1984 meeting in Baltimore. This fact may indicate their perceptions of the value of the CASM project and their commitment to it. Three guests were invited for part of the meeting: two (who had also been guests at St. Michael from the National Science Foundation and one from the Social Science Research Council. The agenda for the Baltimore meeting retained some of the features of the St. Michaels seminar, adapted to the shorter time aval table. At the start of the meeting, to stimulate further thinking on the interview process, parts of two videotaped interviews from the pretest for the 1984 round of the National Opinion Research Center 's General Social Survey were shown and discussed. Unlike the NHIS, the General Social Survey has

154 many questions on respondent attitudes and perceptions and thus involves cognitive processes different from those required to respond to the basically factual questions of the NHIS. Part of the time was spent in small group discussions. The participants were again divided into three working groups, using essentially the came criterion of heterogeneity within groups used to form the groups earlier at St. Hichaels. In this case, all groups were to discuss the same two topics: ~ ~ ~ further ideas for relevant research and (2) proposals for ways of continuing and expanding collaboration between cognitive scientists and survey researchers. On the final day of the meeting, the working groups reported back to the full group. A key feature of the Baltimore meeting, to which considerable time wan allocated, was the presentation and discussion of the several research activities and ideas that had been developed by individuals and small groups of participants following the St. Michaels seminar. Prior written reports on those activities and ideas were updated, and several of them evoked strong interest and constructive comments from other participants. The final agenda item for the Baltimore meeting was to "achieve closures on the CASH project. The participants agreed on the form of this report and discussed means of publicizing the activities and output of CASH so that other researchers might become involved in cross- disciplinary collaboration. Several other kinds of outreach were suggested in these discussions, including: --publishing relevant articles in Journals read by cognitive scientists and predating papers at meetings of associations to which they belong; --extending the dissemination process to international Journals and conferences, such as the 1985 meeting of the International Association of Survey Statisticians; --arranging for appropriate Journals to publish special issues devoted to relevant themes, e.g., cognitive studies and survey research methods; --when sufficient results are available from collaborative research studies, holding symposia in university nettings to present and discuss them; --organizing a short course on cognitive aspects of surveys in conjunction with an annual meeting of the American Statistical Association; --encouraging cognitive scientists interested in surveys as vehicles for research to attend courses in survey methods, e.g., the seamer course presented annually at the Survey Research Center at the University of Michigan; --preparing short annotated bibliographies (~) for survey researchers interested in learning about relevant aspects of the cognitive sciences and (2) for cognitive scientists who want to become familiar with survey research methods; --asking cognitive scientists to participate in proposing and planning specific investigations in the Census Bureau's planned three- year program of research on telephone survey methodology.

155 The question of how and by whom such outreach activities might be undertaken led to the broader question of the future of the CASH project in a more formal sense. The consensus of the participants in Baltimore was that CASH had met and perhaps exceeded its original goals: it had generated several promising interdisciplinary research activities and plans, and it had established an informal network of scientists who appreciate the benefits of collaboration between cognitive scientists and survey researchers. The participants agreed that the necessary momentum had been established for the development of collaborative research and that the important thing now was to proceed with the research. The network would continue to {unction and expand without requiring formal identification or support at this time. Support for specific research plans or, when more research findings are available, for symposia, would, of course, be necessary. Thus' the formal aspects of the CASH project have been completed. Evaluation of the results, however, will continue an the organizers and participants monitor the progress of relevant research, the application of cognitive research findings in surveys, and the use of surveys as vehicles for cognitive research.

Next: APPENDIX C BACKGROUND MATERIALS FOR THE SEMINAR »
Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines Get This Book
×
Buy Paperback | $50.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!