Question Evaluation Methods: Contributing To The Science Of Data Quality

Researchers from many different fields present an interdisciplinary examination of existing methods of evaluating questions for public opinion and other surveys. Workers in various US government statistical agencies initiated the project to identify the strengths and weaknesses of the methods, and sought out specialists in various fields to present papers at an October 2009 workshop in Hyattsville, Maryland. The 19 papers cover behavior coding, cognitive interviewing, item response theory, latent class analysis, split-sample experiments, multi-trait/multi-method experiments, and field-based data methods. Among their topics are how methods could be improved or replaced, and how to use them in ways that exploit their strengths and reduce the impact of their weaknesses. Annotation c2011 Book News, Inc., Portland, OR (booknews.com)

JENNIFER MADANS, PhD, is Associate Director for Science at the National Center for Health Statistics (NCHS). She has published numerous papers in the areas of health status measurement, chronic disease epidemiology, and aging.

KRISTEN MILLER, PhD, is Director of the Question Design Research Laboratory at the NCHS. She currently oversees the question evaluation research program and designs and implements research projects on data quality and comparability in the health sciences.

AARON MAITLAND, MS, is Statistician at the Question Design Research Laboratory at the NCHS, where he is the administrator of Q-Bank, the agency's online database of question evaluation reports for Federal surveys.

GORDON WILLIS, PhD, is a Cognitive Psychologist at the Applied Research Program in the Division of Cancer Control and Population Sciences at the National Cancer Institute. Dr. Willis has more than twenty years of experience in planning, conducting, and reporting of all phases of cognitive laboratory research.

Contributors.

1. Introdution (Jennifer Madans, Kristen Miller, Gordon Willis and Aaron Maitland).

Part I. Behavior Coding.

2. Coding the behavior of interviewers and respondents to evaluate survey questions (Floyd J. Fowler, Jr.).

3. Response to Fowler: Coding the Behavior of Interviewers and Respondents to Evaluate Survey Questions: Response (Nora Cate Schaeffer and Jennifer Dykema).

4. Response to Fowler: Some Thoughts in Response to Jack Fowler’s Paper (Allisú Schoua-Glusberg).

Part II. Cognitive Interviewing.

5. Cognitive Interviewing (Kristen Miller).

6. Response to Miller: Cognitive Interviewing (Gordon B. Willis).

7. Response to Miller: Reactions to Miller (Frederick G. Conrad).

Part III. Item Response Theory.

8. Applying Item Response Theory (IRT) Models for Questionnaire Evaluation (Bryce B. Reeve).

9. Response to Reeve: Item Response Theory(IRT) Models for Questionnaire Evaluation (Ron D. Hays).

10. Response to Reeve: What the IRT Model Can Do for Survey Methodologists and Data Quality (Clyde Ticker, Brian Meekins, Jennifer Edgar and Paul Biemer).

Part IV. Latent Class Analysis.

11. Some Issues in the Application of Latent Class Models for Questionnaire Design (Paul P. Biemer and Marcus Berzofsky).

12. Response to Biemer and Berzofsky (Frauke Kreuter).

13. Response to Biemer and Berzofsky: What Meets the Eye Latent and Manifest Variables in Design and Protesting (Janet A. Harkness and Timothy P. Johnson).

Part V. Split-Sample Experiments.

14. Experiments for Evaluating Survey Questions (Jon A. Krosnick).

15. Response to Krosnick: Experimental Design for Cognitive Interview Pretests (Johnny Blair).

16. Response to Krosnick: Session on Experiments Workshop on Questionnaire Evaluation Methods (Theresa J. DeMaio and Stephanie Willson).

Part VI. Multitrait-Multimethod Experiments.

17. Evaluating the Reliability and Validity of Survey Interview Data Commentary on the MTMM Approach (Duane F. Alwin).

18. Response to Alwin: On Q (Peter Ph. Mohler).

Part VII. Field-Based Data Methods.

19. Using Field Tests to Evaluate Federal Statistical Survey Questionnaires (Brian A. Harris-Kojetin and James M. Dahlhamer).

Index.