Surveys

Bill Anderson, College of Social and Behavioral Science

Surveys are research tools that can generate opinions from respondents about the quality of their experiences as students or from other informants about their perceptions of student outcomes. Surveys are indirect assessment tools in that they do not actually measure student learning but instead provide self-report data about the respondents' perceptions and opinions. [See "Testing" for a discussion of paper and pencil tests as an alternative to or component of surveys.] Typical survey respondent groups are: incoming students, current students/majors, graduating majors, alumni, and employers. Well designed samples may be surveyed to reduce the costs of distribution and analysis when large populations are being studied. Typical survey topics include satisfaction (with instruction, course content, program design, facilities, advising, job readiness, etc.) and estimates of outcomes (learning, accomplishment of program objectives, readiness for graduate school, etc.). Surveys may be oral (e.g., exit interviews) or written (e.g., mailed questionnaires), may be done in person (one-to-one or in groups), by telephone, by mail, or electronically (e.g., web-based or via email), and may involve a variety of types of questions (multiple choice, ranking, Likert, short-answer, open-ended, etc.).

ADVANTAGES OF SURVEY USE

Input from consumers: Students provide their perceptions of the program attributes rather then faculty/staff presuming what those perceptions are. This can be an empowering process which is consistent with the increasing emphasis on the consumer's right to have opinions.
Standardization: Surveys can be used to provide comparative data across time, across programs, and across institutions. In many fields there are movements toward developing common tools such as alumni surveys.
Common usage: Students are very familiar with surveys and are generally a cooperative target population.
Data that is easy to analyze: For most assessment projects it will be sufficient to examine percentages and means, a simple level of data analysis that can be done by staff without a large investment of faculty time.

DISADVANTAGES OF SURVEY USE

Indirect Measure: Surveys do not provide evidence of student learning, they provide opinions about whether the learning occurred or about some other aspect of the program. While opinion data may be a useful component of assessment, it should not be the only data gathered.
Survey Design: Poorly designed surveys can still generate large quantities of useless or inaccurate data; survey design is a much more difficult skill than most survey writers think. It may be possible to find pre-tested instruments that others have already de-bugged and which can generate comparative data.
Costs: While surveys are generally not a high cost strategy, there are costs associated with the time spent in design, pre-testing, preparation, distribution and analysis even beyond the obvious costs of photocopying, postage, etc. Maintaining an accurate alumni database is a useful source of mailing labels for follow-up studies but requires a commitment to ongoing updates of constantly changing data.
Evaluation: Surveys can provide statistical data about a program but it is still the responsibility of the faculty to determine the meaning of the results. If 60% of the respondents "believe they are ready for graduate school", is that a good percentage or an inadequate one? What thresholds should the program use as evidence of effectiveness (these may be identified in the program's objectives)?

SUGGESTIONS

Departments should carefully consider their goals and objectives to see if a survey can provide the information that they need to assess their effectiveness, rather than just choosing to do lots of surveys.
If surveys are to be used, the assessment literature and the academic discipline literature (e.g., a journal on "Teaching in ____") can be searched to see if a pre-tested device is available.
If a new survey is to be designed some departments may want to seek consultation from colleagues more experienced in survey research to help with typical design questions. The new device should be pre-tested on a small group of typical respondents to identify any needed revisions in language, content, or structure.
A schedule of repeated administrations should be developed at intervals that meet the department’s need for assessment data and which enable comparisons to be made over time. Characteristics of the respondents should be considered in setting this schedule, e.g., when they are available (not over holidays), when in-class administration will be least likely to interfere with academic content, etc.
An individual or sub-committee will need to take responsibility for the process of developing, administering, and evaluating the surveys; this may involve enough extra work that support services (student assistants, release time/overload pay) will be needed and departments should examine the extent to which this work can be considered as part of the faculty member's professional development plan.