Speech Communication Department

Assessment Report

2001-2002

Assessment Schedule:

Year	BA/BS	BST	MA/MS	GEN ED
2000-2001	LO 1	LO 1	LO 1
2001-2002	LO 2	LO 2	LO 2	LO 1
2002-2003	LO 3	LO 3	LO 3
2003-2004	LO 4	LO 4	LO 4
2004-2005	LO 5	LO 1	LO 1

Additional Assessments:

Starting in 2001-2002 and every four years following, the Department will conduct a survey of alumni from the previous five years.

Starting in 2003-2004 and every four years following, the Department will conduct a survey of employers of speech communication graduates from the previous two years.

BA/BS Report:

LO 2: Utilize knowledge of communication theories and principles to construct appropriate messages in a variety of settings.

Method of Assessment:

Student speeches were videotaped and assessed using the rubric below. A committee of three faculty members in the Speech Communication Department viewed and assessed the speeches. Speeches were assessed Spring 2002.

Assessment Population:

Student speeches in SPEE 333 Advanced Public Speaking were used. All students presenting speeches for a selected speech assignment (N=10) were assessed.

Assessment Rubric:

Goal: Utilize knowledge of communication theories and principles to construct appropriate messages in a variety of settings. Setting utilized for study: Individual Public Speech	Outcome clearly met (2)	Outcome minimally met (1)	Outcome not met (0)
Student selects appropriate topic for presentation
Thesis clearly stated
Preview clearly stated
Transitions effectively used
Structure of presentation matches preview
Student presentation demonstrates effective verbal skills
Student presentation demonstrates effective nonverbal skills
Student uses and cites appropriate supporting material from a variety of sources
Student presentation demonstrates awareness of audience
Student presentation demonstrates use of logical and coherent arguments
Student presentation demonstrates appropriate use of language

No level of mastery has been set using this specific rubric. However, this rubric was adapted from the rubric for LO 2 on the Department BS/BA assessment plan. Level of mastery has been set for that rubric. In the discussion section there is a sub-section which discusses the placement of these results on the rubric and level of mastery.

Results:

Only one student scored at 1 or higher on all 11 indicators. Three students scored at 1 or higher on 10 of the 11 indicators. One student scored at 1 or higher on 9 of the 11 indicators. One student scored at 1 or higher on 8 of the 11 indicators; one scored at 1 or higher on 7 of the 11 indicators; one scored at 1 or higher on 6 of the 11 indicators; and the remaining two scored at 1 or higher on 5 of the 11 indicators. Two of the 11 speakers averaged below 1 on all 11 indicators. Average speaker score was 1.242. One area (Student uses and cites appropriate supporting material from a variety of sources) averaged below 1 (.797), and a second and third area (/effective nonverbal skills/transitions effectively used) averaged just slightly above 1 (1.064/1.095). Highest areas were topic selection (1.629) and appropriate use of language (1.531).

Discussion:

The results were a bit disappointing, but several factors may have influenced the results. The sample speech topic selected was one that appeared to not require source citations. The sample population was small (N=10) and the sample speech was perhaps not representative of the course assignments as a whole. Nevertheless, some observations seem to follow:

Reminder to all students of the need to use credible sources in all speeches, as credibility is a key factor in speech success, regardless of the type of speech.
Coherent and identifiable structures are required in all speeches, including the use of transitions. Increased emphasis on speech construction may be indicated.
Students need to be reminded of the importance of nonverbals during a presentation.
More individual speaking opportunities (including critique of student performance) should be encouraged in other speech courses.

Level of Mastery:

Learning Outcome 2: Utilize knowledge of communication theories and principles to construct appropriate messages in a variety of settings.

1	Student assignments/activities/critique sheets or presentations demonstrate that students are able to organize and present individual public speeches, to take part in group communication activities, and to engage in interpersonal communication.
2	Student assignments/activities/critique sheets or presentations demonstrate that students are able to organize and present competent individual public speeches, to take part in group communication activities, and to demonstrate interpersonal communication skills.
3	Student assignments/activities/critique sheets or presentations demonstrate that students are able to organize and present competent individual public speeches, taking into consideration the needs and expectations of the audience and the occasion, to take part in group communication activities and demonstrate effective group communication skills, and to demonstrate effective interpersonal communication skills.

Level of Mastery: 100% of assessed population meets or exceeds level 1; 75% of assessed population meets or exceeds level 2.

1 student is at Level 3 (public speaking only assessed)

7 students are at level 2 (public speaking only assessed)

2 students are at level 1 (public speaking only assessed)

Level of mastery was met.

________________________________________________________________________

BST Report:

LO 2: Students demonstrate pedagogical and professional skills sufficient to teach public communication

Method of Assessment: Student digital portfolios were evaluated. Focus was on the studentπs Unit Plan. Units were assessed Fall 2001.

Assessment Population: Graduate and undergraduate students enrolled in SPEE 404 (Teaching Speech Communication). All students are in the English/Speech BST program. (N=9)

Assessment Rubric:

Learning Outcome 2: Students demonstrate pedagogical and professional skills sufficient to teach public communication.

Embedded Outcomes: NCATE 2000 Candidate Performance Standard 1; MNBOT Communication Arts and Literature Standards A: 3-6; 8-13; 15; ISTE Categories 1, 2, 3, and 5; NCA K-12: Standards 4.11 and 4.12

1	Student activities and/or assignments demonstrate minimal pedagogical and professional skills as evidenced by poorly planned and supported teaching activities, insufficient knowledge of content involved in the teaching activities, lack of or misapplication of assessment and evaluation techniques, and lack of or misapplication of teaching methods.
2	Student activities and/or assignments demonstrate acceptable pedagogical and professional skills as evidenced by adequately planned and supported teaching activities, sufficient knowledge of content involved in the teaching activities, correct application of assessment and evaluation techniques, and use of acceptable teaching methods.
3	Student activities and/or assignments demonstrate superior pedagogical and professional skills as evidenced by well planned and supported teaching activities, superior knowledge of content involved in the teaching activities, correct and continuous application of assessment and evaluation techniques, and use of teaching methods suitable to a variety of learning styles.

Level of Mastery: 90% of assessed population meets or exceeds level 2; 25% of assessed population meets level 3.

Results:

Five of the nine students met level 3; three of nine students met level 2; one student failed to meet level 2. Level of mastery was, statistically, not met (89 % of students met or exceeded level 2, not 90 %).

Discussion:

The great majority of students demonstrated acceptable or greater pedagogical and professional skills, with over 50 % demonstrating superior pedagogical and professional skills. One student failed to demonstrate acceptable pedagogical and professional skills. No major changes are planned for the course. Students will continue to be given numerous opportunities to acquire and demonstrate acceptable skills.

MA/MS Report:

LO 2: Demonstrate professional competence in oral communication

Method of assessment: Student oral presentations in Departmental Colloquium were assessed by graduate faculty using a checklist created from the rubric below. Presentations were assessed Spring 2002.

Assessment Population: All graduate students who will graduate this spring or next fall (N=4) took part in the assessment.

Assessment Rubric:

Learning Outcome 2: Upon completion of the graduate program in Speech Communication, all graduate students will be able to demonstrate professional competence in oral communication.

Student oral defense of thesis or alternate plan paper and/or oral presentation in public setting demonstrates that the student is able to adequately synthesize information in a written form of communication for presentation in an oral format. Student is able to highlight the thesis, the key supporting evidence, the key theoretical elements, and the conclusion and present these in a manner that does not hinder the audience from following and understanding the content and the organizational structure. The student uses appropriate verbal and nonverbal skills in the presentation, and attempts to defend statements made in the oral presentation.

Student oral defense of thesis or alternate plan paper and/or oral presentation in public setting demonstrates that the student is able to successfully synthesize information in a written form of communication for presentation in an oral format. Student is able to highlight the thesis, the key supporting evidence, the key theoretical elements, and the conclusion and present these in a manner that helps the audience follow and understand the content and the organizational structure. The student uses appropriate verbal and nonverbal skills in the presentation, and is able to successfully defend statements made in the oral presentation.

Level of Mastery: 100% of the assessed population meets or exceeds level 1; 90% of the assessed population meets level 2.

Results:

Three of the students reached level 2; one student was unable to achieve level 2, but did achieve level 1. Level of mastery was not met.

Discussion:

The small sample population prohibits the department from drawing generalized conclusions from the results. Individual student evaluations, from which the above results were derived, indicated that our graduate students (as they should) demonstrate minimally acceptable oral communication skills, and the majority demonstrate acceptable oral communication skills. The one student who failed to achieve level 2 missed that level due to poor time management, a structural flaw in selecting and narrowing a topic for a specific audience and occasion.

________________________________________________________________________

General Education Assessment

Assessment Review and Results: Category 1b General Education

Goal: To develop skills necessary for reasoned communication. Courses in this category will require individual public speaking which is critiqued by the instructor. Speaking and reasoning competency is an ongoing process which needs to be reinforced throughout the curriculum. Students will be able to:

(a) understand/demonstrate communication processes through invention, organization, drafting, revision, editing and presentation;

(b) participate effectively in groups with emphasis on listening, critical and reflective thinking, and responding;

(d) select appropriate communication choices for specific audiences;

(e) construct logical and coherent arguments;

(f) use authority, point of view, and individual voice and style in communications;

(g) employ syntax, usage and analytical techniques appropriate to academic disciplines and the professional world.

Course(s) which satisfy this category are as follows:

CDIS 201 SPEE 100 SPEE 102

Sample speeches on video from randomly selected SPEE 100, 102, and CDIS 201 sections

a. SPEE 100: Three sections (total population=25 sections)

b. SPEE 102: Two sections (total population=11 sections)

c. CDIS 201: One section (total population=1 section)

Sample Population: 19 speeches (one sample speech was unsuitable due to technical problems). Samples from CDIS 201 were not useable as they were inappropriate given the rubric. The CDIS samples were not individual public speeches critiqued by the course instructor.

Speeches were reviewed on Thursday, January 10. Reviewers were Nan Johnson-Curiskis, Carol Myhre, Lisa Perry, and Warren Sandmann, representing the two departments offering courses in Category 1b.

Results:

N=19

Mean: 1.81578947

Discussion:

The scores were fairly low. These scores were an accurate representation of the quality of the speeches reviewed according to the criteria of the assessment rubric. Several possibilities may account for the low scores:

Students in Category 1b courses are not learning and developing the skills needed for successful oral presentations.
Small sample population: (N was less than one (1) percent of the actual population of student presentations. As noted by reviewers, one sample section of speeches received uniformly low scores (all ones), and thus skewed the total). The sample from which the speeches were drawn included speeches that did not appear to require some of the elements being assessed.
Poor assessment rubric: Reviewers offered numerous suggestions for the improvement of the assessment rubric. A suggested revised rubric is attached.
High expectations from reviewers.

Regardless of the cause, review of the sample speeches did provide some useful information for departments teaching courses in Category 1b:

Instruction needs to emphasize audience analysis: selection of topics appropriate to the audience; selection of supporting material appropriate to the audience; selection of appropriate language for audience.
Instruction needs to emphasize coherent structure throughout the presentation: Many speeches had previews, but the thesis was missing or difficult to discover; many speeches had previews, but the rest of the speech structure did not follow the preview structure; regarding the persuasive/problem-solution speeches, many of the solutions offered did not address the problems described in the speech.
Instruction needs to emphasize the need to sufficiently support statements with credible sources and complete source citations.
Additional presentation opportunities should be considered in all Category 1b courses.
Courses in Category 1b MUST have individual student speeches formally presented and critiqued by the instructor. Courses that do not meet this criterion should not be in Category 1b, as noted in the MSUM general education requirements for Category 1b.

Rubric used for 2001-2002 assessment

4	The speech demonstrated appropriate topic selection and development, the structure of the speech followed an orderly and readily identifiable pattern, sufficient and credible sources were used and orally cited in the speech, and acceptable verbal and nonverbal skills were demonstrated.
3	The speech demonstrated appropriate topic selection, but the topic was not developed as well as it should have been, or was too broad for the occasion and time constraints; the structure of the speech was apparent, but use of transitions and signposting was lacking; some sources were used and cited in the speech, but they were too few or some lacked credibility; verbal and nonverbal skills were generally acceptable, but there were some lapses such as use of fillers and/or lack of eye contact and gestures.
2	The speech failed to demonstrate appropriate topic selection, and/or the topic was not developed as well as it should have been, or was too broad for the occasion and time constraints; the structure of the speech was not readily apparent, and use of transitions and signposting was lacking; few if any sources were used and cited in the speech, and/or those used lacked credibility; verbal and nonverbal skills were generally unacceptable, with many lapses such as the overuse of fillers, lack of eye contact, gestures and movement, and the rate and/or volume of the delivery made it hard to follow the speech.
1	The topic was essentially unidentifiable or almost completely inappropriate for the occasion, and/or the speech was significantly over or under time constraints; no structure to the speech could be discovered; no sources were cited, or those cited lacked any credibility; verbal and nonverbal skills were poor, with routine overuse of fillers, lack of eye contact, gestures and movement, and the rate and/or volume of the delivery significantly impaired understanding the speech.

Suggested Revised Rubric for Future Assessment: Category 1b General Education

(Rubric used needs to match student work assessed. For example, if student work is all individual public speaking, rubric needs to focus on those skills. If the assignments selected focus on group or listening skills, the assessment rubric needs to match those skills. Thus, only some of the criteria listed below would be appropriate for assessment purposes.)

Goal: To develop skills necessary for reasoned communication. Courses in this category will require individual public speaking which is critiqued by the instructor. Speaking and reasoning competency is an ongoing process which needs to be reinforced throughout the curriculum.	Outcome clearly met (2)	Outcome minimally met (1)	Outcome not met (0)
Student selects appropriate topic for presentation
Thesis clearly stated
Preview clearly stated
Transitions effectively used
Structure of presentation matches preview
Student presentation demonstrates effective verbal skills
Student presentation demonstrates effective nonverbal skills
Student work in groups demonstrates effective listening skills
Student work in groups demonstrates effective critical thinking skills
Student uses and cites appropriate supporting material from a variety of sources
Student presentation demonstrates awareness of audience
Student presentation demonstrates use of logical and coherent arguments
Student presentation demonstrates appropriate use of language

Assessment Rubric for O/O #1: Effective Public Communication (Original)

4	Students demonstrate the ability to effectively communicate in public by presenting a minimum of two speeches, engaging in interpersonal communication, and taking part in a group presentation. The speeches demonstrate appropriate topic selection and development, the structure of the speeches follows an orderly pattern, sufficient sources are used and orally cited in the speech, and acceptable verbal and nonverbal skills are demonstrated. Group presentations demonstrate appropriate topic selection and development, the structure of the presentations follows an orderly pattern, sufficient sources are used and orally cited in the presentation, and acceptable verbal and nonverbal skills are demonstrated.
3	Students demonstrate the ability to effectively communicate in public by presenting a minimum of two speeches, engaging in interpersonal communication, and taking part in a group presentation. The speeches demonstrate appropriate topic selection, but the topics are not developed as well as they shouldbe, or are too broad for the occasion and time constraints; the structure of the speech is apparent, but use of transitions and signposting is lacking; some sources are used and cited in the speech, but they are too few or some lack credibility; verbal and nonverbal skills are generally acceptable, but there are some lapses such as use of fillers and/or lack of eye contact and gestures. Group presentations demonstrate generally appropriate topic selection, but development of the topic is limited in that it is too broad or too narrow; the structure of the presentations generally follows an orderly pattern, but there is limited use of transitions and signposting; some sources are used and orally cited in the presentation, but they are too few or lack credibility; verbal and nonverbal skills are generally acceptable, but there are some lapses such as use of fillers and/or lack of eye contact and gestures.
2	Students demonstrate the ability to communicate in public by presenting a minimum of two speeches, engaging in interpersonal communication, and by taking part in a group presentation. The speeches fail to demonstrate appropriate topic selection, and the topics are not developed as well as they should be, or are too broad for the occasion and time constraints; the structure of the speech is not readily apparent, and use of transitions and signposting is lacking; few if any sources are used and cited in the speech, and those used lack credibility; verbal and nonverbal skills are generally unacceptable, and there are many lapses such as the overuse of fillers, lack of eye contact, gestures and movement, and the rate and/or volume of the delivery makes it hard to follow the speech. Group presentations fail to demonstrate appropriate topic selection, and development of the topic is limited; the structure of the presentations fails to follow an orderly pattern, and there is limited use of transitions and signposting; few if any sources are used and orally cited in the presentation, and those used lack credibility; verbal and nonverbal skills are generally unacceptable, and there are many lapses such as the overuse of fillers, lack of eye contact, gestures and movement, and the rate and/or volume of the delivery makes it hard to follow the presentation.
1	Students fail to demonstrate the ability to effectively communicate in public. Speeches and presentations were not given or were of such poor quality as to indicate a lack of knowledge and ability, and students failed to demonstrate acceptable interpersonal skills.