Introduction

There has been much discussion in the past year of "assessment" and "learning outcomes." The Chancellor of the California Community Colleges has proposed that colleges be funded in part on the basis of learning outcomes. Similar ideas have been discussed for many years at Palomar. Palomar’s Mission Statement expresses the goal this way: "We evaluate the relevant skills and knowledge of all of our students so as to guide them toward meaningful and productive educational experiences, patterned to develop their abilites as effectively as their preparation allows. We will evaluate our own performance in terms of our contribution to student learning and success." Our Vision Statement, adopted in 1991, projects a future in which "Palomar College judges its work and its programs and formulates its policies primarily on the basis of learning outcomes and has a comprehensive program for assessing those outcomes and responding to its findings." The Vision Statement encourages the state of California to "reduce reliance on attendance-based funding and reward performance as measured by learning outcomes."

Proposals to reward learning outcomes have raised important questions. Many faculty fear the prospect of "outcomes" defined by bureaucratic agencies for purposes of political "accountability" that fail to serve the interests of students. A highly standardized assessment regime in which faculty are encouraged to "teach to the test" would not promote student learning or long-term success. These fears could be well founded if we wait passively for outside agencies to impose their goals upon us. If that happens, it will be very difficult to reverse the momentum of policies already underway. The goal of learning assessment as it has been defined at Palomar has been to empower faculty, staff, and students to promote student learning. As Dr. George Boggs has written recently, "educators rather than legislators should establish learning outcome goals. This is a challenge that educational leaders must accept."

It was to accept the challenge of developing a faculty-generated definition of learning outcomes and the means of assessment that could measure those outcomes that we applied last year to the Educational Master Planning Committee for an innovation grant to establish the Assessment of Learning Project (ALP). Our first task was to survey our own faculty both on their assessment practices and their views on the need for developing further assessment tools.

Purpose (Top of Page)

The immediate purpose of our project for the fall semester was to survey Palomar’s faculty in order to discover what they really think about assessment of learning as we conduct it now and as we should in the future and to formulate recommendations for further action based on what we found. To this end, we distributed written surveys to all faculty and conducted detailed personal interviews with a random group of full-time faculty. The results of these surveys and interviews are summarized below and given in more detail in the appendices. We were informed in this work by what we have learned of best practices in assessment at other colleges.

Our larger goal, which obviously will not be accomplished this year, can be summarized in three parts: (1) to establish college-level assessment of learning outcomes common to many departments or programs; (2) to raise the quality of ongoing assessment within existing departments and programs, and (3) to improve student learning through developing, applying, and modifying curriculum and instructional methods based on what is learned in the assessment process. An integrated approach will allow the college to develop classroom-level assessment that contributes to the achievement of core learning along with college-wide assessment that provides valid evidence of the value added by courses and programs.

We plan to submit a separate report on best practices in assessment at other colleges.

Recommendations (Top of Page)

We recommend that Palomar College move without delay to do the following:

1. Establish college-wide assessment of common skills on a continuing basis.

· Goals:

· Refine the list of common skills defined in this report.

· Develop faculty-based assessment processes and standards for consistent and credible assessment of these skills across the curriculum before our next accreditation self-study in the academic year 2001-2.

· Establish college-wide assessment of these common skills by 2005, the target date of our Vision Statement.

· Activities:

· Survey best practices in assessment at other colleges

· In the fall semester of 1998, conduct focus groups with faculty, administrators, staff, and students.

· In the spring semester of 1999, develop pilot assessments and standards for their evaluation.

· In the fall semester of 1999, conduct and evaluate pilot assessments.

· In the spring semester of 2000, evaluate the results and plan future activities.

Most of Palomar’s faculty agree on a body of skills that most or all of our students should master before graduating. And of those faculty who responded to the assessment survey, 74 percent agreed or strongly agreed with the statement that Palomar should develop a means to assess the knowledge and skills of all AA candidates and transfer students. At the conclusion of these recommendations we present a list of common skills that we believe can be further refined and defined. These common definitions will allow us to proceed to develop tools by which our faculty can gather more reliable and usable information about the abilities and potentials of our students than we can today. We have set our next accreditation institutional self-study as a target date for developing an assessment plan because institutional outcomes assessment is now part of the institutional effectiveness standard that will be used for our next accreditation.

2. Create standards for assessment in the classroom of skills particular to specific disciplines, certificates or vocational training.

· Goals:

· Establish standards for evaluating assessment methods.

· Establish a process for ongoing experimentation and feedback on assessment.

· Activities:

· Survey best practices in assessment at other colleges.

· In the fall semester of 1998, conduct focus groups with faculty, administrators, staff, and students.

· In the spring semester of 1999, develop pilot assessments and standards for their evaluation.

· In the fall semester of 1999, conduct and evaluate pilot assessments.

· In the spring semester of 2000, evaluate the results and plan future activities.

While many common skills can be reasonably assessed at the college level, many of our courses and programs need to assess skills in a distinctive context of a particular discipline or task. All of the faculty we heard from devote a great deal of time and energy to trying to access student learning fairly and accurately. However, most faculty expressed some frustration that they do not always have either the tools or the standards to do the job as effectively as they might. In the survey, 53 percent of faculty said that they do not have enough resources to adequately assess their students. The results of classroom assessment are not readily available to other faculty. We believe that assessment across the board would be improved by ongoing communication among faculty about the means, standards, and tools of assessment. We believe that the best standards to guide our faculty in developing more creative and effective assessment will be standards developed by and for our faculty.

3. Use assessment of student learning to evaluate and improve the curriculum.

· Goals:

· Integrate assessment into the curriculum development and evaluation process.

· Modify the curriculum to enhance student learning.

· Activities:

· Survey best practices in integrating assessment into curriculum development.

· Conduct small-scale tests of integrating assessment into curriculum in the classroom and other learning environments.

Both the survey and the interview results strongly suggest that teachers at Palomar already use assessment of student learning in order to develop and modify their courses and their teaching. But many faculty express frustration that they have little information and less control over the students who enter their classes or the experiences that build on their class work. The vast majority of faculty members we interviewed proposed ideas for gathering better information to be used in guidance and placement so that students would be fully prepared for the courses they attempt. If we can design into our curriculum paths that our students can more realistically travel, we can raise the level of student success without sacrificing excellence.

4. Involve faculty and staff in ongoing discussions and decision-making with respect to assessment issues and inform them about best practices and current research.

· Goals:

· Inform faculty and staff concerning the best evidence with respect to learning assessment.

· Generate widespread involvement among faculty across departments and divisions in formulating assessment plans and priorities.

· Activities:

· Conduct or sponsor professional development activities to inform faculty of current issues and research in assessment.

· Maintain content of an assessment site on Palomar's Internet homepage.

· Establish interdisciplinary teams of faculty and staff to develop plans for assessment.

Several members of our faculty expressed their concern in interviews about the prospect of externally mandated assessment that would add paper work and reporting requirements without increasing our ability to promote student learning and success. These concerns echo the testimony of experts who have studied assessment programs at various campuses. The best evidence indicates that assessment programs adopted merely to respond to external mandates fail to serve student interests. On the other hand, assessment that is developed by the faculty at a college can change the learning environment for the better and empower faculty with new tools to promote learning. An effective assessment program for Palomar College must be designed and generated by the faculty and staff of Palomar College. And the process of developing such a program must involve an ongoing conversation among the faculty about the priorities and choices that shape it.

5. Allocate sufficient funds to support the foregoing goals and activities.

· Goals: All of the above.

· Activities:

· Allocate at least $90,000.00 for the two academic years 1998-99 and 1999-2000 to support the above assessment activities.

· Fund released time for an Assessment of Learning Project Team, to consist of at least 50% released time for one faculty member and 25% released time for three others, representing different divisions.

· Aggressively seek outside funding to expand and hasten assessment development.

· Designate an academic dean and the Director of Matriculation to facilitate the work of the Assessment of Learning Project.

Developing an effective assessment program that can change the learning environment for the better will not require an enormous investment of resources. But it will require a constant, moderate investment of resources. In the long run, assessment is a good investment, one that will reap returns both in terms of learning and of revenue to the college. But the benefits of a locally designed and controlled assessment program can be achieved only by making a commitment to develop such a program. Palomar has been committed for year, on paper, to precisely such a task. It is time to do what we say we should.

List of Common Skills (Top of Page)

· Communication

· Speaking

· Listening

· Reading

· Writing

· Cognition

· Creative Thinking

· Analysis and Synthesis

· Problem Solving

· Quantitative Reasoning

· Interpersonal Behavior

· Social Interaction

· Teamwork

· Effective Citizenship

· Intrapersonal Behavior

· Transfer of Knowledge and Skills to a New Context

· Aesthetic Responsiveness

Summary of Faculty Survey (Top of Page)

The Assessment of Learning Project (ALP) surveyed the faculty of Palomar College on assessment techniques that they are presently using and their ideal assessment program for the future. A total of 1,219 surveys were sent out to all full-time and adjunct faculty. Two hundred fifty-five of the surveys were completed and returned. Tables displaying the results of the survey are included in appendix B.

Varied assessment techniques are employed by the faculty to assess a student's knowledge of the subject matter. Most of the techniques are traditional pen and paper tests such as multiple choice, fill in, calculation tests, essays and other writing assignments. These are used for both grades and student feedback in many cases. Non-traditional assessment techniques including portfolios, performances, demonstrations and classroom assessment techniques are used by many of the faculty.

The list of skills appearing in the table on the following page was given to the faculty members and they were asked whether the skill was an intended learning outcome, and how often it was directly assessed. All of the skills except three-mathematics, computer literacy and aesthetic responsiveness-are intended learning outcomes for more than fifty percent of the faculty who responded. These skills embrace cognition, communication, inter-personal skills, intra-personal skills, and aesthetic skills.

Nearly forty-three percent of the faculty surveyed either strongly agreed or agreed with the statement that assessment techniques are used to find out if current students in a given course are learning more than students in previous semesters. Nearly thirty-one percent disagreed or strongly disagreed. The remainder had no opinion.

When asked if Palomar College adequately assesses student skills and knowledge before they complete their studies here, nearly twenty-eight percent of the faculty agreed or strongly agreed whereas, twenty-nine percent disagreed or strongly disagreed and forty-three percent had no opinion. Almost fifty percent of the faculty surveyed agree that Palomar College ensures that our AA graduates possess a minimum level of skills and knowledge appropriate to an AA degree. Seventy-four percent of the faculty surveyed strongly agreed or agreed with the statement that Palomar College should develop a means to assess knowledge and skills of all AA candidates and our students transferring to four-year institutions.

Finally, fifty-three percent of the faculty report that they do not have enough resources to adequately assess their students.

Summary of Faculty Interviews (Top of Page)

The Assessment of Learning Project Team also interviewed twenty-nine faculty members, covering a wide range of departments, as a follow-up to the survey. The results of these interviews are attached as an appendix, but a summary is presented here:

· Overall, about half of the faculty interviewed are using a formal entry assessment at the beginning of their courses. In addition, some faculty informally assess students at the beginning of their classes.

· A variety of assessment techniques are used by the faculty interviewed. Most use traditional testing, while other techniques include essays, projects, book reports, and portfolios. These are listed in the appendix.

· Other than assigning grades, most of the faculty interviewed use assessments of student learning to help students in different ways. For example, 1/4 of faculty use results to give feedback to students, and another 1/4 use the results to make improvements in their teaching and in their course curriculum. Nearly half use the results for other creative purposes. See the appendix for different uses. There was only one faculty member who said that s/he does not use the results for anything other than grading.

· About 1/3 of the interviewees have a formal way of finding out whether their students this semester are learning either more or less than their students from previous semesters. Slightly less than 1/3 have an informal way of ascertaining this, and about 1/3 responded that they don't have a way of comparing this semester's results with previous semesters.

· The majority of those interviewed are assessing the following skills throughout the semester:

                 Reading

Writing

Creative Thinking

Listening

Speaking

Reasoning

Problem Solving

Analysis and Synthesis

Transfer of Knowledge and Skills to a New

Context

Computer Literacy

Effective Citizenship

Social Interaction and/or Teamwork

Value Judgments and Independent Decision Making

About half are not assessing Mathematics or Aesthetic Responsiveness.

· The majority of faculty interviewed said that by using computerized assessment tests, student interviews, and personal guidance, Palomar can assess the knowledge and skills of incoming students in order to place them in the proper programs and provide guidance toward a successful college career.

· As to how best to administer the assessment of the above noted skills, the majority of those interviewed suggested that assessment for reading, writing, and mathematics should be done at the college-wide level, and the rest should be done in the classroom. Only a very few of those interviewed believe that it should be done at the department or division level. The majority of faculty who responded to this question said that these skills should be assessed continuously, and about an equal number of faculty suggested pre- and post-testing which could include methods such as a written instrument, interview, and computerized testing.

· Some faculty said that by tracking students' and instructors' records we should be able to find out whether our students this semester are learning more or less than students in previous semesters. But at the same time, faculty believe that it is very difficult to do and that grades are not a good indicator.

· Some faculty suggested that departments need to do self-assessments to evaluate the performance of their individual programs. Other suggestions include creating a new assessment vehicle, comparing pre- and post-test results with other colleges, and tracking students.

· See the appendix for a detailed listing of skills that faculty from each department believe the educated layperson should acquire from their respective disciplines. There were skills cited that were common to several disciplines, even among those disciplines which aren't generally skills-oriented. These include communication, reading, writing, listening, reasoning, and problem solving.

• 

   

Appendix A - Survey Instrument

Appendix B - Survey Responses

(Top of Page)

(Top of Page)

Appendix C -Interview Questions (Top of Page)

Interview Questions

1) Do you do any diagnostic entry assessment at the beginning of your

courses?

If yes, briefly describe this assessment and its goals.

2) What assessment techniques do you employ to test understanding of your

subject?

How effective are they?

3) How do you use assessments of student learning other than to assign grades?

4) Do you have any formal or informal ways of finding out whether your

students this semester are learning more or less than your students in

previous semesters?

If yes, describe these methods.

5) I am going to give you a list of skills.

How do you assess the acquisition of these skills? When? How often?

In responding to the following questions, try to describe how you think

Palomar College should be assessing students' skills and knowledge five

years from now. Assume that resources, administrative support, and faculty

support are available to develop new means of doing things.

6) How should Palomar College assess the knowledge and skills of incoming

students in order to place them in the proper programs and provide guidance

toward a successful college career?

7) Please refer to the list of skills in front of you. Feel free to

comment on any or all of these skills in your response to the following

questions:

a) At what administrative level of the college can these skills best be

assessed? The college, the division, the department, the classroom, or some

other level?

b) When and how often in the student's career should these skills be assessed?

c) How should these skills be assessed?

8) How should we find out whether our students this semester are learning

more or less than students in previous semesters?

9) How should we find out how well individual programs are performing?

10) What specific skills from your discipline are essential for the

educated layperson to acquire?

List of Skills:

Reading

Speaking

Writing

Mathematics

Listening

Reasoning

Problem Solving

Creative Thinking

Aesthetic Responsiveness

Social Interaction and/or Teamwork

Value Judgments and Independent Decision Making

Transfer of Knowledge and Skills to a New Context

Computer Literacy

Effective Citizenship

Analysis and Synthesis

Other (please specify)

Appendix D - Interview Responses

- Diagnostic writing test

- Reading inventory standardized tests:

- Nelson-Denny (nationally normed inventory)

- Test of Adult Basic Education Skills, reading section (for lower or developmental courses)

-Quick assessment for each learner individually: San Diego Quick Assessment or wide range achievement test.

- If new, they must take the ESL CELSA test and must place in level 4

- Writing assessment

- Dance technique very visible, observation

- First project, solo project, assess tools; entering over several weeks

- Need basic math skills to do well (13 question quiz)

- Pre- and post-testing / survey test (oral and written language skills

- Assessment and guidance (DSP&S)

- Writing sample (support class for English 50)

- Formal checklist of computer skills for computer classes in ESL

- 10 questions pre & post test used in geography & geology (NSF grant)

- Informal observation

- Pretty early; writing sample, brief research opinion profile

- Observe proficiency level in Spanish capability

- Personal recommendation

- To get into nursing program: Sci., Eng., very generic. Once in program, no pretest for specific courses, but we go over their folders from previous course(s?)

- Students will introduce themselves and inform the class of what they want to learn and what they expect to learn from the course

- Hand out prerequisite questions in lower level classes; help them to be successful and/or advise them to seek another class

- Personal interviews, or they come in with all prerequisites fulfilled

2. What assessment techniques do you employ to test understanding of your subject? How effective are they?

- Homework/Assignments 4

- Quizzes 11

- Tests: 21

- multiple choice 7

- problem solving 1

- analysis 1

- true/false 3

- fill-in 1

- matching 1

- oral 1

- theory & practice 1

- skills/hands-on 4

- essay 5 read & follow directions 1

- Project 5

- Short Essay 1

- Essay 1

- Short Paper 1

- Research Paper 1

- Book Report 2

- Case Study 2

- National Standardized Exam 1

- Survey of Internship Supervisor 1

- Interaction 1

- Monthly Reports 1

- Videotaping 1

- Analyzing Scenes 1

- Script Analysis 1

- Presentation of Scene 1

- Role Playing 1

- Direct Debate 1

- Writing Sample 1

- Journals 1

- Class Discussions 1

- Worksheet Activities 1

- Lab 2

- Small Group 1

- Group Assessments 1

- Visual Evaluation (Dance) 1

- Movement (Dance) 1

- Hand-in Notes 2

- Individual Assessments 1

- Critique 1

- Portfolios 1

- Goal Paper 1

- IB test 1

- Study and plan of care 1

- Clinical assessment 1

- State Board Exam 1

- Conferencing 1

- Internet use 1

3. How do you use assessments of student learning other than to assign grades?

- Feedback to students

- Informal obervation, feedback, one-to-one individual basis

- Diagnostic, up-to-task feedback

- Portfolios, relationships, constant feedback, individual instruction

- Questions and answers in classroom discussions. Diagnostic feedback.

Participation is very important

- Objectives - employee feedback, student feedback. Job offers, promotion

- To complete students' folders, then review their work on a weekly basis individually with them.

- Self-evaluation on my part

- At the beginning, use for adjustment

To guide what I need to focus on in the course

Use to evaluate how I can change the course

- Use a lot of self assessment curriculum and techniques - adjust way of working

- Feedback for techniques as well as assessment

- Feedback to find out his own effectiveness and to adjust emphasis when necessary

- Finding out what to teach and when to teach it

- She begins teaching at the level where she finds the most errors

- to help students to learn. I see tests more effective as a learning device than to assign grades.

- whether or not a student is progressing

- check attendance: whether or not students' performance is related to their attendance

- No

- Students know what to expect when they go to the next course.

- Make them help another student. " To teach is to learn"

- Assess their progress, Feedback tool

- I like to know where the class is at any time, do not like surprises

Assess whether students are ready to go on to the next level

Recommend to repeat sometimes

- for them to understand responsibility for their own safety

- Growth, creativity

- Tap into life experience

Communication skills, critiques

- Student initiative, attitude in class, participate in discussion

- Filling gaps in understanding teaching tool

- Referral to counseling or other courses. Plan their future

- To assess placement and progress through levels to determine hands-on work: safe practice

4. Do you have any formal or informal ways of finding out whether your students this semester are learning more or less than your students in previous semesters? If yes, describe these methods.

(A) Yes--Formal:

- Depends on the students (students more independent now)

- Formal - testing and term paper

- look at how well they are performing in their past exiting tests. Their affective score: how

many positive responses they have. Compare the number of positive responses from this

semester to the next.

- Passing of English 50 final. Observe writing growth, track students

- Begun a survey of students, their attitudes toward government and knowledge.

- Intuition, looking at past work, use the same test but different versions, use own tests,

reviewing assessment techniques

- Keep accurate records of all the points received by my students on various assignments and I

compare them semester to semester. I also keep some of the written work that my students

have done and compare the quality of work.

- Yes, I recycle exams. I have a file of exams and I make notes on them. I see negative trends

- Formal: quizzes

- High school students, compare final essays

- He selects, eg. the "mineral" quiz to compare last semester with this semester. Also assessed

with computer use vs. non-computer use.

(B) Yes--Informal:

- very informal, need to define some kind of post test. Thinking about mini-certificate for

program

- Can visually tell, by observation, can only go as fast as the group will go. Can tell with written

work, and the way they respond in class discussions (Dance)

- Informal: Finishing class, test pieces on hardwood. Last year I had them restore a piece of

furniture. Quality of the test pieces improved when they had to apply it immediately.

- Informal: Quality of scene presentations

- Observation and intuition. Observe patterns (waves) of students.

- observation

- Informal way: I interview students; I will have them explain their reasoning on certain topics

(interactive courses, not in traditional courses)

- Informal. One-on-one assessments - tracking students

(C) No:

- No, except our state and national board exams

- No

- No, maybe informal. Interview at beginning of semester. Fill out a sheet.

- No

- No formal ways

- No formal ones - In the past, I gave similar final exam problems, but in most courses, no.

- No, really - program evaluation

- No, no experience in previous environment. Have interview at beginning of semester.

- No, never done

- No, the only formal assessment that serves their purpose is the Academic Placement to get into

ESL 101.