An initial self-assessment, conducted by staff within the functional area, provides for the collection and review of evidence by the people most familiar with the area.
SAAR will work with the Executive Director/Director of the department to facilitate the review process. The department under review will need to have a designated professional staff to coordinate the review process with SAAR, a broader group on the self-assessment team, and the entire department ready to support the comprehensive review by providing relevant evidence:
- Department information
- Vision, and value statement
- Organization structure and staff profiles/qualifications
- Strategic plans
- Most recent three years’ annual reports
- Most recent three years’ budget reports
- Institutional data
- Stakeholders, i.e. whom do we serve
- Program participation, and utilization of services.
- Student satisfaction
- Learning outcomes, GPA, retention, graduation, employment, and other student learning outcomes.
- Benchmarking data
- SERU
- NASPA benchmarking studies
- Other benchmarking studies, such as ACUHO-I Housing Assessment.
The self-assessment may also include surveys to staff and students to get their feedback about the program(s) under review. Individual interviews and/or focus groups may also be conducted using purposeful sampling.
Self-Assessment Guide Organization and Process
The Self-Assessment Guides (SAG) translate functional area CAS standards and guidelines into tools for conducting a self-assessment. Educators can use this SAG to gain informed perspectives on the strengths and deficiencies of their programs and services as well as to plan for improvements. Grounded in the reflective, self-regulation approach to quality assurance in higher education endorsed by CAS, this SAG provides institutional, divisional, departmental, and unit leaders with a tool to assess programs and services using currently accepted standards of practice.
CAS developed and has incorporated a number of common criteria that have relevance for each and every functional area, no matter what its primary focus. These common criteria are referred to as “General Standards,” which form the core of all functional area standards.
CAS standards and guidelines are organized into 12 components, and the SAG workbook corresponds with the same sections:
Part 1. Mission | Part 7. Human Resources |
Part 2. Program and Services | Part 8. Collaboration and Communication |
Part 3. Student Learning, Development, and Success | Part 9. Ethics, Law, and Policy |
Part 4. Assessment | Part 10. Financial Resources |
Part 5. Access, Equity, Diversity, and Inclusion | Part 11. Technology |
Part 6. Leadership, Management, and Supervision | Part 12. Facilities and Infastructure |
For each set of standards and guidelines, CAS provides a Self-Assessment Guide (SAG) that includes a recommended comprehensive self-assessment process for program assessment. Seven basic steps to using a SAG are suggested for implementing a functional area self-assessment. These steps are modified below to include our process of a subsequent external review, followed by action planning and beyond:
- Plan the Self-Assessment Process: Map out steps for process, develop timeline, build buy-in with all stakeholders, and explicitly identify desired outcomes of the self-assessment
- Assemble and Educate the Self-Assessment Team: Determine who should be on the team and how to educate the team about the self-assessment process
- Identify, Collect, and Review Evidence: Define what constitutes evidence; then gather, collect, manage, and review evidence
- Conduct and Interpret Ratings Using Evidence: Clarify team’s rating criteria; employ a process for rating [small group, individual, staff]; negotiate rating differences; and manage group ratings.
- External Review: Assemble a team of external reviewers (student, faculty, staff, led by consultants external to the university) to critique the self-assessment, conduct interviews and focus groups in a campus visit, then deliver a report reviewing the functional area.
- Prepare an Action Plan: Identify discrepancies, corrective action, and recommended steps (e.g., identify strengths, weaknesses, recommendations, benchmarks for achievement, resources, timeframe, and responsible individuals) taken from both the self-assessment and external review documents.
- Close the Loop: Combining results from self-assessment and external reviews, put action plans into practice; work to navigate politics and secure resources; identify barriers to overcome; and build buy-in to the comprehensive review results.
The first four steps are completed in conducting a self-assessment.
Step 1: Plan the Self-Assessment Process
Prior to beginning a comprehensive review, division and functional area leaders need to determine the area (or areas) to be reviewed and the reasons for the project. This may be dictated by institutional review cycles or planning for accreditation processes, or it may result from internal divisional goals and needs. Explicitly identifying desired outcomes and key audiences for a self-assessment will help leaders facilitate a process that makes the most sense for the project.
Critical first phases of a comprehensive review include mapping out the planned steps and developing timelines. Leaders will also want to build buy-in with stakeholders of the functional area. In the initial planning stage of the self-assessment process it is desirable to involve the full functional area staff, including support staff members, knowledgeable students, and faculty members when feasible. This approach provides an opportunity for shared ownership in the review.
Step 2: Assemble and Educate the Self-Assessment Review Team
The second step is to identify an individual to lead the self-assessment process. CAS recommends that the coordinator be someone other than the leader of the unit under review; this facilitates honest critique by the review team and enhances credibility of the final report. Once a leader is designated, members of the review team need to be identified and invited to participate. The membership should represent the range of services offered by the unit.
In preparing the team for the self-assessment, it is imperative to train the team on the CAS standards, as well as self-assessment concepts and principles. CAS standards and guidelines are formulated by representatives of 40 higher education professional associations concerned with student learning and development. The CAS standards represent essential practices; the CAS guidelines, on the other hand, are suggestions for practice and serve to elaborate and amplify standards through the use of suggestions, descriptions, and examples. Guidelines can often be employed to enhance program practice. Following a long-standing CAS precedent, the functional area standards and guidelines—presented as an appendix to the self-assessment instrument—are formatted so that standards (i.e., essentials of quality practice) are printed in bold type. Guidelines, which complement the standards, are printed in light-face type. Standards use the auxiliary verbs “must” and “shall” while guidelines use “should” and “may.”
In this self-assessment instrument, the CAS standards have been translated into criterion measures and grouped into subcategories for rating purposes. The criterion measures are not designed to focus on discrete ideas; rather, the measures are designed to capture the major ideas and elements reflected in the standards. For each of the 12 component parts, team members will rate clusters of criterion measures. If the assessment team decides to incorporate one or more of the guidelines into the review process, each guideline can be similarly translated into a measurable statement to facilitate rating.
As a group, the review team should examine the standards carefully and read through the entire self-assessment guide before beginning to assign ratings. It may be desirable for the team, in collaboration with the full staff, to discuss the meaning of each standard. Through this method, differing interpretations can be examined and agreement generally reached about how the standard will be interpreted for purposes of the self-assessment.
Step 3: Identify, Collect, and Review Documentary Evidence
Collecting and documenting evidence of program effectiveness is an important step in the assessment process. No self-assessment is complete without relevant data and related documentation being used. It is good practice for programs to collect and file relevant data routinely, which can then be used to document program effectiveness over time. Available documentation should be assembled by the unit under review and provided to the review team at the outset of the study. The team may request additional information as needed as the review is conducted.
Documentary evidence often used to support a review includes:
- Student Recruitment and Marketing Materials: brochures and other sources of information about the program, participation policies and procedures, and reports about program results and participant evaluations
- Program Documents: mission statements, catalogs, brochures and other related materials, staff and student manuals, policy and procedure statements, reports, contracts, and staff memos
- Institutional Administrative Documents: statements about program purpose and philosophy relative to other educational programs, organizational charts, financial resource statements, student and staff profiles, and assessment reports
- Research, Assessment, and Evaluation Data: needs assessments, follow-up studies, program evaluations, outcome measures and methodologies, and previous self-assessment reports
- Staff Activity Reports: annual reports; staff member vitae; service to departments, colleges, university, and other agencies; evidence of effectiveness; scholarship activities, and contributions to the profession
- Student Activity Reports: developmental transcripts, portfolios, and other evidence of student contributions to the institution, community, and professional organizations; reports of special student accomplishments; and employer reports on student employment experiences
In the SAG, each section provides recommended evidence and documentation that should be collected and compiled prior to conducting ratings. The evidence collected is likely applicable across numerous sections.
Raters can best make judgments about the program expectations articulated in the standards when they have a variety of evidence available. Multiple forms of evidence should be reviewed and reported in the narrative section of the SAG worksheets. Through the rating process, a self-assessment team may identify a need to obtain additional information or documentation before proceeding, in order to lend substance to judgments about a given assessment criterion. Evidence and documentation should be appended and referenced in the final self-assessment report.
Step 4: Conduct and Interpret Ratings Using Evidence
When the self-assessment team has gathered and reviewed necessary evidence, they will be able to assign and interpret ratings to individual criterion measures, following three steps.
Rate Criterion Measures
- Team members individually rate criterion measures based on their understanding of the evidence.
- Team discusses and assigns collective ratings for criterion measures.
Provide Narrative Rationale
- Document the reasoning and evidence for the rating assigned to each subsection, in the space provided for Justification for Rating.
- Explain what evidence has been collected and reviewed to support individual and/or team ratings and judgments.
- Provide information for follow-up and relevant details about ratings (e.g., if Partly Meets is assigned as a rating, what aspects of the program or service do and do not meet which standards statements).
Answer Overview Questions (In the Instrument)
- Respond, in writing in the space provided, to the Overview Questions that immediately follow the rating section of each of the 12 components.
- Use answers to the Overview Questions, which are designed to stimulate summary thinking about overarching issues, to facilitate interpretation of the ratings and development of the self-assessment report.
Criterion measures are used to assess how well areas under review meet CAS standards. These criterion measures are designed to be quantified using 3-point rating scale. In addition to the numerical rating options, Does Not Apply (DNA) and Insufficient Evidence/Unable to Rate (IE) ratings are provided. This rating scale is designed to estimate broadly the extent to which a given practice has been performed.
CAS CRITERION MEASURE RATING SCALE
DNA | IE | 0 | 1 | 2 |
---|---|---|---|---|
Does Not Apply | Insufficient Evidence/ Unable to Rate | Does Not Meet | Partly Meets | Meets |
It may be determined that a criterion measure used to judge the standard is not applicable for the particular program (e.g., a single sex or other unique institution that cannot meet a criterion measure for that reason). In such instances, raters may use a rating of DNA and, in the self-assessment report, describe their rationale for excluding the practice in the criterion measure. The IE response can be used when relevant data are unavailable to support a judgment. When either the DNA or the IE ratings are used, an explanatory note should be provided in the report. Items rated with 0 should generate careful group consideration and appropriate follow-up action.
Program leaders may wish to incorporate additional criterion measures, such as selected CAS guidelines or other rating scales, into the procedures before the self-assessment process begins. Such practice is encouraged, and the SAG instrument can be amended to incorporate additional criterion measures for judging the program. In such instances, additional pages to accommodate the additional criterion measures may be required.
Whatever procedures are used to arrive at judgments, deliberate discussions should occur about how to initiate the rating process and select the optimal rating strategy. In such discussions, it is expected that disagreements among team members will occur and that resulting clarifications will inform all participants. It is important that the team achieve consensual resolution of such differences before proceeding with individual ratings.
CAS suggests a two-tiered (individual and group) judgment approach for determining the extent to which the program meets the CAS standard. First, the self-assessment team members (and functional area staff members, if desired) individually should rate the clusters of criterion measures using separate copies of the CAS Self-Assessment Guide. In addition, they will need to document their reasoning and evidence for the rating assigned to each subsection in the space provided for Justification for Rating. This individualized rating procedure is then followed by a collective review and analysis of the individual ratings.
The individual ratings should be reviewed, discussed, and translated into a collective rating by the team; then the team is ready to move to the interpretation phase of the self-assessment. Interpretation typically incorporates discussion among team members to assure that all aspects of the program were given fair and impartial consideration prior to a final collective judgment. At this point, persistent disagreements over performance ratings may call for additional data collection.
After the team review is completed, a meeting with relevant administrators, staff members, and student leaders should be scheduled for a general review of the self-assessment results. The next step, including discussion of alternative approaches that might be used to strengthen and enhance the program, is to generate steps and activities to be incorporated into an action plan. This step is best done by the unit staff, informed by the results of the review and, when feasible, in consultation with the review team. The Work Forms will guide this process.
Share this information in digital format with the SAAR prior to the External Review Team (ERT) visit for cleanup suggestions and organization for consumption by the ERT.
Rating Examples
Rating Standard Criterion Measures
All CAS standards, printed in bold type, are viewed as being essential to a sound and relevant program or service that contributes to student learning and development. Many of the statements contained in CAS standards incorporate multiple criteria that have been grouped for rating purposes. Consequently, raters may need to judge several standards statements through a single criterion measure. Using the “Ethics” standards as an example, the following illustrates how criterion measures are grouped into subcategories for rating.
Using Guidelines to Make Judgments about the Program
As discussed above, program leaders may wish to include selected CAS Guidelines to be rated along with the standards. To accomplish this, criterion measure statements must be written for the guidelines selected. The self-assessment team can readily create statements to be judged as part of the rating process. Programs generally considered in compliance with the standards especially can benefit by using guidelines because guidelines typically call for enhanced program quality.
Not all programs under review will incorporate guidelines to be rated as part of their self-studies. Even though the guidelines are optional for rating purposes, raters are strongly encouraged to read and review them as part of the training process. When CAS Guidelines or other criterion measures are rated, they should be treated as if they were standards.