Competencies
- Utilize student learning and development theories and scholarly research to inform content and design of learning outcomes and assessment tools. (AERD)
- Consider strengths and limitations of methodological approaches when applying findings to practice in diverse institutional settings and with diverse student populations. (MDCA)
- Participate in the design of qualitative and quantitative AER projects, determining appropriate methods and analyses for each. (MDCA)
In casual conversation, the terms assessment, evaluation, and research might be used interchangeably. However, the purposes differ in significant ways: the purpose of assessment is improvement and accountability; the purpose of evaluation is judgement against a standard; and the purpose of research is to produce generalizable knowledge. To accomplish this purpose of generalizable knowledge, there are several differences in research methods compared to assessment and evaluation.
Audience
While assessment and evaluation tend to have an internal audience, research is intended for a wider audience, most often via publication in peer reviewed research journals, plus presentation on posters and in sessions at professional conferences. As such, research often includes context to assist the less familiar audience with understanding the setting of the project. It also should include discussion of limitations to help the reader determine generalizability, as well as to guide future research.
Literature review
Unlike assessment’s focus on application, since research is focused on more common truths, it is most often framed using a body of scholarly research and theory to inform the design as well as interpretation of results. This literature review, familiar as chapter two in dissertations, is essential to place the research study in the context of what is already known in the field. Most often, a research study tests a theory from a new perspective, such as a new audience, addition of a variable, or revision of an intervention. Research is more likely accepted for publication when it offers a defensibly small addition to a strong foundation of knowledge and methodology.
Design
Many methods of collecting data are utilized in both assessment and research projects, such as surveys, interviews, focus groups, and observations. However, research should supplement the data collection method with thoughtful design, typically based on literature suggesting a causal relationship between one or more independent variables on one or more dependent variables. Research more often uses experimental design, which is the random assignment of participants to different groups, typically in variations of independent or repeated measures.
Independent measures involve using different participants into groups that have varied experience, such as experiences in two different residence halls, or experiences of students who did or did not participate in a program. Repeated measures involve using the same participants over time, such as the experiences of students before, during, and after a program. Independent measures require more participants than repeated measures to average differences between participants, but it does not include the negative effects associated with repeated testing.
Independent measures and repeated measures have numerous variations and combinations worth considering in service to the research question. If participants cannot be randomly assigned to groups, the design is called quasi-experimental, and extra consideration is required to explore confounding variables that could explain both self-selection of group and outcome. In the residence hall example, perhaps a camaraderie observed in Hall A versus B is because preexisting friends selected Hall A, rather than because Hall A had better friend-building programs. Most student affairs programs do not involve the random assignment of participants, so typically a research project in student affairs involves a quasi-experimental design.
Institutional Review Board
Institutional Review Boards are independent research ethics committees “charged with protecting the rights and welfare of participants in clinical trials and other human subjects research studies (UF IRB, 2020). UF faculty, staff, and students are required to obtain IRB approval for all human subjects’ research before conducting the research. A project can be categorized as for internal improvement only and thereby not subject to IRB review. However, a project that seems more like assessment than research is still required to go through IRB if the intention is to pitch findings as generalizable knowledge, say in a conference presentation.
IRB submissions are categorized as full, expedited, or exempt. Studies requiring full review typically involve more than minimal risk to participants, for example the administration of an experimental medication. This review is completed at a meeting of the full review board, which are held monthly. More often a student affairs research project involves the non-invasive study of behavior and is categorized as expedited, in which one IRB member reviews the project. Many student affairs research projects are also categorized as exempt, which is not the same as not being reviewed, but rather means an IRB member determined the project requires no further oversight because it has minimal risk and is educational research, involves benign behavioral interventions, and/or uses publicly available data.
All research projects require the researchers to maintain the IRB 803 training (renewal every three years) in UF’s MyTraining website. Staff have access to the IRB submission site after this training is logged in the system. Most student affairs research falls under IRB-02, which is responsible for non-medical studies on UF campus. An IRB submission needs to include copies of any surveys, including solicitation emails, informed consent documentation, and any incentives. Note that UF considers random drawings in research projects to be a lottery, which is prohibited, so other creative methods of rewarding participation should be considered.
Time
Considerations for the audience, literature review, design, and IRB result in research typically taking longer than assessment projects, especially when including the lengthy process to secure publication of a research article. It is not unusual for an article at the time of publication to be about a program that occurred two years prior. The time commitment, and the expertise required to conduct research, result in much less scholarship coming out of student affairs than possible considering the wealth of programs available for study.
Benefits
While research has been distinguished in this module from assessment, they are not mutually exclusive. Research can be thought of as a robust type of assessment, and as such, any assessment project is worthy of consideration for a full research study. Even if the time and expertise are not available to expand an assessment project in this way, research methods can help improve the reliability and validity of the project.
When deciding whether to invest in a research project, in addition to the time and expertise requirements, there are a variety of benefits to consider. Research projects contribute new knowledge to the field. This contribution brings job satisfaction to the researchers and participants, knowing they have helped improve our profession and service to students. The contribution can also bring notoriety to the researchers, departments involved in any interventions, and the institution, building social capital in professional circles, attracting new talent, etc.
Research is also more likely than assessment projects to qualify for external grant opportunities, possibly funding a program and/or positions. There are also opportunities to partner with faculty at UF and elsewhere, lending expertise and time to the project. There are protocols to collaborate on projects with other institutions, including the single IRB (sIRB) process, as well as a deferment to another institution’s review.
Conclusion
With proper attention, research can be a worthwhile investment for student affairs practitioners. Even when deciding not to pursue research, the methods covered here can improve the value of an assessment project. Furthermore, student affairs staff can inform their practice by being educated consumers of research efforts conducted elsewhere, critiquing the applicability of findings to prove and improve our service to students.