Advanced Online Assessment Techniques

From E-Learning Faculty Modules

Contents

Module Summary

Assessment methods are very domain-field specific and often very particular to various instructors and practitioners in a field. This module highlights some of the strategies that are commonly used in assessing learners in a measurable way to both inform the teaching and learning and also to evaluate learner performance. This module focuses on current capabilities in learning / course management systems (L/CMSes) for automated and manual assessments.

Takeaways

Learners will...

  • Gain perspective on how assessments in a domain field are arrived at and perpetuated (from a universe of possible assessments).
  • Differentiate between low-value and high-value assessments, and formative and summative assessments, and get a sense of when each should be used.
  • Acquire a sense of some different methods of online assessment and the pedagogical rationales behind them.
  • Review different types of online assessments that are technologically feasible in learning / course management systems (L/CMSes) and digital learning object repositories (using SCORM or Tin Can API). Differentiate between automated (machine-executed) and manual (human-executed) assessments.
  • Review the types of assessments that are difficult (or impossible) to achieve in an online or distance setting. Consider how these types of assessments may be mitigated.
  • Strategize how to find the right mix of assessments for a particular curriculum.

Module Pretest

1. What are the basic types of assessments available? How are various types of assessments arrived at in a particular domain field?

2. What is the difference between low-value and high-value assessments? Formative and summative assessments? When should each be used?

3. What are some different methods of online assessment? What are some of the different pedagogical rationales behind the usages of each? Why do assessments have to be measureable?

4. What are some common types of online assessments that are technologically feasible in learning / course management systems (L/CMSes) and in digital learning object repositories? What are automated vs. manual assessments?

5. What are the assessment types which are difficult or currently “impossible” to create online? How are these mitigated?

6. What are some methods to find the right mix of assessments for an online course or learning sequence?

Main Contents

In terms of good design, a learning sequence is defined first by its learning objectives (the measureable goals of the learning) and learning outcomes (the knowledge and skills that learners will have after they have completed the learning sequence).

Learning assessments should be observable and measureable. They should be fair. If multiple professionals are assessing student work, those professionals need to “norm” among themselves in order to ensure that they are assessing students fairly and consistently.

Regulatory agencies and professional associations will help define the types of assessments that are expected for a particular field as well.


1. The Context of Domain Fields

The different types of assessments used in a particular domain field are heavily dependent on the following:

  • the history of the field and its evolution
  • the values and principles of the field
  • the founding members and main practitioners and innovators in the field
  • the way research is conceptualized, conducted, and validated in the field
  • the methods of documentation of the learning
  • the practical knowledge and skills that novices in the field will need to know as professionals, and other factors.

As a general rule, subject matter experts (SMEs) in a domain field are the ones who determine how to effectively assess their learners. This module highlights some basic approaches to online assessment.


2. Low- and High-Value Assessments / Formative and Summative Assessments (and When They Should Generally be Used)

Low-Value Assessments

A low-value assessment is one which is low-risk for the learner and which does not involve a lot of points (or no points, or only extra-credit points). A low-value assessment can also be one which may be done again-and-again for aggregated points. The contents of low-value assessments tend to be fairly simple (vs. difficult).

These may be acclimation types of assignments to help learners get used to a learning / course management system or virtual immersive world—where learners have a kind of treasure hunt in the system. Or these may be syllabus quizzes to ensure that learners have read that electronic document thoroughly.

These types of low-value assessments may be practice assignments. They may be draft assignments that may be revisable for much higher credit. Often, low-value assignments create opportunities for low-risk tries for learners and opportunities for constructive feedback by the instructor.

High-Value Assessments

High-value assessments are often those that coalesce plenty of complex learning. These may be research papers, presentations, interviews of experts, field trips and field research, lab experiments, and even group projects. Cumulative midterms and finals tend to be high-value assessments. Comprehensive exams (or oral exams) may be very high-value ones because these determine whether a student is deemed worthy to join a doctoral program.

These types of assessments tend to assess student capabilities beyond even the subject matter about which they’re being tested. These are worth plenty of points, and they may determine entry to or removal from a program of learning.

Formative Assessments

A formative assessment is one which is used to help the learner learn (and to help the instructor teach), like “muddiest point” and “minute papers”) (Angelo & Cross, 1993). These tend to be low-value assessments.

Summative Assessments

A summative assessment assesses the learner’s comprehension and is used to “place” a learner in a certain learning level. It is not used to help the learner learn. Rather, it’s for the instructor to assess the learner. Often, there is a lot of build-up to summative assessments, and these assessments bring together plenty of complex learning and high-level executive function thinking by the learners. These tend to be high-value assessments.


3. Different Methods of Online Assessment and their Respective Pedagogical Rationales

Convergent Learning Assessments

In a very general sense, learning may be defined as convergent or divergent. Broadly speaking, convergent learning means that learners must put out a particular answer that is already pre-defined. The learning converges to a common space. This level of learning is often done at the lower levels of Bloom’s Taxonomy of Learning Domains—in terms of remembering and understanding. These types of assessments may be multiple-choice exams, for example, with pre-determined answers. (Some very complex types of learning may be considered convergent particularly if there may be only one right answer—as in some physics or chemistry or mathematics problems.)

Divergent Learning Assessments

Divergent learning refers to more innovative or creative types of learning where a wide number of responses may well be acceptably correct. This refers to the higher levels of Bloom’s Taxonomy of Learning Domains: applying, analyzing, evaluating, and creating. In higher levels of learning, learners have a stronger base of knowledge in the field and are better able to handle complex ideas. Examples of assignments and assessments would include case analyses, research papers, problem-solving and troubleshooting, presentations, and innovative designs.

Essay exams are considered more divergent sorts of qualitative assessments because learners often may respond to a writing prompt in a variety of different ways and still be considered accurate. Writing is often assessed based on criteria defined in a rubric.

Group assessments are also considered a type of divergent learning assessment. These require communications, collaboration, planning, and coordination—which are also generally assessed along with the quality of the intermediate and final products that the groups evolve. J.A. Nicolay (2002) writes that it is hoped that such connections will forge professional connections between the learners well beyond the classroom. Instructors often want to ensure that all the team members contribute sufficiently to the group work. They want to see synergies that emerge from the collaborations, or the combinations of skill sets. They will also be evaluating the closeness of the final product to the stated project standards—which are defined early in checklists or rubrics or some other forms. Some group projects may be assessed by professionals in the field (such as professional architects or artists or engineers—who will evaluate the project based on in-field standards).

Project-based learning is another form of divergent learning because projects tend to culminate a range of learning in an applied way. Often, projects may be expressed in semi-professional portfolios that students will take with them into the employment environment. A professional from the field may be brought in to guest-critique student projects or portfolios. Instructors may conduct live one-on-one consultations with the student about his or her work.

Another form of divergent learning involves service learning (field experiences) with students placed in a real-world context (domestically or abroad) and learning in the professional context. In this situation, an on-ground supervisor will be monitoring the student and supervising him / her. The professor at the home campus has some input and may make a site visit or two, but in this case, the student has multiple feedback channels from professionals in the field.


4. Common Types of Online Assessments in L/CMSes and DLO Repositories (Automated and Manual Methods)

In online learning that uses learning / course management systems (L/CMSes), there are built-in assessment tools that may enhance the work of the instructor.

Automated Comprehension-Type Exams

For convergent learning, there are a number of automated tools that enable the creation of various question types:

  • True/false
  • Multiple choice
  • Matching
  • Ordering
  • Sorting

Text Analysis

  • Short-answer (based on text analysis)
  • Essays

Short-answers may be graded based on text-analysis, but these are very sensitive to mis-correction based on how the answers are set up. Some L/CMSes enable the correcting of full essays based on text analysis algorithms. Whether these are effective or not should be tested.

While many of the bulleted question types seem simple, it is helpful to note that all sorts of multimedia and equations may be integrated into the questions.

Message Boards

Another type of assessment occurs in Message or Discussion Boards. These enable learner interactivity with the sharing of text messages, images, slideshows, links to online resources, videos, and other elements. While some instructors use the back-end quantitative function to measure student interactivity, the preferred method is to have trained graduate teaching assistants (GTAs) and the faculty members interacting with learners to enhance their work.

Peer Assessments

Shared learning spaces where peers may post and share their work enable peer assessments. The rubrics used for the assessments may be those that are instructor-created, student-created, or instructor-and-student created. The peer assessments are generally anonymous, and the focus is to develop both the learners receiving the constructive critique and the learners creating the constructive critiques.

Tracking over Time

L/CMSes and virtual immersive worlds both enable the tracking of learners’ participation and work over time. Such longitudinal assessments are often used programmatically to track student work over time.

Manual Methods

Manual types of assessments are usually the evaluations for divergent learning

  • Assessments of student papers and research projects
  • Assessments of student designs
  • Assessments of portfolios

It is too prohibitively difficult to try to program a computerized response to a wide range of diverse learner projects.


5. Different (or Impossible) Online Assessments and Possible Mitigation

Traditionally, it is thought that some types of assessments should be done in a face-to-face setting. Orals are usually held in a face-to-face way with the doctoral committee. This is mitigated with real-time video-to-video assessment to mitigate for the distance. (Web conferencing is not at the quality where this has been used commonly as a substitute.)

Some difficult laboratory experiences require direct human presence because there are no simulations for many of the complex machines that are used. Some mitigation have been to use local experts to oversee some of this work, but this involves a lot of bureaucratic work to ensure the quality of the learning and assessment.


6. Methods for Formulating the Right Mix of Assessments

Again, the subject matter experts (SMEs) have wide discretion in formulating the right mix of assessments for learners. Some basic principles follow:

  • Offer a variety of assessments to accommodate those with various learning styles
  • Begin with low-value assessments to help learners acclimate to the technologies and the learning domain
  • Spell out all expectations for all assessments
  • Offer work examples where helpful for divergent types of assessments (Make sure to have student permissions before posting their work.)
  • Make sure that there is sufficient time for learners to prepare for high-value assessments
  • Make sure to have accessibility mitigation for those who have special needs (Brown, 2002, pp. 61 – 68).

Do a walk-through of a course to make sure it all makes sense. Keep channels of communications open with learners, so that mistakes may be corrected, and changes may be made to enhance the value of the assessments for both formative and summative purposes.

Examples

How To

Possible Pitfalls

The learning in a domain field changes quickly. It helps to stay atop a field and make sure that the assessments are fresh and applicable to what is actually being taught and what is actually being required in the domain field.

Module Post-Test

1. What are the basic types of assessments available? How are various types of assessments arrived at in a particular domain field?

2. What is the difference between low-value and high-value assessments? Formative and summative assessments? When should each be used?

3. What are some different methods of online assessment? What are some of the different pedagogical rationales behind the usages of each? Why do assessments have to be measureable?

4. What are some common types of online assessments that are technologically feasible in learning / course management systems (L/CMSes) and in digital learning object repositories? What are automated vs. manual assessments?

5. What are the assessment types which are difficult or currently “impossible” to create online? How are these mitigated?

6. What are some methods to find the right mix of assessments for an online course or learning sequence?

References

Angelo, T.A. & K.P. Cross. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. San Francisco: Jossey-Bass Publishers.

Brown, J.M. (2002). “Enhancing on-line learning for individuals with disabilities.” Assessment strategies for the on-line class: From theory to practice. New Directions for Teaching and Learning. M.D. Svinicki & R.E. Rice, Eds. No. 91. Jossey-Bass.

Nicolay, J.A. (2002). Group assessment in the on-line learning environment. Assessment strategies for the on-line class: From theory to practice. New Directions for Teaching and Learning. M.D. Svinicki & R.E. Rice, Eds. No. 91. Jossey-Bass.

Extra Resources

Anderson, R.S., Bauer, J.F., & Speck, B.W. (2002, Fall). Assessment strategies for the on-line class: From theory to practice. New Directions for Teaching and Learning. M.D. Svinicki & R.E. Rice, Eds. No. 91. Jossey-Bass.