Evaluating a Static Online Course for Quality

From E-Learning Faculty Modules


Contents

Module Summary

Evaluators of online courses often deal with a pre-packaged online course that they are supposed to evaluate for learning value. While a “static” online course without online learners, without direct instructor presence, without assignments, without feedback to the assignments, without assessments…may be informative in its own way, it is important to approach a packaged course with clear awareness that much is missing. This module discusses some ways to effectively assess a static online course for quality.

Takeaways

Learners will...

  • review what a “static” online course is and some of its basic components
  • consider why assessing a static online course is limited and in what ways
  • consider basic inferences required to assess a static online course
  • brainstorm questions that may be asked of the faculty member(s) teaching the course
  • consider what additional quality assessments may be done of a live and ongoing course as well as a finalized recorded course

Module Pretest

1. What is a “static” online course? What are some basic components of an online course?

2. Why is assessing a static online course limited? In what ways is assessing a static online course limited?

3. What are some basic inferences required to assess a static online course?

4. What are some questions that may be asked of the faculty member(s) teaching the course in order to assess it?

5. What additional quality assessments may be done of a live and ongoing course? What are additional quality assessments that may be done of a finalized recorded course?

Main Contents

GeneralizationOnlineLearningSequence.jpg

1. What is a “static” online course? What are some basic components of an online course?


A “static” online course involves the basic components of the course as hosted on the learning management system (LMS). These elements include the syllabus, the learning contents, the assignments, the assessments, the discussion boards, the scheduled events, and other elements. Micro-factors include the settings on the LMS. For example, which enablements are available (vs. those which are locked down)? Are students able to start their own discussion threads, for example? Are they able to post digital files or objects in their discussion board threads? Are they allowed to select their own groups or not? Also, it is important to see what the activated components are in an online courses (these would be the features in the left menu bar in the Canvas LMS). What features are activated for use, and how are they harnessed for use? The settings may include how group members are assigned—manually or randomly or otherwise.

The basic components of an online course also include third-party applications (such as by third-party content providers, mapping applications, virtual labs, and others) and social media platforms (social video sharing platforms, social image sharing platforms, and others) as integrations.


2. Why is assessing a static online course limited? In what ways is assessing a static online course limited?


A static course is just the digital residua of the course. Using that information to assess an online course’s quality is limited for several reasons.

  • There are no animating instructor and graduate teaching assistants present.
  • There are no learners.
  • There are no assignments or messages being exchanged.

Looking at the design only indicates something about available information. It may be possible to make some simple inferences about intentionality.

Also, the inert contents of a course are only a part of the available materials for a course. During the teaching, an instructor will bring other contents into play…and will use teaching methods to enliven and enrich the learning.

From just the static course, it is hard to see how the course works in reality and in time. With live instructors and learners, it is easier to see the alchemy about how the various elements interact.

To get past some of these limitations, it is helpful to do a walk-through of the course…from the view of the learner…and to go through the videos and the simulations and other experiences…to see what the experiential sequence is like and what the cumulative learning will be (along the way, and at the end of the course).


3. What are some basic inferences required to assess a static online course?


To use a static course to evaluate the quality, there are some basic inferences made. They include the following:

  • What is promised in the syllabus will be carried on as advertised.
  • All the learning objectives mentioned will be taught to.
  • All the learning outcomes mentioned will be followed-through on by the instructor, and the main work will be assessed (fairly).
  • Anything mentioned in a rubric or an assignment will be assessed.
  • All feedback indicated will be provided in a timely manner and as advertised.
  • Student sociality will be somewhat responded to and somewhat observed for crediting.
  • Formal and informal online resources given will be referred to (formally or informally) at some point in the learning.

These inferences may be true or false, but it helps to surface assumptions in case these are not accurate ones to hold.


4. What are some questions that may be asked of the faculty member(s) teaching the course in order to assess it?


Instructors of online courses should be asked if the assumptions in #3 are true.

Online instructors should also be asked the following:

  • What teaching methods do you use?
  • What are your expectations of your learners?
  • How many hours of work should a learner invest in order to do well in your course?
  • How would you describe the various sub-populations of learners in your main population of learners? How do you work to meet their needs?
  • How do you customize learning for your learners?
  • What accessibility mitigations do you have for this course? What parts of your online course are inaccessible to a learner with various combinations of disabilities?
  • Do you invite guest speakers to your online learning course? Do you have live events? If so, what are these?

There are other questions that would be relevant as well.


5. What additional quality assessments may be done of a live and ongoing course? What are additional quality assessments that may be done of a finalized recorded course?


In a live an ongoing course, it is possible to assess the following:

  • How clearly students understand the course organization and how well they are oriented to it
  • How learners regard their instructor and / or GTAs
  • How student interrelate with each other
  • The general tone or mood of the course
  • What ideas and practices learners understand easily and those that they don’t
  • The assignments that learners are benefitting from vs. those that they are not (generally speaking)
  • The videos learners actually watch vs. those that they don’t (and actually where learners stop watching)
  • How much cognitive scaffolding is used by the learners
  • How high achieving learners engage with the curricular materials, and the requests they make of the instructor and his / her peers
  • The accuracy of the syllabus in practice
  • The follow-through of the instructor in practice
  • How cohesive the course seems to be
  • How well third-party learning objects and videos fit with the learning sequence and experiences

And others… If the evaluator is a participant in the course, that experiences provides other insights as well, in addition to the observations from mere lurking. (In one sense, the observations here are more granular.)

Beyond observing in a live course, it is possible to analyze a finalized recorded course. This means that a course evaluator has access to the built course…and then the residual messages and interactions and other contents created during the learning. Such contents are complete…but there is a dynamism that is lacking in a final packaged sort of online course.

Examples

This module does not have any public examples.

How To

There are various ways to evaluate static online courses based on different theories and models. The approach is beyond the purview of this short module.

Possible Pitfalls

Evaluating a course from the outside—as a person who is not taking the course directly—involves strengths and weaknesses. Some strengths include the objectivity of engaging an issue from the outside, but some weaknesses include the uses of inferences to understand what may be going on with a course. Also, different online instructors will take on very different approaches to teaching the same course, and evaluating a static course does not offer much insight about those skills (especially in the context of an instructor inheriting a course designed by others besides him or her). As with any sort of analysis or evaluation, it is important to understand assumptions going in as well as the various strengths and weaknesses of the approach.

Module Post-Test

1. What is a “static” online course? What are some basic components of an online course?

2. Why is assessing a static online course limited? In what ways is assessing a static online course limited?

3. What are some basic inferences required to assess a static online course?

4. What are some questions that may be asked of the faculty member(s) teaching the course in order to assess it?

5. What additional quality assessments may be done of a live and ongoing course? What are additional quality assessments that may be done of a finalized recorded course?

References

Extra Resources