Assessing self-paced, online courses for use with your students using an evidence-based checklist

Photo by Karolina Grabowska on Pexels.com

The issue 

The proliferation of self-paced, online learning platforms in recent years has provided a wealth of options for adult learners looking for quick upskilling and refresher courses and for teachers and trainers looking for support resources for students. These options range from popular curated services like LinkedIn Learning by Microsoft, used by an estimated 27 million adults (Trent-Gurbuz, 2020), to platforms that are little more than eBay for online courses such as Udemy (Viktor, 2021).  

The problem with these offerings is the variability of the quality of the course design and whether or not these courses result in the sort of authentic engagement that fosters long-term learning. 

Authentic engagement means that students are focused, interested and motivated in such a way that they can apply what they’ve learned to new contexts. (University of Waterloo, 2021) 

Even courses offered by providers who curate their content and thus have some sort of quality control, there are many courses that  consist of little more than well-produced videos with multiple choice and true/false quizzes that test short term knowledge recall. This may be fit for purpose when it comes to lower-order recall and simple procedural work as described in the unistructional stage of Bigg’s SOLO taxonomy (Biggs, N/A). However, for more complex topics and concepts, as the learning in these courses is self-paced with no direct facilitator or peer to peer interactions, the burden of creating authentic engagement even over a short time needs to be driven by the design of the user interface, content and activities. 

Checklist for assessing resources 

Given the wealth and variability of content and their desire to provide quality, fit for purpose resources for their students, how can education professionals effectively evaluate the quality of these resources using an evidence-based approach? (Hannafin, 1997) 

As with all learning resources, we start with our students and the context. Who needs to learn what? What is being taught? What are the relevant student demographics, available technologies for access and creation, the estimated time investment/commitment and the desired learning outcomes? (University of New England, 2021). 

In this context, we are talking about self-paced learning for adult learners. Adult learners are thought to be largely self-directed, intrinsically motivated learners who are problem-centred and interested in the immediate application of knowledge (Merriam, 2001).  However, they are also notoriously time-poor, juggling multiple priorities at work and home, so need learning solutions that will satisfy their learning goals efficiently. Knowing who your learners are and what their context is will help you apply the evaluation points that follow more effectively. 

 Learning objectives stated and met 

Even if all a teacher is after is a video resource to demonstrate a specific software skill, they need to ensure either they or the resource explicitly sets out what learners will be able to do at the end of undertaking learning. This allows the learner as well the teacher to evaluate whether the resource is fit for purpose. 

Educators can support this by writing learning objectives for the third-party course resources they wish to cur and then using the learning objectives as criteria to judge the resource. 

If they do not, educators risk providing irrelevant content that will lead to student frustration and disengagement. If the resource itself does not provide learning objectives, teachers can dig further into the resource itself, seek out another resource or provide a sufficient introduction so that students understand what they will gain from their time investment. 

 A user interface design that reduces cognitive load 

To reduce cognitive load and facilitate access to the relevant content, the user interface should be designed so that it is easy to navigate, access and use with minimal learning and instruction necessary so as not to distract from the content (Centre for Extended Learning – University of Waterloo, 2021). [I think you’re using Harvard style, in which case, the full stop comes after the citation) 

 Content and activity design that is accessible, flexible and culturally appropriate 

When it comes to content and activities, Universal Design for Learning principles that provide multiple ways into content and activities allows for learners to connect with what is comfortable for them.  Elements such as text alternatives for visuals that convey meaning, closed captions for videos and transcripts for audio support a diverse range of abilities and accessibility may be a requirement of your institution. Consideration of language proficiency and cultural diversity will ensure that the design of content and activities supports rather than bars learners from what they need to achieve (CAST, 2018). 

 Sufficient feedback points for self-assessment and self-correction 

In the absence of student-instructor and student-peer interactions, teachers must if and how feedback is provided to help learners assess whether they’ve achieved the learning objectives (Hattie & Timperley, 2007) of the resource or short course made available to them. 

A focused research study focusing on the experiences of adult learners in a self-paced online learning environment bears this out. In the study, learners were in a corporate learning setting undertaking self-paced, online training. The findings were that self-paced, online learning was effective when learners were provided with ample opportunity to self-assess and self-correct. The mechanisms used were quizzes, practice exercise and/or simulations (Dobrovolny, 2006). 

 Resources to enable the application of learning to the student context

Adult learners must be provided with the opportunity to apply their learning and connect it to their context in order for the learning to take hold (Horton, 2011). 

Providing them with the tools and resources to do so outside the course will allow them to leverage their time investment and experiment with implementing elements of what they’ve learned in their work or learning. 

The aforementioned study of adult learners in a corporate setting (Dobrovolny, 2006) found that in the absence of take-away resources, learners naturally returned to the course and created their own job aids to refresh their learning. 

Allow me to introduce myself

Second Life first days, meeting Jimmy Wales and some of my fav gear from the early 2000s

It was 2005, my late husband and I had just closed up shop on our web design business and I was in need of a job. My background had been in corporate and broadcast video, but in Adelaide – the small city in South Australia I still call home – there were no jobs going.

My background included broadcast video, copywriting, print journalism and corporate video. Writing and project managing website and multimedia development was a smooth transition. But in a small city with limited clients and no capital reserves, our business wasn’t working for us.

I got a temp gig at an educational technology and research government-owned organisation with a mandate to boldly go forth and explore innovations in elearning. After a few months cataloguing whitepapers, I heard that they were looking for someone to set up and produce a national e-learning podcast. I stuck my hand up and got a full time role.

My time at education.au (sadly decommissioned since) drew me into a love of online education I still have today. I got to travel around the country and train and learn from educators in primary, secondary and tertiary education. I fell in love with the potential of virtual worlds for scenario-based, immersive learning and designed learning experiences there. I moved on to elearning design for an RTO and then a university.

Along the way I’ve picked up interests in Universal Design for Learning, Web Content Accessibility, Project Management and a focus on ensuring learning activities in online courses are constructively aligned and designed with the student’s educational goals in mind.

As I write this, the world is still reeling from the impacts of a global pandemic that was made worse by a lack of information literacy, digital literacy and the encroaching darkness of an entrenched cognitive bias of polarisation, brought on by the algorithms of platforms that promised to bring us together and to democratise information. I hope we as a species can help each other through this and be smarter, kinder and wiser as we build a better future. And we can only do that if we keep on learning.

Representation vs Appropriation

I’ve recently been involved in a series of conversations regarding representation in order to promote inclusion and diversity.

The online learning space can often feel and be cold in terms of human contact. Injecting humanity and personality into what we present online is, in my opinion, vital to appealing to the affective realm and helping students to make positive emotional attachments to the online space and each other.

As educators we can ensure we have a profile image and a bio. We can create videos where we at least periodically show our faces. And if we use images in our course materials, where people are part of the imagery – we can and should strive to show a diverse range of humanity.

Some platforms allow for the use of gifs of real people – often actors and comedians, but sometimes just meme-able folks – to use as emotional responses or reactions. And that is where appropriation and a question of “digital blackface” can creep in.

Some US Academics and social commentators argue that a white person using a GIF of a Black person expressing exaggerated reactions harks back to minstrel shows, to times when radio, film and television programs depicted people of colour as emotionally unsophisticated and uneducated and that doing so is offensive. For those Australians who point to this as more of an American problem – we need to no further than the tv series Boney in the 1970s where a white, New Zealand-born actor portrayed an Australian Aboriginal Detective Inspector named Napoleon Bonaparte.

Dr Aaron Nyerges, a Lecturer in American Studies at the US Studies Centre, argues that in using these exaggerated depictions, one is promoting stereotypes that denigrate and debase others.

Others agree, pointing out that using these seemingly fun, harmless GIFs can be seen to perpetuate negative stereotypes about Black people as overly-animated, loud, aggressive, angry and hypersexual.

Alex Chung, CEO of Giphy, an online GIF database, is reportedly working on modifying their filters so that their service isn’t offering up negative stereotypes.

You may notice that on corporate platforms such as Zoom and Yammer, you are now invited to choose the skin colour of your thumbs up icons. This points to an understanding by these providers that appropriate representation matters.

So does this mean you shouldn’t use images at all in your courses rather than risk a minefield? Certainly not. Representation matters. Using representations of diverse people and genders in positive ways or even just in neutral photos in your course materials is a positive step towards racial and gender diversity. Finding them on some stock photo sites can be a challenge – but take some extra time and don’t settle for only using white, male fingers on a computer keyboard or always grabbing the first image you can find of someone in a hardhat rather than taking a few extra minutes to search for a female construction worker or using Bob and Fred in your next text-based scenario because you’re in a hurry.

The marketing people in higher education know that presenting a diverse range of people on our brochures and websites is essential so that students can picture themselves at our institutions. Let’s ensure our students can also picture themselves in their chosen disciplines, not as comic relief but as welcome and honoured professionals.