Inspired by DevOps: how Heriot-Watt University is automating teaching and assessment of Computer Science

Faced with increased student numbers and remote learning, the Computer Science department needed to rethink its methods

The last few years have seen big changes in both higher education and software development, characterised by collaboration between remote groups and underpinned by automation.

Teaching was moving online even before the pandemic, in part because many universities have expanded their global reach, opening campuses around the world. When students cannot all pack into the same lecture theatre new methods are required.

Meanwhile, the advent of enterprise open source and cloud-based development platforms means that software production has become a collaborative effort between developers all around the world.

Rob Stewart, assistant professor of computer science at Edinburgh's Heriot-Watt University, and his colleague assistant professor and computer science lecturer Manuel Maarek have been looking at how best to prepare their students for the changing world of work while at the same time seeking to improve the way computer science and related topics can be taught and marked remotely.

Scaling up and scaling out

Heriot-Watt has expanded to include campuses in the Middle East and Asia as well as in the UK. At the same time, the popularity of IT-related courses has increased.

We've become a global university, so we've had to ask how do we scale our marking accordingly? - Rob Stewart, Heriot-Watt

Stewart, Maarek and their colleagues saw a need to move away from ad hoc marking and assessment in favour of online lab-helper interaction, increased collaboration to speed up the coding assignment marking cycle, and automation of course deployment - with the added aim of better preparing students for the world of software development, which is moving very much in the same direction.

"The growth in popularity of computer science courses has exploded with things like cyber security, AI and data science," said Stewart. "At the same time, we've become a global university, so we've had to ask how do we scale our marking accordingly?"

Their efforts to meet the challenge were inspired by open-source software development, where developers from different groups critique and augment each other's efforts, and where code is run through testing regimes on each iteration, and where manual tasks are automated where possible.

"We've absorbed that into our teaching, and we're reflecting that back to produce students that are ready for the industrial practice of open source," Stewart explained.

Peer testing

Software testing is hugely important to industry, but often undervalued by educators. Maarek and his colleagues wanted to help students develop an appreciation of the value of testing, at the same time speeding up the feedback cycle without overburdening the tutors and lab helpers.

"We decided to focus the teaching of computing on test-driven development so that students would experience the evaluation of software as well as coding," Maarek explained. "That was the starting point for us to try to rethink how we teach. We wanted to focus on the feedback, to improve the feedback that students receive on their programming."

Feedback is a dish best served fresh. Rapid feedback encourages critical thinking and active learning, and it can also spark discussion and further collaboration.

The Computer Science department has created a peer-testing learning activity framework and online platform that allows second and third-year CS students to test each other's code and provide immediate and personal feedback. So far it has proved both popular and enlightening, said Maarek.

"We want the students to have experience in actually applying those tests, not just on their own code - it usually passes when they write a test for their own code - but also on their peers' code and then to start a conversation about the results of the test."

We decided to focus the teaching of computing on test-driven development - Manuel Maarek, Heriot-Watt

Peer testing has definitely lifted the quality of students' work, said Stewart.

"[Previously] once they had finished their assignment, they would leave the room and they would never think about it again, but knowing that one of their peers was going to see the code that they've written, they actually spend an extra 15 to 20 minutes improving the quality and readability of the code and the documentation and they take a bit more pride in the work."

But the platform is not just about enabling peer testing. The functionality has been expanded to meet the other objectives, automated course rollouts, faster marking and interactive online assistance, too.

The platform

Originally developed in Python and Django, the platform was ported to DevOps platform GitLab in 2018. GitLab was chosen over alternatives such as GitHub because it is open source, in keeping with the university's ethos, and because it can be self-hosted.

The system lives on the University's own web server, permitting customisation, tweaking and bespoke access and security controls. "Self-hosting has been a big win for us from a user-experience perspective," said Stewart.

The system was first trialled in one course in 2018 and has since been rolled out to ten more. The Haskell libraries developed by the university to work with the GitLab API are open-sourced under the BSD-3-Clause licence.

The platform allows lecturers a better overview of how individual students are progressing, and identify common misunderstandings and areas where learners have problems.

The built-in automation tools also meet another key requirement - scaling up the testing of coding assignments. Continuous integration pipelines run unit tests against student code as soon as they push it, providing instant feedback to students and tutors, which allows interventions to be both timely and (despite the 'automation' tag) highly personalised.

"We have dramatically reduced the time it takes us to mark student code," Stewart said. "If you've got 300 students in a class you need an efficient way to mark, and the continuous integration aspect of GitLab allows us to see whether or not all the tests pass. Although CI is generally used for DevOps, we've actually used it for reducing our marking times and making that process efficient at scale."

Pandemic-proofing

The platform has had another unanticipated benefit.

"We never had pandemics in mind when we designed the system," quipped Stewart. "We were just looking to implement a feedback loop that allows students and teaching staff to discuss the quality of code. However, what we've developed inadvertently is a pandemic-proof system for allowing students to continue their learning in a remote fashion, and for us to provide effective feedback to students."

Considering all the other changes occurring in education and software, together with a largely positive response from students, Stewart is confident there will be no return to the manual way of doing things.

"We won't be going back, it's really been highly effective."