Anybody who's applied to university since the 1990s will be familiar with UCAS, the Universities and Colleges Admission Service that has been fully online since 2006.
UCAS processes around 2.5 million applications to 340 universities and colleges from roughly 650,000 students each year. The busiest periods are on application deadline day in January and A-Level results day in August, when almost 500,000 students are accepted to study in higher education.
With that requirement to scale services to match those peak periods, UCAS adopted the Splunk Enterpise platform to manage data collection from hundreds of thousands of endpoints.
"We operate on a cyclical basis where we have a particular peak and particular point of limelight - we're in the media over A-Level results day. Splunk was introduced to support the improved fault resolution, diagnosis and general insight into the demand on our systems for 2013," Peter Raymond, solution architect at UCAS told Computing.
Raymond explained how Splunk is installed across 40 servers and around 70 log sources, all of which are deployed through Amazon Web Services, with everything forwarded to a Splunk server for indexing.
The "peaky" nature of UCAS operations, Raymond explained, demonstrates the advantages of using cloud-based services.
"UCAS is a great fit for cloud architectures, because they offer us scalability, out-of-the-box opportunities, high availability and resilience across multiple availability zones," he said.
But while Amazon is used to host the servers, it's Splunk that is harnessed to crunch and gather data.
"We construct various queries against that data to tell us things like the number of users we have on the system. Over results day 2013 we were dealing with 180 logins per second - so we have some high-volume systems," said Raymond, adding that on A-Level results day 2013 itself, 386,000 places were confirmed.
UCAS investigated a number of possible solution before selecting Splunk Enterprise, testing the alternatives during summer 2013.
"We went through a process of looking at the players in the market then created a selection of candidates for a proof-of-concept with a weighting process and scoring," Raymond told Computing, adding the simplicity of Splunk played a role in UCAS's selection.
"It's got a good low barrier to entry; you can get going pretty quickly. They've got a flexible pricing model, which is based on the amount of data you send to the server. So it's easy to get going then keep growing as your needs grow," he said.
UCAS has used Splunk for less than half a year but has already seen the benefits over last August's A-Level results day and application deadline day on 15 January.
"We've got great evidence from the IT ops and the application support people that they're resolving faults quicker and they're able to proactively identify issues," Raymond said, speaking on deadline day.
"I've literally just come from a big screen where we've got the number of logged in users for our systems, with any indications of problems, user experience, so it enables us to proactively manage issues and intervene early."
Ultimately, Splunk's platform is allowing the organisation to provide better services to students applying for places at colleges and universities.
"They benefit from UCAS delivering them a reliable system; they benefit from the higher education institutions having access to reliable systems because that's all part of the confirmation and clearing process," said Raymond.
With the move towards mobile, UCAS is seeing logins from multiple devices at the same time, but as Raymond explained, Splunk has had no problems coping with the extra data collection.
"What's interesting is how you can look at user agents and see that people have multiple logins to your system at the same time," he said.
"They've got it on their mobile, their desktop, granny's looking at it, the parents are looking at it, so there's obviously a move towards mobile, but we see multiple devices accessing our systems."
The improved ability to collect data has also allowed UCAS to reuse it in order to predict demand for services, helping provide a better service to students at peak times.
"We have such good insight into what the demand has been on a system that we can then construct test cases for future performance testing, which enables us to make sure we have the right scale for the future," said Raymond.
"We could do that before, but what Splunk gives you is better insight," he concluded.
By eliminating high entry costs for big data analysis, you can convert more raw data into valuable business insight.
A discussion of the "risk perception gap", its implications and how it can be closed