Even Hadoop couldn't predict this general election, says Hadoop co-founder Doug Cutting

However, using big data methodologies, future elections could be forecast this way, he says

This general election is the hardest to call for many generations, with the two main parties polling neck and neck and some of their core support leaching away to smaller parties.

Even statistician Nate Silver, who famously predicted the results of the 2015 US election in every state, has proclaimed the general election to be "incredibly messy".

Were Silver able to call on advanced big data technologies such as Hadoop, would the results be easier to call? Probably not, says Hadoop's co-creator Doug Cutting.

"I think Nate Silver uses Excel," he says. "At least he was during the US election."

Cutting continues: "Excel's powerful enough for the amount of data he has. He's not doing direct surveying of the voters. In the last US election he was combining the results of other polls, doing a kind of meta-poll. Maybe there were 50 or 100 polls but it would be not that many data points."

Predicting the results of an election in this way is not a "big data" problem, at least not using Silver's methodology.

"It's definitely a data problem, but the size of the data is not that big. Even if you think about the way the pollsters work, they don't poll that many people."

"Silver used very clever statistics to combine those and do a better job than anyone else."

Hadoop is not really designed for this sort of analysis, he explains, and would not help Silver to refine his results.

"Hadoop is designed to be a general purpose platform that can support arbitrary data computations in a way that's scalable and affordable. It can support a wide variety of file formats, file systems, security parameters ... it's got a wide library of tools [statisticians] could use but [for Silver] it's probably overkill."

That said, using a different methodology, such as if all early returns were made available for analysis, something like Hadoop could be use to crunch the data and produce a rapid result, he said. Alternatively, pollsters might want to turn to more innovative ways to make predictions.

"There are new ways that people are doing things that are like polling. Skybox takes pictures of parking lots outside of retailers and can predict their earnings. That's definitely data intensive and compute intensive to do the image analysis - I suppose you could even try to identify the makes of the cars and come up with the income profiles of the customers."

However what could be even more useful in this election is perhaps a tool to help whichever of the main parties wins the most seats to choose the right coalition partner...