Google open sources Python-on-Go transcompiler Grumpy - built to supercharge YouTube

Grumpy transcompiles Python into Go for better performance from concurrent workloads

Internet behemoth Google has open sourced Grumpy, the Python runtime environment for Go, which was developed by the company in a bid to improve the performance of YouTube, Google's popular video publication and sharing website.

The company developed the tool because of the difficulty of making concurrent workloads perform well on CPython, which handles the YouTube front-end and is tasked with serving millions of requests per second.

Other Python runtimes weren't able to solve the concurrency issue without introducing unsatisfactory trade-offs, according to Dylan Trotter, a staff software engineer within YouTube Engineering at Google.

"So we asked ourselves a crazy question: What if we were to implement an alternative runtime optimised for real-time serving?" asked Trotter, rhetorically, in a blog posting. "Once we started going down the rabbit hole, Go seemed like an obvious choice of platform since its operational characteristics align well with our use case."

YouTube needed first-class language interoperability and Go's powerful runtime type reflection system made this straightforward, he added.

"Grumpy is an experimental Python runtime for Go. It translates Python code into Go programs, and those ‘transpiled' programs run seamlessly within the Go runtime. We needed to support a large existing Python codebase, so it was important to have a high degree of compatibility with CPython (quirks and all). The goal is for Grumpy to be a drop-in replacement runtime for any pure-Python project," wrote Trotter.

However, two compromises had to be made: first, Grumpy forgoes support for C-language extension modules. This means that Grumpy can't leverage existing Python C extensions, but the benefit in terms of performance justifies that decision, claims Trotter.

"Second, Grumpy is not an interpreter. Grumpy programs are compiled and linked just like any other Go program. The downside is less development and deployment flexibility, but it offers several advantages," wrote Trotter.

These include opportunities for optimisation at compile time via static program analysis. "But the biggest advantage is that interoperability with Go code becomes very powerful and straightforward: Grumpy programs can import Go packages just like Python modules!" he adds.

However, Trotter warns that although Grumpy has been freshly released to open source, it remains alpha-grade software. "Most of the language constructs and many core built-in types work like you'd expect [but] there are still holes to fill — many built-in types are missing methods and attributes, built-in functions are absent and the standard library is virtually empty."