Khan Engineering

Khan Engineering

We're the engineers behind Khan Academy. We're building a free, world-class education for anyone, anywhere.


Latest posts

Making Websites Work with Windows High Contrast Mode

Diedra Rater on March 21

Kotlin for Python developers

Aasmund Eldhuset on Nov 29, 2018

Using static analysis in Python, JavaScript and more to make your system safer

Kevin Dangoor on Jul 26, 2018

Kotlin on the server at Khan Academy

Colin Fuller on Jun 28, 2018

The Original Serverless Architecture is Still Here

Kevin Dangoor on May 31, 2018

What do software architects at Khan Academy do?

Kevin Dangoor on May 14, 2018

New data pipeline management platform at Khan Academy

Ragini Gupta on Apr 30, 2018

Untangling our Python Code

Carter J. Bastian on Apr 16, 2018

Slicker: A Tool for Moving Things in Python

Ben Kraft on Apr 2, 2018

The Great Python Refactor of 2017 And Also 2018

Craig Silverstein on Mar 19, 2018

Working Remotely

Scott Grant on Oct 2, 2017

Tips for giving your first code reviews

Hannah Blumberg on Sep 18, 2017

Let's Reduce! A Gentle Introduction to Javascript's Reduce Method

Josh Comeau on Jul 10, 2017

Creating Query Components with Apollo

Brian Genisio on Jun 12, 2017

Migrating to a Mobile Monorepo for React Native

Jared Forsyth on May 29, 2017

Memcached-Backed Content Infrastructure

Ben Kraft on May 15, 2017

Profiling App Engine Memcached

Ben Kraft on May 1, 2017

App Engine Flex Language Shootout

Amos Latteier on Apr 17, 2017

What's New in OSS at Khan Academy

Brian Genisio on Apr 3, 2017

Automating App Store Screenshots

Bryan Clark on Mar 27, 2017

It's Okay to Break Things: Reflections on Khan Academy's Healthy Hackathon

Kimerie Green on Mar 6, 2017

Interning at Khan Academy: from student to intern

Shadaj Laddad on Dec 12, 2016

Prototyping with Framer

Nick Breen on Oct 3, 2016

Evolving our content infrastructure

William Chargin on Sep 19, 2016

Building a Really, Really Small Android App

Charlie Marsh on Aug 22, 2016

A Case for Time Tracking: Data Driven Time-Management

Oliver Northwood on Aug 8, 2016

Time Management at Khan Academy

Several Authors on Jul 25, 2016

Hackathons Can Be Healthy

Tom Yedwab on Jul 11, 2016

Ensuring transaction-safety in Google App Engine

Craig Silverstein on Jun 27, 2016

The User Write Lock: an Alternative to Transactions for Google App Engine

Craig Silverstein on Jun 20, 2016

Khan Academy's Engineering Principles

Ben Kamens on Jun 6, 2016

Minimizing the length of regular expressions, in practice

Craig Silverstein on May 23, 2016

Introducing SwiftTweaks

Bryan Clark on May 9, 2016

The Autonomous Dumbledore

Evy Kassirer on Apr 25, 2016

Engineering career development at Khan Academy

Ben Eater on Apr 11, 2016

Inline CSS at Khan Academy: Aphrodite

Jamie Wong on Mar 29, 2016

Starting Android at Khan Academy

Ben Komalo on Feb 29, 2016

Automating Highly Similar Translations

Kevin Barabash on Feb 15, 2016

The weekly snippet-server: open-sourced

Craig Silverstein on Feb 1, 2016

Stories from our latest intern class

2015 Interns on Dec 21, 2015

Kanbanning the LearnStorm Dev Process

Kevin Dangoor on Dec 7, 2015

Forgo JS packaging? Not so fast

Craig Silverstein on Nov 23, 2015

Switching to Slack

Benjamin Pollack on Nov 9, 2015

Receiving feedback as an intern at Khan Academy

David Wang on Oct 26, 2015

Schrödinger's deploys no more: how we update translations

Chelsea Voss on Oct 12, 2015

i18nize-templates: Internationalization After the Fact

Craig Silverstein on Sep 28, 2015

Making thumbnails fast

William Chargin on Sep 14, 2015

Copy-pasting more than just text

Sam Lau on Aug 31, 2015

No cheating allowed!!

Phillip Lemons on Aug 17, 2015

Fun with slope fields, css and react

Marcos Ojeda on Aug 5, 2015

Khan Academy: a new employee's primer

Riley Shaw on Jul 20, 2015

How wooden puzzles can destroy dev teams

John Sullivan on Jul 6, 2015

Babel in Khan Academy's i18n Toolchain

Kevin Barabash on Jun 22, 2015

tota11y - an accessibility visualization toolkit

Jordan Scales on Jun 8, 2015


App Engine Flex Language Shootout

by Amos Latteier on Apr 17, 2017

This is the second time I've been to Silicon Valley. Some years ago - never mind how long precisely - I started writing open source software. It's taken me to some strange places like Washington DC and Kwajalein, but never until recently to the suburbs of San Jose.

The reason I'm here is for a hackathon. I recently was hired by Khan Academy. I live in Vancouver, the one in Canada. I work remotely. In fact most of the people I work closely with are remote. This is one of the great things about this new job. Also it turns out that I get to work with smart and friendly people. And instead of writing a CRM for insurance salespeople like I was doing at my last gig, my job is to help anyone anywhere get a free world-class education. There's got to be a catch. Maybe it's this hackathon.

What kind of hackathon is this anyway?

Apparently it's healthy. Growing up my mother was very focused on healthy foods. My sister and I used to break into the carob chips, looking for anything candy-like. So if it's anything like that I know that I'm in trouble.

The offices are covered in decorations when I arrive. This seems less like a hackathon and more like a craft party. I feel a bit out of place, so I cut a cape out of felt and put it on. That's better.

Rather than endurance coding, we do a lot of socializing and collecting donations for a food bank interspersed with project work. I decide to work with someone I don't know on a fun-sounding project.

A fun-sounding project

The co-worker I don't know explains the project to me. Right now Khan Academy runs on App Engine. Classic, not flex. There's interest in moving to flex. It turns out that they are pretty different. There's a ton of institutional knowledge here about classic built up over years. But we don't have a lot of experience with flex. This project is about getting some experience with flex.

Plus the project is a language shootout. Right now most of the backend code is in Python, but there are fantasies about moving to another language. Who doesn't fantasize about other languages? I'm currently infatuated with Elixir. It's possible that I like it because I haven't used it for any serious projects yet. But I have tried writing distributed systems (and even worse, debugging distributed systems written by others) in Python, and that was a bad enough experience to make me look for alternatives.

Anyway we're probably not going to switch languages anytime soon, but it's fun to dream.

The task we set ourselves is to consume a pub/sub feed and put the results into BigQuery. The feed collects information about about exercises attempted by users of Khan Academy. There is a lot of exercise data.

We must first validate the message using a hmac digest. We use a secret stored in metadata. Next parse the pub/sub message. Then send it to BigQuery. We have to insure that the right table exists and then insert the data.

Oh, and we need to run a web server, cause that's how you do push subscriptions.

So it's not too much work, but not completely trivial.

The contenders

Of course I'm going to write a service in Elixir. In fact that's the reason I chose this project.


Elixir is a new marketing campaign for Erlang. Well technically it's more than that, but from my point of view the great things about Elixir are OTP and BEAM. My coworker already has the Elixir implementation started. I only have to add a few things to get it working, but there are no official Google Cloud libraries so that ends up being my biggest challenge.


Go is a language that not many people who I know seem to be enthusiastic about. But hey, why not try it? There are bound to be decent Google Cloud Platform library for it, right? And it compiles fast.


Kotlin is a language that I haven't heard of until recently. But my manager loves it, so I guess I should test it. Actually it doesn't seem to be as dreadful as I expected. Plus some people say that they've gotten the JVM working pretty well these days. One downside is that it doesn't compile as fast as go does.


Python is the language I've hitched my professional career to. Happily for me Python isn't dead yet. Khan Academy uses a lot of Python. Yeah, it's slow. Also under flex you have to choose your own WSGI server. I'm not sure we chose the best one. If we had more time I'd like to look into this more.


Crystal is a language I've never heard of. But my coworker checks in a microservice written in one night. He says it's fun.

Interlude in which my mind is blown

We're making progress on the project, mostly due to my co-worker's efforts. I'm spending most of my time trying to understand the code he's checking in. But then comes the Python Bee. It's like a spelling bee, but in Python. That sounds cool; I know Python. So it turns out there's a wrinkle. You have to speak out your program, character by character without looking at the screen while someone else types it in for you. OK, so you have to keep it all in your head and not mess up on indenting. I think I can do this. Oh wait, it turns out that you have a partner, and you can't communicate with them, and you each take turns saying the next character of the program! So it's not just keeping the program in your head, it's guessing what program is in your partner's head too.

Two people are writing a prime number sieve in Haskell using mind reading. This is happening right in front of me right now.

Now it's my turn. Ok, ok, ok, ok. IndentationError. Ugh, not only does this look really hard, it is really hard too.

Some benchmarking results

We don't get around to doing message validation on all the different microservices. So our benchmark isn't fair. In fact it's not even particularly precise. We just start the microservices, point the pub/sub firehose at them and see how far behind they fall during the day, and how long it takes them to catch up with the backlog in the evening.

Still it isn't hard to see how each language fares.

Language Speed
Kotlin Fast
Elixir Not quite as fast as Kotlin
Go Pretty much as fast a Elixir
Python Quite a bit slower

We can't get Crystal running well enough to compete. We run into ssl errors and don't have enough time to track down a solution.

After doing this sloppy benchmark I'm not sure that it's really a great argument to switch away from Python. Speed is only one consideration among many.

In practice it seems that the quality of Google Cloud Platform libraries is probably one of the most important factors in picking a language for our benchmark, since we rely so heavily on Google Cloud Platform.

I am impressed with Elixir's performance, though. It almost kept up with Kotlin, and hey confirmation bias.

I also wonder about whether we should be consuming pub/sub messages with a push queue. I suspect that performance would be much better using a pull queue, since then we could batch our BigQuery inserts. Often the best way to improve performance is changing the algorithm not the language.

Anyway we present our results. People nod respectfully. Ours is just one in a vast and riotous sea of hackathon projects.

We haven't figured out the organization's future language strategy, but we have gotten some practical experience with App engine flex. Plus we've validated that microservices seem to work OK for a simple task that's bounded by external APIs.


I'm still really in awe of the Python Bee performances I saw. I asked the Haskellers how they did it. Foldl I was told. Foldl makes sense for this kind of thing. If you are both thinking "Foldl, foldl, foldl" then it's easier to read each other's minds. I'm probably not going to learn Haskell, but I know about reduce. Next year I just need to find a partner who's also thinking "Reduce, reduce, reduce". That and remember how many spaces we're indented.