GitHub Copilot makes LeetCode interviews completely pointless.

I spent some time this week playing with GitHub Copilot. GitHub Copilot is “your AI pair programmer”. It provides autocomplete suggestions for your code as you’re writing it. It’s built on OpenAI’s GPT-3, a very impressive language model.

It has improved my productivity significantly. I used it while writing a client library for the ListenBrainz API. The ease with which it understands what you want to do and does it for you is impressive. Writing boilerplate code is a thing of the past now.

Here’s an example: the ListenBrainz API returns playlist objects in JSPF format as JSON. I wanted to parse the JSON and create wrapper classes for playlists and tracks. I put the JSON in a comment and wrote “class Track” and “class Playlist” in VSCode. Copilot did the rest. It understood the structure of the JSON from the docstring and then created a wrapper class out of thin air. There were a few things I would have done differently, but it was perfectly reasonable code. I won’t include the code in this post for brevity, but here’s a gist if you want to see what it came up with.

I wanted to see if it would be able to write more complicated code. I gave it some LeetCode problems, starting with the classic “Invert a Binary tree”. It suggested a correct solution, also noticeably more succinct than what I would have written.

def invertTree(self, root):
    """
    :type root: TreeNode
    :rtype: TreeNode
    """
    if root:
        root.left, root.right = invertTree(root.right), invertTree(root.left)
    return root

It was also able to solve more complicated problems for topics like linked lists (problem, solution) and dynamic programming (problem, solution).

This makes me think about the point of LeetCode interviews. They were already non-ideal because of the low probability that most engineers would ever solve these problems in real life and GitHub Copilot makes them even more pointless. Any engineer with the right tooling can now solve these problems. What’s the point of asking people to do stuff that they can just get an AI to do? 

The caveat is that you shouldn’t trust code that someone else (especially an AI) wrote and you don’t understand. But it shouldn’t be hard to understand the solution once it’s written in front of you.

There is also the argument that LeetCode interviews aren’t about engineering or algorithmic knowledge, they’re more about how much time the candidate is willing to dedicate to the company before getting selected. The interviews will favor the people who want it the most. If that’s the case, then LeetCode interviews are evergreen. But any organization trying to optimize for engineering skill should get rid of them.

(Thanks to Matt Stuchlik, Rohit Dandamudi and Mukesh Kharita for reviewing drafts of this post.)