I've been watching GitHub Copilot since it went live on June 21, 2022, and six months of paid adoption has given us some real data on how developers use it and what its limitations are
When it comes to generating boilerplate code from context, completing well-understood patterns, writing docstrings and comments from function signatures, and translating code between languages, Copilot is incredibly strong, saving developers a substantial amount of time, for example, a task that would take 10 minutes to write boilerplate can be completed in 2 minutes with Copilot
In our 200‑person backend team we hooked Copilot into the VS Code instances that run on our internal dev boxes. The first thing we measured was the round‑trip time from keystroke to suggestion; on a typical 16 GB machine it hovered around 150 ms, which is fine for interactive use but becomes noticeable when you fire off dozens of completions in a tight loop. To get a sense of productivity impact we logged the time spent on creating new REST handlers in Go. Before Copilot the average developer spent about 12 minutes per handler; after enabling it the median dropped to 4 minutes. The variance widened because the tool sometimes produced a full handler that still needed a manual audit of error handling and logging conventions.
On the other hand, Copilot struggles with novel algorithm design, complex business logic with multiple interacting conditions, security-sensitive code where correctness is non-negotiable, and code that requires understanding of your specific codebase's patterns and naming conventions, in these cases, the suggestions are often plausible-looking but incorrect
One night at 2 am I was reviewing a pull request that had been auto‑generated from a Copilot suggestion for an upload endpoint. The code used `os.path.join` to concatenate a user‑supplied filename with a base directory, and Copilot had filled in a simple `if not filename.startswith('/'):` check that looked reasonable. Our static analysis pipeline, which runs Bandit and CodeQL, flagged a potential path‑traversal because the check didn’t account for `../` segments. We rolled back the change, added a proper `secure_filename` call from Werkzeug, and added a rule to reject any Copilot suggestion that manipulates file paths without sanitization. That incident cost us a few hours of debugging but reinforced the need for a gate.
Research published in 2022 showed that Copilot can suggest insecure code in certain scenarios, specifically, for code involving security-sensitive patterns, Copilot can suggest code that is vulnerable to injection, path traversal, or uses deprecated cryptographic functions
Developers using Copilot for security-sensitive code should apply the same scrutiny they would apply to any untrusted source, this is not a reason to avoid Copilot, but rather to use it with a critical eye
From a code‑review standpoint the signal‑to‑noise ratio improved. Reviewers spent roughly 30 % less time on style and formatting because Copilot tends to follow the project's `.editorconfig` and lint rules out of the box. However, the mental load shifted to verifying that the suggested algorithm matches the business intent. In one sprint we saw the number of review comments per PR drop from an average of 7 to 4, but the remaining comments were almost all about edge‑case handling, which is exactly where Copilot is weakest.
After six months of production use, I think the most accurate characterisation of Copilot is that it is a very good autocomplete that sometimes suggests complete function implementations, it reduces the time from thinking about what to write to having a draft implementation
Copilot shifts the developer's cognitive work from writing to reviewing and adjusting, most developers who use it consistently report that they would not want to code without it, which says a lot about its usefulness
The key to getting the most out of Copilot is to understand where it excels and where it struggles, by doing so, developers can use it to speed up their workflow and focus on the more complex and creative aspects of coding