Four years ago I wrote a slightly inflamatory blog on what still sucks in front-end dev. One of the points was on Bower and how JS library management sucks. Sadly, the tools have changed, but JS library managment still sucks.
We’ve had a few NPM-related trash fires over the years. left-pad is still a meme, and another one happened today? Well it turns out the ecosystem is as fragile as ever. Let’s start by assuming everyone here is acting in good faith (although that’s not necessarily the case, let’s do the whole innocent until proven guilty), apart from whoever wrote the malicious code.
16th September: right9ctrl rewrites the code to remove the dependency on flatmap-stream and pushes out a new version, as a semver-major version bump, which means updates will not automatically be upgraded.
5th October: Someone using the NPM account of user “hugeglass” pushes flatmap-stream 0.1.1 to NPM. Appended to the end of the minimised code that was pushed to NPM was an obfuscated snippet that decrypted some further content — the “payload” that did something nefarious.
29th October: Through sheer luck, the malicious code that bootstraps the malware uses an API that’s been deprecated in Node 10. Jayden Seric notices the warning and raises an issue on one of the packages which brings in flatmap-stream as a transitive dependency.
20th November: Ayrton Sparling identifies that the bug on the Nodemon package is a result of the compromised flatmap-stream package, and raises an issue on event-stream, blaming right9ctrl for introducing the issue and Dominic Tarr for handing over maintenance to another user.
26th November: I’m made aware of the issue through my employer, and initial evaluation reveals the encrypted payload is decrypted based on the presence of a specific value of description in package.json. It looks like a highly targeted attack but no-one is sure what the actual payload is as we do not know what package is targeted. Through a bit of teamwork and communication via the issue, Jacob Burroughs successfully finds the key and decodes the payload.
It turns out it was highly targetted towards a cryptocurrency wallet product to steal encryption keys for cryptocurrency wallets. Along with the rest of the dev community we’ve wiped our brows as we’ve got away with it, we didn’t have malicious code running on our dev machines, our CI servers, or in prod.
Nothing’s stopping this happening again, and it’s terrifying. The Hacker News and GitHub threads on the issue quickly denigrated into a blame game, pulling out license clauses and debating what open source ettiquette means. However, this is no one person’s fault, it’s a systemic failure that has led to this.
The second is in package distribution. Locking package versions did not come until relatively recently (npm-shrinkwrap was usually a sub-optimal experience, but has been replaced through Yarn and NPM package locks), but other facets that have been common in distributing Linux system packages have not been adopted by NPM. The first is code-signing, and the second is reproducible builds.
If we had code-signing, would it have solved the issue? Maybe. It would have flagged to the upstream managers that the dependency has now been published by a different person, which may cause someone to re-review the code. But they may just accept it and move on, after all, who has time to review the code? And if you trust someone for one build, that means you trust them for all future builds too.
If we had reproducible builds, would it have solved this issue? Maybe, as it would have made it easier to compare the source in GitHub to what actually got installed. In this model, NPM would have to build the packages from the committed source code, rather than it being built and published locally, to ensure that the published open source code and built component correspond, to make review easier. But someone would still need to have reviewed the source code that built the dependency.
And for that, I have no answer. But take yet another cautionary tale from this, learn from it what we can and let’s take a small step towards getting better.
I might have mentioned earlier in the post that I’m bringing out a book, something I’ve been working on for the past 3 years. It’s called The Full Stack Developer and it looks at the underlying principles of full-stack web development, rather than focussing on a particular framework. I’d really appreciate it if you could have a look at it and consider pre-ordering if it sounds like the kind of thing you could learn something from? Thanks!