The Alternative Implementation Problem

🌈 Abstract

The article discusses the challenges of building alternative implementations of programming languages and software systems, and the advantages of joining the canonical implementation instead of competing against it.

🙋 Q&A

[01] The Alternative Implementation Problem

1. What are the key insights the author has gained from their experience with alternative implementations of programming languages?

• The author has observed that positioning a project as an alternative implementation of something is often a losing proposition, as the alternative implementation is subject to the whims of the canonical implementation and has limited control over the direction of the project.
• The canonical implementation keeps evolving and adding new features, while the alternative implementation is forced to try to keep up, often playing "catch-up" and living in the shadow of the canonical implementation.
• The author has found that it is often better to join the canonical implementation rather than trying to compete against it.

2. What are the examples the author provides to illustrate this dynamic?

• The author discusses the challenges faced by the PyPy project, which developed an advanced JIT compiler for Python but struggled to keep up with the rapid evolution of CPython.
• The author also discusses the LuaJIT project, which offered great performance gains over the default Lua implementation but faced challenges due to Lua's continued addition of new features.
• The author's own experience with building YJIT, a Ruby JIT compiler, within the CRuby codebase rather than as an alternative implementation, has been more successful.

3. What is the author's advice for younger programmers considering creating their own programming language? The author advises against creating a subset or superficially similar language to an existing one, as this can lead to confusion and the new language being viewed as "almost-but-not-quite" the original. Instead, the author recommends doing "your own thing" and evolving the language at your own pace, without being chained by the expectations of matching the performance, feature set, or library ecosystem of another implementation.

[02] Caveats and Broader Perspectives

1. What are the exceptions to the author's main argument, where alternative implementations can be successful? The author notes that the advice does not apply in situations where there is an open standard, such as implementing a JSON parser, or in the case of multiple browser-based implementations of JavaScript, where the language specification is governed by an external standards body that understands the importance of JIT-compiled implementations.

2. How does the author's perspective relate to the broader industry trends of prioritizing innovation over stability? The author acknowledges that the industry often prizes innovation over stability, leading to a constant evolution of programming languages and APIs, which can be challenging for alternative implementations to keep up with. The author laments this state of the industry, noting that the latest features in languages like Python and Ruby may not always provide "killer" improvements over older versions.

3. What historical example does the author provide to illustrate the challenges of alternative implementations competing with a canonical one? The author discusses the experience of the Lotus 1-2-3 team for Windows, which had to constantly add the same features as Excel in order to maintain compatibility, ultimately losing market share to the Microsoft spreadsheet.