Don’t Judge An LLM Only By The Web App

🌈 Abstract

The article discusses the growing problem of people talking about large language models (LLMs) while only using the web version, not the API, and the free version, not the paid version. It argues that the web version is designed to be more "chatsplainy" and artificially crippled to not cost the company as much money, while the API version is more powerful and expensive to run.

🙋 Q&A

[01] The problem with people talking about LLMs

1. What are the key issues the article discusses regarding how people talk about LLMs?

• People are only using the web version of LLMs, not the API
• People are only using the free version of LLMs, not the paid version
• The web version is more of a demo and is artificially crippled compared to the API version

2. Why does the author argue the web version is designed this way?

• The web version is more "chatsplainy" and friendly to hook users into paying for the API version
• The web version is artificially crippled to not cost the company as much money to run

3. What examples does the article provide to illustrate this point?

• The article discusses how Gemini 1.5 Flash's web version is limited to 32K tokens, while the API version likely supports much higher token limits
• The article cites a Hacker News post that argues ChatGPT and other LLMs are better at shortening text rather than truly summarizing it

4. What does the article suggest about judging LLMs based on the free web version?

• The article argues that people should not judge an LLM's capabilities based solely on the free web version, as the API version is often much more powerful and capable

[02] Comparing web version and API version performance

1. What examples does the article provide to illustrate the differences between the web version and API version performance?

• The article mentions that in the past, the author's experience with GPT-3.5 Turbo API was quite different from the web client version
• The article cites a Reddit comment that states the API version "blows the chat out of the water when used correctly and for specific purposes"

2. What key points does the article make about the limitations of judging LLMs based on the web version?

• The web version is more of a "showroom" or demo version, not the full product
• The web version can perform better in some ways, but often performs worse than the API version
• The API version is more powerful and capable, especially when used for specific, targeted purposes