How To Think Real Good | Meta-rationality

🌈 Abstract

The article discusses the author's long-standing interest in thinking about thinking, and how this led to various projects and collaborations around understanding effective and accurate thinking. The main focus is on critiquing the LessWrong rationalist community's emphasis on Bayesian probability as the primary tool for rationality, and arguing that it is only a small part of what is needed. The author proposes that figuring out how to think effectively is a much broader and more complex challenge that requires drawing on a wide range of intellectual tools and methods.

🙋 Q&A

[01] Problem Formulation

1. What are the key points made about problem formulation?

• Before applying any technical method, you need to already have a pretty good idea of what the form of the answer will be.
• A successful problem formulation has to make the distinctions that are used in the problem solution, and make the problem small enough that it's easy to solve.
• All problem formulations are "false" because they abstract away details of reality, but the goal is to find an idealization that is useful for the particular problem at hand.
• Heuristics for good problem formulation include: working through specific examples, iterating between formulation and solution, and solving a simplified version of the problem first.

2. Why is problem formulation so important according to the author? The author argues that finding a good formulation for a problem is often most of the work of solving it. Formal methods like Bayesian probability require a precise specification of the problem, but figuring out that specification is a major challenge in itself that the Bayesian framework does not address.

3. What is the author's view on the relationship between problem formulation and problem solving? The author states that problem formulation and problem solution are mutually-recursive processes - you need to go back and forth between trying to formulate the problem and trying to solve it, as progress in one area informs the other.

[02] Rationality Without Probability

1. What examples does the author provide of solving problems without using probability theory? The author discusses his work on classical planning problems in robotics, where he was able to solve the problem using modal logic and model theory, without any use of probability. He also describes work with Phil Agre on developing a theory of effective action that dealt with uncertainty without representing it probabilistically.

2. What are the author's key points about the limitations of probability theory for rationality? The author argues that probability theory is not the only, or even the best, way to deal with uncertainty. It can collapse together many different sources of uncertainty in unhelpful ways. The author also suggests that Bayesians sometimes inappropriately attribute unconscious probabilistic reasoning as an explanation when the actual reasoning process is unclear.

3. What is the author's view on learning from diverse fields outside one's own? The author emphasizes the importance of learning from fields very different from your own, as each has ways of thinking that can be useful in surprising ways. Applying tools and perspectives from anthropology, psychology, and philosophy was crucial to the work he did with Phil Agre.

[03] Heuristics and Cognitive Styles

1. What heuristics or rules of thumb does the author suggest for effective thinking? Some key heuristics mentioned include:

• If a problem seems too hard, the formulation is probably wrong - go back and observe the real-world situation.
• Learn as many different kinds of math as possible, as it's difficult to predict what will be relevant.
• Get a superficial understanding of many mathematical fields, so you can recognize when they might apply.
• Collect a "bag of tricks" - a set of obscure technical methods that you've mastered and can apply opportunistically.
• Find teachers who are willing to explain how a field works, not just the subject matter.

2. How does the author characterize the cognitive styles of highly effective problem solvers? The author notes that very smart people often have unique cognitive styles that give them an edge. He contrasts his own tendency towards "interesting" approaches with his collaborator Ajay's preference for the simplest, most obvious solutions that work. The author suggests trying to figure out and develop your own cognitive style as a strength.

3. What is the author's view on the role of formal methods versus informal reasoning in effective thinking? The author suggests that insights from informal reasoning and observation are probably more important than understanding technical rationality methods. He argues that the Bayesian framework and other formal tools only address a small part of what is needed for effective thinking.

[04] Conclusions

1. What are the key takeaways from the author's overall perspective on thinking about thinking? The main points are:

• Figuring out how to think effectively is an extremely difficult, open-ended challenge without a single general method.
• Problem formulation and selection is as important as problem solving itself, and requires careful observation of the real world.
• A good problem formulation exposes the relevant distinctions and abstracts away irrelevant details, making the problem easier to solve.
• Progress often requires applying a diverse range of intellectual tools and methods, not just focusing on one formal framework like Bayesian probability.
• Meta-level knowledge about how different fields and methods work is critical, but very hard to come by.

2. Why does the author see this as a challenge that requires a collaborative community effort? The author acknowledges that his own attempt to write a comprehensive book on "How to Think Real Good" was too ambitious for a single person. He suggests that the LessWrong community's collaborative approach is well-suited to tackling this broad and difficult challenge of understanding effective thinking.

3. What does the author hope to achieve by sharing these ideas and inviting further discussion? The author seems to hope that by outlining some of his key insights and heuristics around thinking about thinking, and inviting further discussion and collaboration, the community can make progress on this important but elusive problem. He recognizes the limitations of his own "brain dump" approach, and sees the value in a more systematic, collective effort.