Thinking, fast and slow with LLMs and PDDL

🌈 Abstract

The article discusses the limitations of large language models (LLMs) like ChatGPT in solving complex reasoning tasks, and introduces the Planning Domain Definition Language (PDDL) and symbolic solvers as a way to address these limitations. It covers the following key points:

🙋 Q&A

[01] Introduction

1. What are the limitations of LLMs like ChatGPT in solving complex reasoning tasks?

• LLMs are fundamentally probabilistic and can make mistakes, even on basic reasoning tasks.
• They often fail to produce the optimal sequence of actions required to solve problems, especially when there are additional constraints.

2. How does the article propose to address the limitations of LLMs?

• The article introduces the Planning Domain Definition Language (PDDL) and symbolic solvers as a way to formally describe and solve complex reasoning problems.
• PDDL allows for the precise definition of the problem domain, including objects, predicates, and actions, which can then be used by specialized planning solvers to find the optimal sequence of actions.

[02] PDDL and Symbolic Solvers

1. What are the key components of a PDDL problem definition?

• The problem definition includes a list of objects, the initial state, and the goal state.
• The initial state is described using predicates, which are statements that can be true or false.
• The goal state is also described using predicates, which must all be true for the goal to be achieved.

2. How are actions defined in PDDL?

• Actions in PDDL have parameters, preconditions, and effects.
• Preconditions are the predicates that must be true for the action to be executed.
• Effects are the predicates that become true or false after the action is executed.

3. What are the benefits of using PDDL and symbolic solvers compared to LLMs?

• PDDL and symbolic solvers can guarantee the correctness of the solution, as they systematically evaluate all possible actions and constraints.
• They can handle more complex problems, including those involving quantities, time, and conditional effects.
• However, setting up the PDDL problem definition can be more cumbersome than prompting an LLM.

[03] Integration with LLMs

1. How can PDDL and LLMs be integrated?

• LLMs can be used to generate appropriate goals for a PDDL problem, which can then be solved by a symbolic planner.
• This approach can be more tractable than having an LLM generate the entire sequence of actions, especially for complex tasks.

2. What are the limitations of this integrated approach?

• While the integrated approach can address some of the limitations of LLMs, it will eventually hit the same limitations as the PDDL problem definition and solver capabilities.
• Tighter integration between open-world reasoning (LLMs) and symbolic planning is an area of ongoing research.

[04] Future Developments

1. What are some recent advancements in planning and reasoning systems?

• The article mentions the Unified Planning library and the Scikit-decide framework, which provide more flexible and integrated approaches to planning problems.
• These systems can support replanning and plan repair, which are important for real-time applications.

2. What are the potential future directions for combining LLMs and symbolic planning?

• LLMs could be used to prioritize search in symbolic planning, using common sense information.
• Symbolic representations and predicates could be used to improve the perception and understanding of the real world for LLMs.