Can ChatGPT do data science?

🌈 Abstract

The article discusses the challenges data scientists face when using ChatGPT for data science tasks, and provides recommendations for designing AI-powered data science tools.

🙋 Q&A

[01] Conversational Challenges in AI-Powered Data Science

1. What are the key challenges data scientists face when using ChatGPT for data science tasks?

• Sharing context is difficult - data scientists struggle with determining how much information and context to provide to ChatGPT.
• ChatGPT makes opaque assumptions that can lead to incorrect results.
• Data scientists have misaligned expectations about ChatGPT's capabilities and the format of its responses.
• ChatGPT generates repeated or unorganized code that doesn't match data scientists' preferences.
• Data scientists need to extensively validate the code generated by ChatGPT.

2. What strategies did data scientists use to overcome these challenges?

• Techniques for prompt construction, such as one-shot prompting, few-shot prompting, chain of thought prompting, and asking ChatGPT to assume the role of an expert.
• Scaffolding with their own domain expertise to guide ChatGPT.
• Choosing alternative resources instead of relying solely on ChatGPT.

[02] Recommendations for AI-Powered Data Science Tools

1. What are the three main recommendations made for designing AI-powered data science tools?

• Provide preemptive and fluid context when interacting with AI assistants, such as interfaces to efficiently select and manage context.
• Provide inquisitive feedback loops and validation-aware operations, where the system guides the user and proactively asks clarifying questions.
• Provide transparency about shared context and domain expertise solutions, with mechanisms for efficient sharing of context and assumptions.