Smart Expert System: Large Language Models as Text Classifiers

🌈 Abstract

The paper introduces the Smart Expert System, a novel approach that leverages Large Language Models (LLMs) as text classifiers. The system simplifies the traditional text classification workflow, eliminating the need for extensive preprocessing and domain expertise. The performance of several LLMs, machine learning (ML) algorithms, and neural network (NN) based structures is evaluated on four datasets. The results demonstrate that certain LLMs can surpass traditional methods in sentiment analysis, spam SMS detection, and multi-label classification. Furthermore, the system's performance can be further enhanced through few-shot or fine-tuning strategies, making the fine-tuned model the top performer across all datasets.

🙋 Q&A

[01] Introduction

1. What are the key challenges with traditional ML and NN approaches to text classification?

• Traditional ML and NN approaches involve a complex multi-stage pipeline that includes feature extraction, dimensionality reduction, classifier selection, and model evaluation. This process requires significant domain expertise to preprocess the data and engineer relevant features.
• Each step in the process must be carefully tuned to optimize performance, which is a time-consuming endeavor involving trial-and-error experimentation.
• The traditional methods may not generalize well across different datasets or languages without substantial reconfiguration.

2. How does the proposed Smart Expert System address these challenges?

• The Smart Expert System utilizes LLMs as text classifiers, which simplifies the conventional text classification process by eliminating the need for extensive preprocessing and feature engineering.
• The system leverages the advanced capabilities of LLMs, offering a more streamlined and efficient method for classifying text without the complexities of traditional ML/NN methods.
• By integrating LLMs into the core of text classification workflows, the system aims to reduce complexity and enhance performance across various domains and applications.

[02] Methodology

1. What are the key components of the proposed Smart Expert System framework?

• Data aggregation from various public and private sources
• Leveraging domain-specific data through zero-shot prompting, few-shot learning, or fine-tuning techniques to adapt pre-trained LLMs
• Optional involvement of domain knowledge experts to configure customized prompts
• LLM API for seamless user interaction and real-time query processing
• Evaluation subsystem for continuous monitoring and performance enhancement

2. What novel evaluation metric is introduced in the paper? The paper introduces a new performance evaluation metric called the Uncertainty/Error Rate (U/E rate), which quantifies the frequency at which an LLM either refuses to classify content or provides an output deemed unrelated or beyond its capabilities. This metric complements traditional performance metrics like accuracy and F1 score.

[03] Experimental Results

1. How did the performance of LLMs compare to traditional ML algorithms and NN architectures across the different datasets?

• On the COVID-19-related tweets dataset, certain LLMs like GPT-3.5 and GPT-4 outperformed traditional ML algorithms and NN models in sentiment analysis.
• On the e-commerce product text classification dataset, NN-based models like GRU and LSTM showed the best performance before fine-tuning, but LLMs like GPT-4 and Gemini-pro achieved superior results after fine-tuning.
• On the economic texts sentiment classification dataset, fine-tuned LLMs like Qwen-7B(F) exhibited the highest accuracy and F1 scores, surpassing traditional models.
• On the SMS spam collection dataset, fine-tuned LLMs like Llama-3-8B(F) and Qwen-7B(F) achieved near-perfect accuracy and F1 scores, outperforming all other models.

2. What insights did the U/E rate provide in the evaluation of LLM performance?

• The U/E rate highlighted instances where LLMs exhibited behavior divergent from deterministic ML/NN models, such as refusing to analyze content or producing hallucinated results.
• After fine-tuning, the U/E rate across models and datasets was reduced to 0, except for the Llama-3-8B(F) model in tweet classification, indicating improved consistency and reliability of the LLM outputs.

[04] Discussion

1. What are the key limitations of LLMs as zero-shot text classifiers identified in the paper?

• Inconsistency in standardized output formats, which can hinder integration with existing systems
• Restrictions on content classification, where LLMs may refuse to classify certain types of content
• Proprietary constraints of commercial closed-source LLMs, such as API request rate limits and associated costs
• Significant hardware resource demands and time-intensive processing, which can impact real-time or high-throughput applications

2. How did the paper's findings on the impact of few-shot learning and fine-tuning strategies vary across models and datasets?

• The impact of few-shot learning was found to be highly model and dataset-dependent, with some models showing only marginal performance changes, while others experienced significant improvements or decreases.
• Fine-tuning strategies, on the other hand, consistently led to substantial performance enhancements, with fine-tuned LLMs like Qwen-7B(F) and Llama-3-8B(F) outperforming all other models across the evaluated datasets.

[05] Conclusion and Future Work

1. What are the key contributions of the proposed Smart Expert System?

• The system simplifies the traditional text classification process by leveraging LLMs as text classifiers, eliminating the need for extensive preprocessing and domain expertise.
• The introduction of the Uncertainty/Error Rate (U/E rate) metric provides a more comprehensive evaluation of LLM performance, highlighting their reliability in real-world applications.
• The empirical evaluation demonstrates that certain LLMs, especially after fine-tuning, can outperform traditional ML and NN models in various text classification tasks.

2. What are the future research directions identified in the paper?

• Addressing the challenges associated with fine-tuning, such as the need for substantial computational resources and expertise.
• Exploring alternative strategies to improve LLM performance, such as more detailed background provisions and precise label definitions.
• Refining the handling of non-standardized outputs, potentially by employing a secondary LLM to process initial results.
• Continuing to make LLM-powered expert systems more accessible and user-friendly across various applications.