Content-Based or Collaborative Insights from Inter-List Similarity Analysis of ChatGPT Recommendations

ChatGPT has demonstrated remarkable versatility across various domains, including Recommender Systems (RSs). Unlike traditional RSs, ChatGPT generates recommendations through natural language, leveraging contextual cues and large-scale knowledge representations. However, it remains unclear whether these recommendations implicitly encode collaborative patterns, rely on semantic item similarities, or follow a fundamentally different paradigm. In this work, we systematically analyze ChatGPT's recommendation behavior by comparing its generated lists to collaborative and content-based filtering baselines across three domains: Books, Movies, and Music. Using established list similarity metrics, we quantify the alignment of ChatGPT's recommendations with traditional paradigms. Additionally, we investigate the most recommended items by ChatGPT and the other recommenders, comparing the distribution of frequently recommended items across models. Our findings reveal that ChatGPT exhibits strong similarities to collaborative filtering (CF) and amplifies popular yet underrepresented items in the dataset, suggesting a broader domain knowledge encoded in the language model and the need for future research on leveraging LLMs for recommendation tasks.