Join Maillist for updating: https://forms.gle/wM8o5yVvRRbq2Q1c9 Introduction The joint workshop of FinNLP, FNP, and LLMFinLegal aims to explore the intersection of Natural Language Processing (NLP), ...

Introduction With the advent of LLM in finance, financial text analysis, generation, and decision-making tasks have received growing attention, including financial classification, financial text summa...

Join Maillist for updating: https://forms.gle/wM8o5yVvRRbq2Q1c9 Introduction The joint workshop of FinNLP, FNP, and LLMFinLegal aims to explore the intersection of Natural Language Processing (NLP), ...

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide variety of Financial Natural Language Processing (FinNLP) tasks. However, systematic comparisons among widely used LLMs remain underexplored. Given the rapid advancement and growing influence of LLMs in financial analysis, this study conducts a thorough comparative evaluation of five leading LLMs, GPT, Claude, Perplexity, Gemini and DeepSeek, using 10-K filings from the 'Magnificent Seven' technology companies. We create a set of domain-specific prompts and then use three methodologies to evaluate model performance: human annotation, automated lexical-semantic metrics (ROUGE, Cosine Similarity, Jaccard), and model behavior diagnostics (prompt-level variance and across-model similarity). The results show that GPT gives the most coherent, semantically aligned, and contextually relevant answers; followed by Claude and Perplexity. Gemini and DeepSeek, on the other hand, have more variability and less agreement. Also, the similarity and stability of outputs change from company to company and over time, showing that they are sensitive to how prompts are written and what source material is used.

Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide variety of Financial Natural Language Processing (FinNLP) tasks. However, systematic comparisons among widely used LLMs remain underexplored. Given the rapid advancement and growing influence of LLMs in financial analysis, this study conducts a thorough comparative evaluation of five leading LLMs, GPT, Claude, Perplexity, Gemini and DeepSeek, using 10-K filings from the 'Magnificent Seven' technology companies. We create a set of domain-specific prompts and then use three methodologies to evaluate model performance: human annotation, automated lexical-semantic metrics (ROUGE, Cosine Similarity, Jaccard), and model behavior diagnostics (prompt-level variance and across-model similarity). The results show that GPT gives the most coherent, semantically aligned, and contextually relevant answers; followed by Claude and Perplexity. Gemini and DeepSeek, on the other hand, have more variability and less agreement. Also, the similarity and stability of outputs change from company to company and over time, showing that they are sensitive to how prompts are written and what source material is used.

Join the FinNLP-FNP-LLMFinLegal Workshop at COLING-2025 in Abu Dhabi, UAE, exploring financial technology, narrative processing, and large language models in fi...

Democratizing Internet-scale financial data. Contribute to AI4Finance-Foundation/FinNLP development by creating an account on GitHub.