4
views
0
recommends
+1 Recommend
0 collections
    0
    shares
      • Record: found
      • Abstract: found
      • Article: found
      Is Open Access

      BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models

      Preprint
      , , , ,

      Read this article at

      Bookmark
          There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

          Abstract

          Large Language Models (LLMs) often suffer from overconfidence during inference, particularly when adapted to downstream domain-specific tasks with limited data. Previous work addresses this issue by employing approximate Bayesian estimation after the LLMs are trained, enabling them to quantify uncertainty. However, such post-training approaches' performance is severely limited by the parameters learned during training. In this paper, we go beyond post-training Bayesianization and propose Bayesian Low-Rank Adaptation by Backpropagation (BLoB), an algorithm that continuously and jointly adjusts both the mean and covariance of LLM parameters throughout the whole fine-tuning process. Our empirical results verify the effectiveness of BLoB in terms of generalization and uncertainty estimation, when evaluated on both in-distribution and out-of-distribution data.

          Related collections

          Author and article information

          Journal
          17 June 2024
          Article
          2406.11675
          1a696b79-eead-4d1a-85da-730355736138

          http://arxiv.org/licenses/nonexclusive-distrib/1.0/

          History
          Custom metadata
          27 pages, 3 figures, 9 tables; preprint, work in progress
          cs.LG cs.AI cs.CL stat.ML

          Theoretical computer science,Machine learning,Artificial intelligence
          Theoretical computer science, Machine learning, Artificial intelligence

          Comments

          Comment on this article