skip to main content
Language:
Search Limited to: Search Limited to: Resource type Show Results with: Show Results with: Search type Index

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Bioinformatics, 2020-02, Vol.36 (4), p.1234-1240 [Peer Reviewed Journal]

The Author(s) 2019. Published by Oxford University Press. 2019 ;The Author(s) 2019. Published by Oxford University Press. ;ISSN: 1367-4803 ;EISSN: 1460-2059 ;EISSN: 1367-4811 ;DOI: 10.1093/bioinformatics/btz682 ;PMID: 31501885

Full text available

Citations Cited by
  • Title:
    BioBERT: a pre-trained biomedical language representation model for biomedical text mining
  • Author: Lee, Jinhyuk ; Yoon, Wonjin ; Kim, Sungdong ; Kim, Donghyeon ; Kim, Sunkyu ; So, Chan Ho ; Kang, Jaewoo
  • Wren, Jonathan
  • Subjects: Original Papers
  • Is Part Of: Bioinformatics, 2020-02, Vol.36 (4), p.1234-1240
  • Description: Abstract Motivation Biomedical text mining is becoming increasingly important as the number of biomedical documents rapidly grows. With the progress in natural language processing (NLP), extracting valuable information from biomedical literature has gained popularity among researchers, and deep learning has boosted the development of effective biomedical text mining models. However, directly applying the advancements in NLP to biomedical text mining often yields unsatisfactory results due to a word distribution shift from general domain corpora to biomedical corpora. In this article, we investigate how the recently introduced pre-trained language model BERT can be adapted for biomedical corpora. Results We introduce BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining), which is a domain-specific language representation model pre-trained on large-scale biomedical corpora. With almost the same architecture across tasks, BioBERT largely outperforms BERT and previous state-of-the-art models in a variety of biomedical text mining tasks when pre-trained on biomedical corpora. While BERT obtains performance comparable to that of previous state-of-the-art models, BioBERT significantly outperforms them on the following three representative biomedical text mining tasks: biomedical named entity recognition (0.62% F1 score improvement), biomedical relation extraction (2.80% F1 score improvement) and biomedical question answering (12.24% MRR improvement). Our analysis results show that pre-training BERT on biomedical corpora helps it to understand complex biomedical texts. Availability and implementation We make the pre-trained weights of BioBERT freely available at https://github.com/naver/biobert-pretrained, and the source code for fine-tuning BioBERT available at https://github.com/dmis-lab/biobert.
  • Publisher: England: Oxford University Press
  • Language: English
  • Identifier: ISSN: 1367-4803
    EISSN: 1460-2059
    EISSN: 1367-4811
    DOI: 10.1093/bioinformatics/btz682
    PMID: 31501885
  • Source: Geneva Foundation Free Medical Journals
    Open Access: PubMed Central
    Oxford Journals Open Access Collection
    Journals@Ovid Open Access Journal Collection Rolling
Back to results list
INSPIRE LIBRARY - TON DUC THANG UNIVERSITY (84-028) 37 755 057 Feedback
19 Nguyen Huu Tho St. Dist.7, HCM thuvien@tdtu.edu.vn

Copyright © 2017 INSPIRE LIBRARY - TON DUC THANG UNIVERSITY

Searching Remote Databases, Please Wait