Pretrained language models have transformed the way we process natural languages, enhancing the performance of related systems. BERT has played a pivotal role in revolutionizing the field of Natural Language Processing (NLP). However, the deep learning framework behind BERT lacks interpretability. Recent research has focused on explaining the knowledge BERT acquires from the textual sources used for pre-training its linguistic model. In this study, we analyze the latent vector space produced by BERT's context-aware word embeddings. Our aim is to determine whether certain areas of the BERT vector space have an explicit meaning related to a Knowledge Graph (KG). Using the Link Prediction (LP) task, we demonstrate the presence of explicit and meaningful regions of the BERT vector space. Moreover, we establish links between BERT's vector space and specific ontology concepts in the KG by learning classification patterns. To the best of our knowledge, this is the first attempt to interpret BERT's learned linguistic knowledge through a KG by relying on its pre-trained context-aware word embeddings.

Semantic Interpretation of BERT embeddings with Knowledge Graphs / De Bellis, A.; Biancofiore, G. M.; Anelli, V. W.; Narducci, F.; Di Noia, T.; Ragone, A.; Di Sciascio, E.. - 3478:(2023), pp. 181-191. (Intervento presentato al convegno 31st Symposium of Advanced Database Systems, SEBD 2023 tenutosi a ita nel 2023).

Semantic Interpretation of BERT embeddings with Knowledge Graphs

De Bellis A.;Biancofiore G. M.;Anelli V. W.;Narducci F.;Di Noia T.;Di Sciascio E.
2023-01-01

Abstract

Pretrained language models have transformed the way we process natural languages, enhancing the performance of related systems. BERT has played a pivotal role in revolutionizing the field of Natural Language Processing (NLP). However, the deep learning framework behind BERT lacks interpretability. Recent research has focused on explaining the knowledge BERT acquires from the textual sources used for pre-training its linguistic model. In this study, we analyze the latent vector space produced by BERT's context-aware word embeddings. Our aim is to determine whether certain areas of the BERT vector space have an explicit meaning related to a Knowledge Graph (KG). Using the Link Prediction (LP) task, we demonstrate the presence of explicit and meaningful regions of the BERT vector space. Moreover, we establish links between BERT's vector space and specific ontology concepts in the KG by learning classification patterns. To the best of our knowledge, this is the first attempt to interpret BERT's learned linguistic knowledge through a KG by relying on its pre-trained context-aware word embeddings.
2023
31st Symposium of Advanced Database Systems, SEBD 2023
Semantic Interpretation of BERT embeddings with Knowledge Graphs / De Bellis, A.; Biancofiore, G. M.; Anelli, V. W.; Narducci, F.; Di Noia, T.; Ragone, A.; Di Sciascio, E.. - 3478:(2023), pp. 181-191. (Intervento presentato al convegno 31st Symposium of Advanced Database Systems, SEBD 2023 tenutosi a ita nel 2023).
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11589/262723
Citazioni
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
social impact