What is ReasonBERT?
ReasonBERT is a pre-training method that enhances language models with the ability to reason over long-range relations and multiple, possibly hybrid, contexts. It is a deep learning model that uses distant supervision to connect multiple pieces of text and tables to create pre-training examples that require long-range reasoning. This pre-training method is an improvement to existing language models like BERT and RoBERTa.
How does ReasonBERT work?
Imagine you have a query sentence containing an entity pair, and you mask one of the entities. If you have another sentence or table that contains the same pair of entities, you can use it as evidence to recover the masked entity. ReasonBERT collects multiple pieces of evidence that are jointly used to recover the masked entities in the query sentence, allowing for the scattering of the masked entities among different pieces of evidence to mimic different types of reasoning.
Multiple types of reasoning are simulated by ReasonBERT, including intersecting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. For example, if you want to find “the beach soccer competition that is established in 1998,” the model needs to check multiple constraints (i.e., intersection reasoning type) in order to find the answer. In another example, if you want to find “the type of the band that released Awaken the Guardian,” the model needs to use bridging reasoning to first infer the name of the band “Fates Warning.”
The masked entities in a query sentence are replaced with [QUESTION] tokens, and the new pre-training objective, span reasoning, extracts the masked entities from the provided evidence. ReasonBERT uses existing LMs like BERT and RoBERTa, and trains them with the new objective, which leads to the creation of ReasonBERT. When tabular evidence is present, the structure-aware transformer TAPAS is used as the encoder to capture the table structure.
Why is ReasonBERT important?
ReasonBERT is an important development in language models because it allows models to reason over long-range relations and multiple contexts. This can improve the accuracy of natural language processing models, especially in fields where reasoning is important, such as legal or scientific research. ReasonBERT’s ability to collect and utilize multiple pieces of evidence allows for deeper reasoning and more accurate results.
ReasonBERT can also be used in many practical applications. For example, in the field of customer service, ReasonBERT can be used to understand customer queries better and provide more accurate answers. In the healthcare industry, it can be used to deepen understanding of patient medical histories and offer more personalized treatment plans. In the legal industry, it can help legal professionals to accurately interpret and analyze complex legal documents.
ReasonBERT is a pre-training method that enhances language models with the ability to reason over long-range relations and multiple, possibly hybrid, contexts. It is an improvement to existing language models like BERT and RoBERTa, and allows for deeper reasoning and more accurate results. ReasonBERT can be used in many practical applications, including customer service, healthcare, and legal research. With its ability to simulate different types of reasoning and utilize multiple pieces of evidence, ReasonBERT is a powerful tool for natural language processing.
