BERT – Google’s Public NLP Neural Network Training

What is BERT?

Bidirectional Encoder Representations from Transformers, or BERT, is a Google NLP pre-training code repository. When users use Search, they often don’t have the knowledge or spelling to be able to ask what they really want to know. This is where understanding natural language comes into play by much of the Search algorithm.

BERT is the latest break through in NLP (Natural Language Processing) as a result of its new methodology in understanding keywords and phrases as a whole. The problem in the past has always been, how do you teach a computer to know something, that a human doesn’t always know?

Users have grown to accommodate the Search algorithm by searching in ways called “keyword-ese,” by stringing together keywords. For example, users search “gym Madrid open,” rather than “gyms open in Madrid now,” because they’ve have grown to understand how to search so that a Search Engine can make best sense of the phrase.

BERT aims to pivot from the prior system to the latter system, and make Search more of a dialogue, rather than an adjustment to learning “how to search” on Google.

With the latest advancements from our research team in the science of language understanding–made possible by machine learning–we’re making a significant improvement to how we understand queries, representing the biggest leap forward in the past five years, and one of the biggest leaps forward in the history of Search. 

Google AI Blog

How does BERT work?

What makes BERT a revolutionary model, is that traditionally, search phrases were parsed word by word to determine meaning. BERT uses the proceeding and preceding words to determine the meaning of the middle word.

BERT models can therefore consider the full context of a word by looking at the words that come before and after it—particularly useful for understanding the intent behind search queries.

Google AI Blog

This breakthrough is based on a system called “Transformers.” Transformers is the latest break through in neural network training, where an entire phrase is considered in order to determine meaning of each keyword. This has become particularly important for determining the search intent behind a user query.

Neural networks usually process language by generating fixed- or variable-length vector-space representations. After starting with representations of individual words or even pieces of words, they aggregate information from surrounding words to determine the meaning of a given bit of language in context. For example, deciding on the most likely meaning and appropriate representation of the word “bank” in the sentence “I arrived at the bank after crossing the…” requires knowing if the sentence ends in “… road.” or “… river.”

Google AI Blog

Watch the following video to understand how BERT would utilise ever word in a phrase, to determine their combined meaning. In this particular case, Transformers have been applied to translation of an English phrase, to a French phrase, and have seen success rates of roughly 41%.

Google's BERT Transformers

Transformers, being the new neural network processors, are far quicker than existing trainers, but require more advanced hardware. Google has adopted the use of Cloud TPUs in order to process Search results more effectively utilising BERT, and delivery them to a user at the same speed.

BERT has so far increased Search query accuracy by 10% in English alone, within some months of its deployment. As its premised on Machine Learning, this value will only increase with data exposure over time. Combined with the maintained delivery speed, BERT has overall resulted in Search becoming 10% more efficient in provision of value to a user!

Application of BERT for SEOs

BERT is in the process of being fully deployed for English. Once fully deployed (projected to be in February 2020), BERT will be taken to other languages, as NLP trainers can be applied to any language. This means that regardless of the language settings of your site, you will be subject to BERT re-rankings, and in particular, featured snippets.

For featured snippets, we’re using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi and Portuguese.

Google AI Blog

As a result, featured snippet changes will see some dramatic changes in the coming months as the deployment is extended throughout English, and then to additional languages.

These changes will most likely be reflected in increased adoption of answer passages and structured data markup.

1 thought on “BERT – Google’s Public NLP Neural Network Training”

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: