Also known as autoencoding models.
The term “auto-encoding” in this context refers to the model’s ability to encode (i.e., represent) input sequences (like sentences) in a way that captures their internal structure and semantics without an explicit decoding phase to reconstruct the original input.
In the context of NLP, encoder-only models like BERT are sometimes referred to as “auto-encoders,” but this term differs from its traditional use in machine learning. Unlike classic autoencoders, which have distinct encoding and decoding phases for tasks like dimensionality reduction, models like BERT focus solely on encoding. They transform input text into complex representations, capturing linguistic patterns and semantics, without a decoding phase to reconstruct the original input. BERT uses self-supervised learning, particularly Masked Language Modeling (MLM), where it predicts masked words based on their context, emphasizing the model’s ability to automatically encode text. This is distinct from the non-contextual representations of traditional autoencoders. Thus, in NLP, “auto-encoding” describes the capability of models like BERT to autonomously learn meaningful text representations, highlighting a divergence in terminology usage between general machine learning and the specific field of natural language processing.
BERT and similar models use self-supervised learning techniques, where the model is trained to predict or fill in parts of the input data based on the context provided by the input itself. For example, BERT uses a training objective called Masked Language Modeling (MLM), where some words in the input are masked, and the model predicts these masked words based solely on their context. This process still involves learning an encoding of the input text that is rich and informative enough to allow for accurate predictions, but it doesn’t involve reconstructing the original text in the same way a traditional autoencoder would.
See Also: Decoder only models, Encoder-Decoder models