10 NLP Predictions for 2022

Natural language processing (NLP) has been one of the hottest sectors in AI over the past two years. Will the string of big data breakthroughs continue into 2022? We checked in with industry experts to find out.

There’s been a veritable arms race to develop large transformer models over the past couple of years. It started in 2020 with OpenAI’s GPT-3 with 175 billion parameters. Then Microsoft and Nvidia teamed up on MT-NLG (Megatron-Turing Natural Language Generation), which sported 530 billion parameters. Finally in 2021, Google gave us its Switch Transformer with 1.6 trillion parameters.

Don’t expect the race to build ever larger transformer models to slow down in 2022, says Natalia Vassilieva, director of product for machine learning at AI hardware maker Cerebras.

“These larger models promise better results in a variety of natural language tasks with arguably one of the most interesting being an ability to generate an answer to any posted question,” she writes. “However, these giant models are usually trained on very large corpora of publicly available generic texts crawled from all over the Internet. So the answers generated by these models will rely on that public data.”

What’s more, there’s a gap between what these models trained on generic data can do versus what a model trained on a company’s domain-specific data can do, Vassilieva says. We’ll start to close that gap with the large transformer models this year.

Read more