Access and Feeds

GPT-3 Competition: Rapid Development of Massive Natural Language Systems

By Dick Weisinger

GPT-3 has received reviews of astonishment since it was first announced in late May 2020 by OpenAI. GPT-3 (Generative Pre-Trained Transformer 3) is the third generation of natural language processing models developed by OpenAI, an artificial intelligence research operation originally funded by Elon Musk.

GPT-3 has been described as “one of the most interesting and important AI systems ever produced.” After being trained on 45 terabytes of text data, it is able to compose essays and write computer code. It’s ability has stunned many and caused a rethink about the possibilities of machine intelligence.

The burst of excitement around how people would use GPT-3 was tempered when in late September 2020 it was announced that Microsoft had secured privileged access to the GPT-3 source code. Others could license GPT-3 output, but not directly access the source code. The result has been keen interest by a number of groups to try to reproduce GPT-3 capabilities.

Earlier this month, the Beijing Academy for AI (BAAI) announced a “Superscale Intelligence Model.” The Chinese system called Wu Dao 2.0 is 10 times bigger than OpenAI’s GPT-3 and not only composes natural language text in both Chinese and English, it can also generate images and voice.

In the US, the group EleutherAI is trying to replicate the success of GPT-3 with an open-source approach. EleutherAI has built a model they’re calling GPT-J-6B is nearly on par with what GPT-3 can do.

Connor Leahy, one of the founding members of EleutherAI, said that “we think it’s probably fair to say this is currently the best open-source autoregressive language model you can get by a pretty wide margin.”

Chinese AI researcher Blake Yan said that “these sophisticated models, trained on gigantic data sets, only require a small amount of new data when used for a specific feature because they can transfer knowledge already learned into new tasks, just like human beings. Large-scale pre-trained models are one of today’s best shortcuts to artificial general intelligence.”

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)

Leave a Reply

Your email address will not be published.


five + 3 =