Peng Cheng Laboratory (PCL) and Chinese tech giant Baidu jointly released the world’s first large-scale knowledge enhanced 100-billion-scale model Wednesday.
The AI model, named PCL-Baidu Wenxin (model version: ERNIE 3.0 Titan), is also said to be the world’s largest Chinese monolithic model that includes a parameter scale reaching 260 billion.
The model has achieved state-of-the-art results in more than 60 tasks including machine reading comprehension, text categorization, and semantic similarity calculation, as well as in over 30 small sample and zero-sample tasks.
It is said that pretrained large-scale models have become a new highland in artificial intelligence (AI).
The model is attributed to the collaboration between PCL’s self-developed computing system Peng Cheng Cloud Brain II and PaddlePaddle, Baidu’s deep learning platform, which helped address key challenges of hyper-scale model training to enhance the model’s performance.
Researchers applied model compression technology to simplify the PCL-Baidu Wenxin model for real-world scenarios. The compressed model retains only 0.02 percent of the original size but can achieve comparable performance.
The model will further address difficulties such as the lack of field and scenario data in AI technology’s industrial application.
In the near future, the model code will be open source from the OpenI Qizhi community and also via Pengcheng Cloud Brain II to fully tap the empowerment capabilities of the AI large-scale model, help scientific and technological innovation, and promote industrial development.
“The knowledge-enhanced PCL-Baidu Wenxin model learns from an integration of large-scale knowledge and massive data, improving effectiveness and efficiency while achieving great interpretability,” said Wang Haifeng, Baidu CTO.
At present, the model code has been made open on PaddlePaddle and widely used in internet products, including Baidu’s search engine, information flow and smart speaker. It will also empower various sectors such as energy, finance, media and education via Baidu AI Cloud.
In terms of the finance sector, the model can analyze and recognize relevant contract clauses within one minute, dozens of times faster than before, greatly improving work efficiency.