Xiaotao Chen
Senior Engineer, End-side AI Inference Optimization, Meituan
Graduated from School of Computer Science, National University of Defense Technology, he has been engaged in AI inference, quantization, and training optimization. He has worked and interned at Intel, Tucson Future, and is currently working on end-side AI inference and optimization at Meituan. In the postgraduate stage, he mainly does research related to distributed deep learning system, and has in-depth understanding and development of low-bit communication, Parameter Server, Horovod, MXNet and other frameworks; after work, he mainly engages in AI inference deployment optimization related content, including: 1. Inference engine development: model quantization (PTQ, QAT), graph optimization, operator optimization, etc.; 2. Deployment Framework development: architecture design, tool development, service construction, etc.
Topic
Application and Practice of a Cross-Platform High-Performance Edge-End AI Inference Deployment Framework
Introduction: In many business scenarios of Meituan, various AI algorithms need to be deployed on different hardware from different business requirements, hardware costs and other considerations. In order to isolate the algorithm from the underlying hardware, so that the algorithm can be deployed to any hardware with one key, we have designed and developed a high-performance edge-side AI inference deployment framework that supports multiple hardware and is flexible and easy to use, which greatly improves the efficiency of the algorithm deployment, and at the same time has a high degree of scalability, which allows for the continuous addition of new hardware and inference back-end. Currently, the framework supports AI models that already support mainstream vision tasks such as classification, detection, segmentation, keypoints, OCR, etc. The supported hardware are: Rexchip RV1106/RV1126/RK3588, Aixin AX620U/AX650N, Allwinner V851, Android Arm, and other 7 major categories of common hardware.
