Benchmarks offers a fully reproducible platform to compare the performance of various MLOps engines, frameworks, and languages on mainstream AI models. By examining crucial performance metrics, this repository empowers researchers and practitioners to make informed decisions about the tools they adopt in their AI workflows.
🕹️ Benchmarks offers a fully reproducible performance comparison of MLOps engines, frameworks, and languages specifically tailored for mainstream AI models.
This repository enables you to make informed decisions about which engine or precision best suits your requirements, particularly in the context of Large Language Model (LLM) inference workflows.
Overview
This project highlights key performance benchmarks of state-of-the-art AI models such as Mistral 7B v0.1 Instruct and Llama 2 7B Chat, showcasing metrics across various precision levels and inference engines. Here are some highlights of what you can expect:
Quick Performance Metrics
You can glance at the performance with metrics measured in tokens per second and GPU memory consumption.
Performance Metrics for Mistral 7B v0.1 Instruct:
Tokens per Second:
| Engine | float32 | float16 | int8 | int4 |
|---|---|---|---|---|
| Nvidia TensorRT-LLM | 117.04 ± 2.16 | 206.59 ± 6.93 | 390.49 ± 4.86 | 427.40 ± 4.84 |
| ctransformers | - | - | 86.14 ± 1.40 | 87.22 ± 1.54 |
GPU Memory Consumption:
| Engine | float32 | float16 | int8 | int4 |
|---|---|---|---|---|
| Nvidia TensorRT-LLM | 79536.59 | 78341.21 | 77689.0 | 77311.51 |
| ctransformers | - | - | 10255.07 | 6966.74 |
MLOps Engines
The benchmark assesses a range of MLOps engines, providing a summary of their support matrices. This resource can help you navigate through various ML engines and frameworks to find the optimal choice based on the performance and support for different precisions.
Why Use Benchmarks?
Benchmarks play a vital role in aiding decision-making for AI deployments. Here’s why this repository is invaluable:
- Alleviates confusion by providing a clear comparison of engine options based on performance metrics specific to your use case.
- Helps understand the trade-off between quality and speed, allowing prioritization of needs.
- Offers a fully reproducible script, equipped with best practices for robust benchmarking on GPU devices.
Seamless Usage and Workflow
The project is structured for easy benchmark management and execution. Simply download the required models with the provided scripts and start benchmarking with a straightforward command line interface.
Contributing to Benchmarks
The repository welcomes contributions! You can create new benchmarks by following specific guidelines to ensure consistency and quality. Learn about the steps to contribute your implementations and become part of the collaborative effort.
For more insights and detailed analytics, check out our release blog and ensure you're leveraging the full potential of benchmarked MLOps engines in your AI projects.