What is DeepSpeed?
It is a deep learning optimization library that makes distributed training easy, efficient, and effective. It can train DL models with over a hundred billion parameters on the current generation of GPU clusters while achieving over 5x in system performance compared to the state-of-art. Early adopters of DeepSpeed have already produced a language model (LM) with over 17B parameters called Turing-NLG, establishing a new SOTA in the LM category.
DeepSpeed is a tool in the Development & Training Tools category of a tech stack.
Key Features
Distributed Training with Mixed PrecisionModel ParallelismMemory and Bandwidth OptimizationsSimplified training APIGradient ClippingAutomatic loss scaling with mixed precisionSimplified Data LoaderPerformance Analysis and Debugging
DeepSpeed Pros & Cons
Pros of DeepSpeed
No pros listed yet.
Cons of DeepSpeed
No cons listed yet.
DeepSpeed Alternatives & Comparisons
What are some alternatives to DeepSpeed?
TensorFlow
TensorFlow is an open source software library for numerical computation using data flow graphs. Nodes in the graph represent mathematical operations, while the graph edges represent the multidimensional data arrays (tensors) communicated between them. The flexible architecture allows you to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API.
PyTorch
PyTorch is not a Python binding into a monolothic C++ framework. It is built to be deeply integrated into Python. You can use it naturally like you would use numpy / scipy / scikit-learn etc.
scikit-learn
scikit-learn is a Python module for machine learning built on top of SciPy and distributed under the 3-Clause BSD license.
