Definition
A technique for reducing AI model size and computational requirements by using lower-precision numbers (e.g. 8-bit instead of 32-bit).
In Depth
Quantization converts model weights from 32-bit floating point to 8-bit or 4-bit integers, reducing memory usage by 4-8x with minimal quality loss. This enables running large models on smaller GPUs or even CPUs. Techniques include post-training quantization and quantization-aware training.
Example Usage
Running a quantized Llama model locally for development and testing instead of making API calls to cloud-hosted models.
Business Context
Quantization makes AI deployment economically viable on edge devices and smaller servers, reducing inference costs by 50-80%.
Related Terms
AI Safety
The field focused on ensuring AI systems behave as intended, don't cause harm, and remain aligned wi...
Attention Mechanism
A neural network component that allows models to focus on the most relevant parts of the input when ...
Edge Computing
Processing data near the source rather than in centralized cloud servers.
Inference
The process of using a trained AI model to make predictions or generate outputs on new data, the 'pr...
Prompt Engineering
The practice of designing and optimizing input prompts to get desired outputs from AI models.
Also Known As
Tags
Quick Info
Need help implementing this in your business?
Get Started