Definition
A neural network component that allows models to focus on the most relevant parts of the input when producing each part of the output.
In Depth
Attention mechanisms compute weighted relevance scores between all input elements, allowing the model to 'attend to' different parts of the input depending on context. Self-attention (used in transformers) lets each token attend to all other tokens in the sequence.
Example Usage
When translating 'The cat sat on the mat,' attention lets the model focus on 'cat' when generating the subject in the target language.
Business Context
Attention is what makes modern AI so capable. It is the core innovation behind every transformer model from GPT to Claude to Gemini.
Related Terms
Generative AI
AI systems that can create new content including text, images, code, and data based on training patt...
Hallucination
When AI models generate false or nonsensical information that appears plausible.
Inference
The process of using a trained AI model to make predictions or generate outputs on new data, the 'pr...
Large Language Model
An AI model trained on vast amounts of text data capable of understanding and generating human-like ...
Model Quantization
A technique for reducing AI model size and computational requirements by using lower-precision numbe...
Also Known As
Tags
Quick Info
Need help implementing this in your business?
Get Started