Glossary

Cross-Attention

Cross-Attention is a transformer mechanism where queries from one sequence attend to keys and values from a different source sequence, enabling models to integrate contextual information across modalities or sequences, such as decoder states querying encoder outputs.

Cross-Attention is a transformer mechanism where queries from one sequence attend to keys and values from a different source sequence, enabling models to integrate contextual information across modalities or sequences, such as decoder states querying encoder outputs.