The error ImportError: cannot import name 'EncoderDecoderCache' from 'transformers' typically appears when working with encoder-decoder architectures—such as T5, BART, or MarianMT—after upgrading the Hugging Face transformers library to version 4.36.0 or later. This issue stems from a significant internal refactoring of how Key-Value (KV) caches are handled during generation. Understanding the structural changes in the library is the fastest way to resolve the import error and adapt your codebase to the latest best practices.
Why This Error Occurs: The KV Cache Refactor
Prior to version 4.Because of that, 36, the transformers library managed past key-values for encoder-decoder models using a specific class named EncoderDecoderCache. This class acted as a wrapper around two separate cache objects: one for the encoder (usually static) and one for the decoder (dynamic, growing with each generation step).
Starting with version 4.In practice, 36, Hugging Face introduced a unified Cache interface (DynamicCache, StaticCache, QuantizedCache, etc. ) to standardize KV cache handling across all model architectures—decoder-only (like Llama, GPT), encoder-only (like BERT), and encoder-decoder (like T5, BART). So naturally, the legacy EncoderDecoderCache class was deprecated and eventually removed from the public API namespace Less friction, more output..
If your code explicitly imports this class:
# This causes the error in transformers >= 4.36
from transformers import EncoderDecoderCache
The interpreter raises the ImportError because the symbol no longer exists in the top-level transformers module And that's really what it comes down to..
Immediate Fix: Update Your Imports
The most direct solution is to remove the import entirely. Even so, in the vast majority of use cases, you do not need to manually instantiate or interact with the cache class. The generate() method handles cache creation and management internally The details matter here..
Before (Broken):
from transformers import T5ForConditionalGeneration, EncoderDecoderCache
model = T5ForConditionalGeneration.from_pretrained("t5-base")
# Manual cache handling (old pattern)
cache = EncoderDecoderCache()
outputs = model.generate(input_ids, past_key_values=cache)
After (Fixed):
from transformers import T5ForConditionalGeneration
model = T5ForConditionalGeneration.from_pretrained("t5-base")
# Let the library handle the cache automatically
outputs = model.generate(input_ids)
If you were passing a cache object to generate() to implement custom logic (like prefix tuning or cache slicing), you must migrate to the new Cache classes It's one of those things that adds up..
Migrating to the New Cache API (v4.36+)
The modern Cache API offers more granular control and better performance, especially for features like Static Cache (required for torch.compile) and Quantized Cache (for memory efficiency) Most people skip this — try not to..
1. Dynamic Cache (Default Behavior)
This is the drop-in replacement for the old dynamic behavior. It grows dynamically as tokens are generated.
from transformers import DynamicCache
# Explicitly creating a dynamic cache (rarely needed manually)
past_key_values = DynamicCache()
outputs = model.generate(input_ids, past_key_values=past_key_values)
2. Static Cache (For Compilation & Speed)
If you are using torch.compile or targeting maximum inference throughput on GPU, you should use StaticCache. This pre-allocates the maximum memory required for the KV cache based on max_length Easy to understand, harder to ignore..
from transformers import StaticCache
import torch
max_length = 512
batch_size = 1
# Get config details for correct shape allocation
config = model.config
past_key_values = StaticCache(
config=config,
max_batch_size=batch_size,
max_cache_len=max_length,
device=model.device,
dtype=model.dtype
)
# Note: When using StaticCache, you usually cannot use standard `generate()`
# easily without a custom loop or specific generation config flags.
# It is primarily intended for manual generation loops or compiled graphs.
3. Handling Encoder-Decoder Specifics
In the old EncoderDecoderCache, you accessed encoder and decoder caches separately. In the new architecture, the encoder outputs are typically stored inside the model's forward pass, and the decoder cache (DynamicCache or StaticCache) only manages the decoder's self-attention and cross-attention keys/values And it works..
When calling model.On top of that, generate(), the encoder runs once, and the resulting hidden states are passed to the decoder cache as cross-attention keys/values automatically. You generally do not manage the "encoder cache" separately anymore.
Common Scenarios & Solutions
Scenario A: Custom Generation Loops
If you wrote a manual token-by-token generation loop (e.g., for constrained decoding or logging probabilities), you likely passed past_key_values between steps.
Old Pattern:
# Old loop structure
past_key_values = EncoderDecoderCache()
for _ in range(max_new_tokens):
outputs = model(input_ids, past_key_values=past_key_values)
logits = outputs.logits
past_key_values = outputs.past_key_values # This was EncoderDecoderCache
next_token = logits.argmax(-1)
input_ids = next_token.unsqueeze(-1)
New Pattern:
# New loop structure
past_key_values = DynamicCache() # Or StaticCache
for _ in range(max_new_tokens):
outputs = model(input_ids, past_key_values=past_key_values, use_cache=True)
logits = outputs.logits
# The returned object is now a standard Cache instance (DynamicCache)
past_key_values = outputs.past_key_values
next_token = logits.argmax(-1)
input_ids = next_token.unsqueeze(-1)
Note: The outputs.past_key_values returned by encoder-decoder models in v4.36+ is a DynamicCache (or StaticCache) instance containing both self-attention and cross-attention layers for the decoder.
Scenario B: prepare_inputs_for_generation Overrides
If you subclassed a model (e.g., T5ForConditionalGeneration) and overrode prepare_inputs_for_generation, your method signature and logic handling past_key_values must be updated.
The past_key_values argument passed to this method is now a Cache object (or a tuple of tensors in legacy compatibility mode, though tuple support is fading). get_seq_length()replacespast_key_values[0][0].shape[2]. You should use the methods provided by the Cache` class:
past_key_values.*past_key_values.Now, update(new_key, new_value, layer_idx, ... )` replaces manual tuple slicing/concatenation.
Scenario C: Checkpoint Loading / torch.load
If you saved a model checkpoint using torch.save(model.state_dict()) or trainer.save_model() on an older version, and are loading it on a newer version, the model weights themselves are compatible. The error only occurs at the Python code level (imports), not the weight tensor level. Simply fixing the Python import statements resolves the issue.
That said, if you pickled the entire cache object (e.g., torch.save(past_key_values, "cache.pt")) on an old version, you cannot load it directly on the new version because the class definition (EncoderDecoderCache) is gone. You must re-run the forward pass to regenerate the cache using the new classes Not complicated — just consistent. And it works..
Verifying Your Transformers Version
Before applying fixes, confirm your version:
pip show transformers
# or in python
import transformers
print(transformers.__version__)
- Version < 4.36:
EncoderDecoderCacheexists. The error implies a typo or environment mismatch. - Version >= 4.36:
EncoderDecoderCacheis removed
Conclusion: Embracing a Streamlined Future
The transition away from EncoderDecoderCache and toward the unified Cache class is more than a simple API update; it represents a significant step forward in the design and usability of Hugging Face Transformers. While migrating existing code requires attention to detail—particularly in handling custom generation loops and prepare_inputs_for_generation overrides—the long-term benefits are substantial Took long enough..
By consolidating the caching mechanism into a single, intuitive class hierarchy (DynamicCache, StaticCache), the library now offers:
- A Unified API: Interacting with cached key-value states is now consistent across all model architectures, whether they are encoder-decoder, decoder-only, or encoder-only. This simplifies learning and reduces cognitive load when working with different models.
- Improved Efficiency: The new
Cacheobjects are designed for optimal memory usage and performance, with clear methods for updating and retrieving states. This paves the way for future optimizations and more efficient inference, especially in long-running generation tasks. - Enhanced Clarity: The removal of redundant classes like
EncoderDecoderCacheeliminates confusion and makes the codebase cleaner and more maintainable. The logic is now centralized, making it easier to understand and contribute to.
This change is a testament to the library's evolution, prioritizing a dependable, efficient, and developer-friendly foundation. Day to day, we encourage you to use the guidance above to update your projects. The initial effort will be rewarded with code that is not only compatible with the latest versions but also more maintainable and aligned with the future direction of the Transformers ecosystem. Embrace the change; it is a step toward a more streamlined and powerful machine learning workflow.