Reinforcement learning (RL) has been making strides in integrating Transformer architectures, known for handling long-term dependencies. This is crucial as RL algorithms learn to make sequential decisions in complex environments. A significant challenge in RL is understanding past observations and actions’ impact on future outcomes. Transformers have been adapted to RL to enhance memory capabilities. But understanding their effectiveness, especially in long-term credit assignments, is crucial. Researchers have introduced measurable definitions for memory and credit assignment to study Transformers’ impact. They evaluated memory-based RL algorithms, which enabled tasks to isolate the memory and credit assignment capabilities. Transformers enhance memory but don’t improve long-term credit assignment. For practitioners, this study guides the selection of RL architectures based on their applications’ requirements.