Recent work has shown the fundamental role that cognitive strategies play in visuomotor adaptation. While algorithmic strategies, such as mental rotation, are flexible and generalizable, they are computationally demanding. To avoid this computational cost, people can instead rely on memory retrieval of previously successful visuomotor solutions. However, such a strategy is likely subject to strict stimulus-response associations and rely heavily on working memory. In a series of five experiments, we sought to estimate the constraints in terms of capacity and precision of working memory retrieval for visuomotor adaptation. This was accomplished by leveraging different variations of visuomotor item-recognition and visuomotor rotation recall tasks where we associated unique rotations with specific targets in the workspace and manipulated the set size (i.e., number of rotation-target associations). Notably, from Experiment 1 to 4, we found key signatures of working memory retrieval and not mental rotation. In particular, participants were less accurate and slower for larger set sizes and less recent items. Using a Bayesian-latent mixture model, we found that such decrease in performance is the result of both an increase in guessing behavior and of less precise samples from memory. In addition we estimated that participants’ working memory capacity was limited to 2-5 items, after which guessing increasingly dominated performance. Finally, in Experiment 5, we showed how the constraints observed across Experiments 1 to 4 can be overcome when relying on long-term memory retrieval. Our results point to the opportunity of studying other sources of memories where visuomotor solutions can be stored (e.g., episodic memories) to achieve successful adaptation.