This was written for the movie The Graduate, starring Anne Bancroft as Mrs. Robinson, a middle-aged woman who seduces the much younger Dustin Hoffman. Bancroft, who died in 2005, had a long and successful film career, but is best known for her part in this movie.

Regarding the famous line, "Where have you gone Joe DiMaggio?": DiMaggio was a star baseball player for the New York Yankees who was briefly married to Marilyn Monroe. Simon was using him to represent heroes of the past. DiMaggio was a little miffed when he heard this, since he was still very much alive even though he retired from baseball in 1951, but he realized that he had become a new icon now with the baby boomer generation due to this song's success. Simon, who is a huge fan of The Yankees, explained in a 1990 interview with SongTalk magazine: "The Joe DiMaggio line was written right away in the beginning. And I don't know why or where it came from. It seems so strange, like it didn't belong in that song and then, I don't know, it was so interesting to us that we just kept it. So it's one of the most well-known lines that I've ever written."

Paul Simon was a much bigger fan of Mickey Mantle than Joe DiMaggio. On The Dick Cavett Show, Simon was asked by Mantle why he wasn't mentioned in the song instead of DiMaggio. Simon replied, "It's about syllables, Mick. It's about how many beats there are."