The 84-year-old music great was a huge fan of Presley during his younger years, but believes the king of rock and roll squandered his talent after 1957, when he started making teen-focused pop songs, like many other artists of that era.
"My early favourites were Elvis. Then came Elvis Presley, Little Richard, Fats Domino, Chuck Berry, the Everly Brothers," the 16-time Grammy winner told Alchemy with Anthony Mason.
"Those were the people I really loved."
Simon said the period between 1954 and 1957 had the biggest influence on his sound.
"When I first started to listen … as I look back and I do look back, for me the big years that influenced the sound that I still go for occasionally, it's like '54 to '57," he said.
"And after that, I still like things, but I didn't derive anything from them. I didn't take any sounds from stuff … as I was concerned, really by '57, I'd lost interest in Elvis Presley."
Simon said it was Presley's early recordings for Sun Records that truly captivated him.
"Once he went into the army, all that stuff that he recorded, the first group of songs that he made on Sun Records - That's All Right, Mama, Mystery Train, Good Rockin' Tonight, Blue Moon of Kentucky - they were not made for teenagers," he said.
"They were made for the audience that listened at that point and bought records and they were older than teenagers."
Ultimately, Simon viewed Presley's later career as a missed opportunity.
"As I say, I liked (Presley) up to (the year 1957)," he said.
"And after that, when, you know, the material that he picked, and the movies, and everything was just, to me, an incredible waste of a great talent."