But his love for golf still burns like an old flame, and he remains ready to tee it up at any opportunity.
"I have my golf clubs in the back of my car, ready to go," he said. "I just can't get out there."
That wasn't a problem in Perry's younger days, when he hit the links as often as possible and expected to pursue the game at the college and/or professional levels.
"It was what I wanted to do," he said. "Before high school, I'd be out on the golf course every day in the summer with my brothers."
Perry, whose other interests include playing piano, was an all-conference golfer in his first three years of high school, and he regularly shot in the mid 80s before he started devoting much more time to basketball.
Two brothers, Ahmad and Chad, played collegiate golf at Hampton, and the latter was an academic All-American.
"We used to go out and compete, and it was really fun," Perry said. "I really miss it."
Their father, from whom Perry said he learned much of what he knows about golf, made things easier for his sons by springing for the membership fee at a country club in his hometown of Durham, N.C.
And he continued his help when Perry turned from golf to basketball, even organizing an AAU club to give the son another opportunity to play.
"He was very supportive when I made the transition from golf to basketball," Perry said.
Perry left the shadow of the Duke and North Carolina campuses to come to Kentucky. While a Kentuckian might think that a surprising move for a North Carolinian, Perry said his family and friends supported it all the way.
"They were real supportive," he said. "They know how good Kentucky is and what kind of program they have."
Perry brought with him from the Tar Heel State a passion for playing defense that he said was instilled in him by his coach at Hillside High School, Chet Mebane.
"He taught me to love to play defense," Perry said. "He said, 'If you don't enjoy it, then why are you out there?' His whole system was based around defense."
Perry said because of that, he already had a leg up when he started learning Kentucky's defense.
"Some of the concepts we started going over in practice, I'm already used to," he said. "Some of the defense is just taking it to another level."
Perry is also a dangerous outside shooter, as he displayed at the Kentucky Derby Festival all-star game by coming within one shot of winning the 3-point contest. Kentucky coach Tubby Smith said while Perry is "more of a perimter player," he could be used at either the three or four position.
"It really doesn't matter," Perry said of his preference for a position. "In high school I played the four or five, but I want to show people I can play other positions as well.
"If (Smith) has faith in me, he'll put me in, and I have confidence I can go in and do my best."