He's right. Earlier this week Vanderbilt coach Bobby Johnson said his team improved last year even though the Commodores finished 2-10. Brooks' comments were similar as he recounted the games Kentucky lost despite having a chance to win them all in the fourth quarter.
The common problem Kentucky, Vanderbilt and South Carolina have is playing in the SEC Eastern Division where Georgia, Tennessee and Florida are always good - and always on their schedules.
South Carolina, Vanderbilt and Kentucky have combined for only seven wins against the SEC East's elite teams in the last 12 years.
No one expects that to change this year, either, as media members predicted Georgia would win the Eastern Division followed by Florida, Tennessee, South Carolina, Vanderbilt and Kentucky.
Want more? Out of 52 players named to the media's preseason all-SEC team, only five were from Vanderbilt (3), South Carolina (1) and Kentucky (1). By comparison, Georgia had seven, Florida five and Tennessee five. Defending national champion LSU, the preseason pick to win the Western Division, had 11 players honored.
"It's difficult to beat those three (SEC East) teams because they expect to win and find ways to win," Holtz said.
He remembers being at Notre Dame and his team had the same feeling. At South Carolina, he said players start wondering what might go wrong in the third quarter. It's a feeling Kentucky fans, and players, understand.
"Florida will be outstanding this year. Georgia will be one of the better teams in the NFL. I know the athletes Tennessee has recruited," the South Carolina coach said. "But teams outside the SEC having trouble beating those teams, too."
Like Kentucky, South Carolina has had chances to beat the top three but failed. Like Kentucky, South Carolina thought it was moving forward. Hal Mumme took UK to the 1998 Outback Bowl and 1999 Music City Bowl before the program hit hard times again. The Gamecocks went to the Outback Bowl in 2000 and 2001. They won each time. However, like UK, the program went back, not forward.
Don't be fooled by the close losses
Holtz, like Brooks, knows not to be fooled by close losses to a Tennessee or Florida.
"We are light years away from those programs. Light years," Holtz said. "We're not just a play or two away from being like them. We are talking about having to change a culture, the way people act and believe."
The South Carolina coach recently brought about a dozen former South Carolina players who are playing in the NFL or had played there to talk to his team about the way it had to act to win. Kentucky needs to do the same and take advantage of former players like Eric Kelly, Dennis Johnson and Tim Couch.
"You reach a certain level of success and think you are pretty good," Holtz said. "But when you get comfortable where you are, you really are dying. Trying to maintain success is the dumbest thing I've ever done. You better keep trying to get better.
"The talent level is not void at South Carolina. But we are not going to win on talent alone. That's what those pro players emphasized. We have to win as a team and believe we can win."
So what about the one media member who not only predicted South Carolina would win the SEC Eastern Division, but also would be the league's overall champion?
"He probably voted with a crayon. I can't believe they'd give him a sharp instrument," Holtz said. "I've never gone into a season thinking we couldn't win a championship. We are not going to roll over and die, but I sure would like to know what his logic is for picking us No. 1."
Logic? There's none because as Holtz correctly noted, the East's have-nots still have to learn to believe they can beat Florida, Georgia and Tennessee before they can start doing it.