Customer replaced agency/website and ranking tanked, wants me to bring it back to its old glory

Hey, this a new client. Has some great high authority links. as he was a semi celeb in the past. he replaced his agency about a year and half ago, they replace his website with a very shitty godaddy website that looks like a link farm. unfortunately customer didnt realize this until I made him aware. Now he would like for me to build him a new content website from scratch 100+pages. at the moment the website has about 48 keywords ranked our of which 15 no1 for the brand names but all the rest of the traffic dies down. What are the pitfalls here? is it possible the website is penalized already by google? I figure after fixing the thousand of broken links i can bring back a lot of traffic, am i missing something here? Edit1: just to add, a year and a half ago customer had 1k+ keywords listed.