But this turned out to be a blessing in disguise. Mr Shanmugam said that one thing that Chong Pang constituents appreciated was a person's sincerity and the knowledge that he was trying his best to help them.
"So within a short period, they could see, I was there genuinely to listen to what they have to say, and try and deal with it and help them," he added.
"Once you build up the trust, it's also a hallmark of the background of people in my constituency, if you earn their trust, they are completely with you, they don't change their minds easily. They support you."
Mr Shanmugam said that in the early years, he was at the constituency four to five times a week on weekdays, heading down in the evening after a full day's work at his law firm.
Often, he would have dinner only after 11pm, and get to bed only at 1am or 2am.
"For constituency work, you have to be on the ground. Your physical presence matters," said Mr Shanmugam, who added that he shadowed various MPs for more than a year to learn more about the job before making the decision to enter politics.
"You cannot intellectualise it and say, 'It's okay, they know I care for them'. You have to be there, you have to interact, you have to be present at the dinners, at the wakes, to be present at the functions and you have to work for the people."
More than 20 years on, that work ethic doesn't seem to have changed.
When asked how he manages work-life balance now, Mr Shanmugam replied with a laugh: "I don't manage it.
It's just mostly work. It's the one thing I haven't been able to manage."
He added: "You can't be effective in this job without a very supportive spouse. And I am lucky to have a totally supportive wife."
Mr Shanmugam's wife is a fourth generation Singaporean and a clinical psychologist who often accompanies him to functions.
"We spend time together with her coming with me to many of my events in the constituency," he said.
"She comes with me, and we try to engage each other on the way to my constituency and on the way back, we talk to each other. And on the weekend, I try and set aside two to three hours to spend sometime with her."
What does he think of the criticism of Ms Tin? Mr Shanmugam said he was "bemused".
"I think there is a youth constituency out there, they also have views, and it's good to have a mix of parliamentarians of different age, and the young have a certain idealism, a different way of looking at things, and it's good to have some representatives who are young in Parliament, who will reflect the views of her generation.
"That ought to be heard in Parliament," he said.
"So rather than focusing on a young person as to what he or she alone can do, you look at the whole slate. The slate as a whole is better because it has people from different backgrounds, different ages - adds to diversity."