For me personally the problem is mostly his ape-like face. He’s got no nose, large chimpanzee lips and rounded human teeth. It doesn’t look intimidating at all, I love the tusks and the horns but the face is just… not all that. From the majority of the other complaints I’ve heard the ape-face is the major factor people hate.
Though I’m mostly over it myself, now that we have Eyedol who looks monstrous and amazing Gargos’ shortcomings don’t’ really bother me much anymore.
Plot twist - Gargos’s removed face shell that got torn off in battle eventually made it’s way to the earth and over centuries eventually became known as…The Mask Of The Ancients kappa 
I mean the mask did talk “An Immortal warrior, I do believe ill have good use for you” plus it gave Spinal the ability to literally suck shadow meter from his opponents, kind of similar to how Gargos can with his command grab. The only issue is the picture of it in Spinal’s ending does not look like part of Gargos’s face at all lol
I don’t really think this is the case, but I like coming up with theories haha
Probably wouldn’t surprise me if that was actually true.