The gimmick here is the star, Paul Dale (real name: Dale Paullin from Iowa and still alive, thanks), who was, in the old politically-incorrect parlance, a midget, but is now called (in the phrase I still see as demeaning) "a little person."* It is (as the poster tag puts it) "the amazing story of a man forced to live in a child's world." Well, not in a child's world per se. But, in an adult world perceiving him as a child and all that that entails, in ways both exploitative and marginalizing. And in an ironic way, the movie shows that his world isn't small, but people's perceptions of him (in their own small minds) are.
When most kids are going through "growth spurts," Harry Musk is left behind. He keeps anticipating when he's going to catch up to everybody else but it never happens. His father (Will Geer) is a simple farmer and shelters Harry from the vagaries of the "normal" world that entails the kids at school beating him up for being a "shrimp," his own sister outpacing him in the growth department (and bitching that she can't bring friends around because Harry's such a freak). The one friend Harry has is Janie who becomes his friend at 8, and grows up and out of his life. Harry is crushed.
Because of the sister thing and because Harry isn't much help around the farm, when he turns 21, he decides to go out and make his way in the world by offering his services as a "little person" to a promoter. The guy wants him for carnies and things like that and for Harry it's a chance to be useful and... somebody wants him. But when the guy turns out to be an exploitative creep, Harry runs away. He makes his way to a nearby town, where a gal appears interested in him...but ultimately not in that way. Then, he starts working as a shoeshine boy, and for awhile he seems content.
Perhaps the best term to use for Harry Musk is "disenfranchised." Nobody sees him for who he is, but for what he is...and what most see is how they can con him or make a buck from him. It's only in the film's third act that he finds a full measure of happiness and that is in a segregated community of "outsiders."
This one showed up on the cusp of the '50's, where folks were still reeling from a devastating war and pushing hard to return to "normal." But, "normal" is not the same to everybody. The decade roiled with a fallback to earlier prejudices and tribal suspicions. Now that we didn't have to pull together, it was safe splinter back into groups of "us" and "them." In its own weird way It's a Small World is a plea for tolerance, knowing that the world will come up short and that the only recourse is to fall back into the familiar tribe. The larger world can be a despicable place, but there are pockets of comfort...no matter how small.