For me, "Caring Is Creepy" probably has more to do with Garden State than with the Shins' album Oh, Inverted World.
Sure, I've listened to the album over and over through the years, but I don't think it really became a mainstay in my library until 2004, when the film's soundtrack was released.
Perhaps it's the whistling intro. Maybe it's the heavy reverb. Whichever your preference, my guess is I'm not alone having this track among one of my favorites.