Any time you are attacked too many times because you have been successful, you begin to realize a good many folks are not 'for' you. I think it must have been tiring. The feeling that you are loved is often an illusion.
I've always felt that Charlie looked pretty good for his age until the mid to late 1960s when suddenly turned into an old man. I think he had a couple of mini strokes around that time which seemed to age him.