If you enjoy jumping in cold water, I think that’s a sign you’re in robust health. Weak, fatigued, and ill people don’t generally get any pleasure out of that kind of thing.
No doubt the cold is a stress, but stress can be healthy as long as you’re healthy and you don’t push it too far. For sick people, cold showers and the polar bear swim seem like a really bad idea.