As you know, I've been swapping sets of summer and winter wheels for years now and it's worked well for me... It's given me that extra confidence that if I drive carefully in a snowstorm, I've got the right tires to back me up. Maybe I'm extra cautious because about 25 years ago I was driving on snowy/icy roads. I checked my braking ability on a flat, straight bit with no traffic coming and some wide shoulders. All was good. But then I went over a rise and was coming down a north-facing, mild decline. Despite my attempts to steer into the skid and despite lots of snowy parking lot practice, I couldn't get my front-wheel-drive Honda under control as my wife and I skidded sidewise, off the road, and into a telephone pole. It was a slow-speed crash at less than 10 mph, I'm sure. Wiped out the driver's side door but didn't do much more. Except, I think, that it resulted in some permanent damage to my wife's confidence in my driving abilities.
At that time, I wasn't up to speed on tires as I am now. I don't know what tires they were, but they were likely some sort of all seasons.
So I think we can only go by what the tire manufacturers and reviewers tell us: that summer tires, driven below 40 degrees Fahrenheit lose their grip and, furthermore, that the tires themselves can be damaged as the rubber becomes hard and brittle. All seasons are better, but they're a bit of a compromise.
If you're occasionally caught in bad circumstances, I think you can mitigate the problem by driving super cautiously, as if you could lose grip at any second.
Having two wheel sets is a pain to store away and a bother, perhaps, to switch out. It's also an extra expense to get a second set of rims. (The extra rubber is no extra cost providing you keep the car long enough as one set isn't being worn down when the other is on the car and vice versa.) However, with practice, a jack, a torque wrench, and a few other basic tools, you can swap out the wheels in about an hour and know you're doing all you can for your safety, that of your family, and that of those you share the road with.
Bummer of an answer, eh?