I can see where you're coming from because this is how I thought of it when I became ill.
However I wasn't deconditioned to start with. I swam twice a week 50*25m lengths in 45 minutes, I also did muscle toning classes. I was fitter than most. I bought into this idea, in part, because I was used to being fit. So, as I lost that level of fitness I thought perhaps I was deconditioned, but I was still probably at least as fit as a sedentary person, if not more so.
In hindsight, the danger here is -
PEM can be delayed so you might feel okay doing the exercise but be clobbered 3 days later, but if you haven't learned about PEM you may not make the connection.
It can take several weeks to fully recover, & if you haven't made the connection & are mildly enough affected, you'll go back to the pool and do it again. So you go give yourself another dose of PEM before you've gotten through the first bout.
Rinse and repeat several times and you've got compound PEM - you won't know what's happened & so you drop out temporarily.
Others seem to manage fine & keep it up. They may be even more mild than you, or be suffering from fatigue not chronic fatigue syndrome, or FM. You don't know that though & think you've just caught a bug.
When you're well enough to have another go, you do it all again. Or you decide maybe water exercise isn't for you (maybe you caught a bug at the pool before?) and you try the same thing with some other form of exercise.
So you continually put yourself into a series of compound PEM.
N=1 & all that, but I believe this kind of thinking and logic is what pushed me into severe ME. Long term follow ups are essential with this kind of thing because in the short term things may seem fine but long term there can be a lot of damage done.