That means less money for all of us, while the prices of luxury assets are inflated. Houses are considered in that pool because they are investment assets, hence house prices rise while we all get less wealthy.

Actually it means it can mean that there is less available liquidity in the system if the very rich use it to bid up the price of assets instead of using it to invest in economic stimulative activities like hiring people to do work.