People worrying about money in politics are overemphasizing the instrumentality of money, rather than the underlying structural incentives that tend toward special interests dominating complex issues.
If money were removed from the equation, the concentrations of power, motivations to influence it, and asymmetries in those motivations will all continue to exist; political influence will just be exercised through other means, and some particular faction or class of people will still be at the sweet spot of having the greatest competence and the greatest motivation to apply those means.
A lot of people are reluctant to acknowledge it, but this is inherent to politics itself, and there is no straightforward way to fix it, other than to limit the reach of political power.
It certainly would be naive to think that simply “getting money out of politics” would “fix” things but it is still a desirable goal for anyone who, themselves, does not currently have extremely large quantities of money.