I've only ever had this problem in new tanks, as time goes on their numbers decrease. I guess the increased bacteria helps decrease the floating food so the dusters have nothing to eat ?
Arrow crabs will work sometimes, I honestly use a pair of plastic forcepts. As far as whether they will naturally die off i wouldn't bet on it, unless you run an ultra low nutrient system. Are these the colonial dusters or the rock boring type?