for the most part, like corals with like is generally fine - euphilia with euphilia, zoas with zoas, etc. when it comes to mixing, and different types of corals touching, its more exceptions than rules - for example most mushrooms mix in with most zoas ok, but you might have shading/crowding issues; Anthialia and xenia seldom have enough aggressiveness to hurt other corals, but other corals can hurt them. then you have the really aggressive corals, generally of the large polyped stony variety, that can reach out inches or feet to attack and kill neighbors - and you have to take them on a case by case basis.
so to address your examples:
GSP/mushrooms/Kenya trees - GSP and kenya will probably get along with minimal agitation as neither is very aggressive, mushrooms might add in fine depending on the variety, but would be more likely to cause some agitation. since they are all different, its better not to plan on them touching, but since none of them are that aggressive you should have plenty of time to adjust them if you see them irritating each other before any permanent damage is done. as dave said, they may out-compete each other - like weeds in a garden - but they wont generally outright kill each other quickly.
- hammer and frogspawn get along fine, they are the same family - like with like - crowd them together with no issues.
- the euphilia family (hammer/frogspawn/torch/octospawn) are pretty aggressive, and will likely burn the crap out of a GSP/mushrooms/Kenya trees/most any coral if they touch. they may even reach out several inches to do so.
- anemones pretty much sting everything they touch. its more a question of "will it kill it or just irritate it ?". best to keep you coral out of its way if possible.
