Then favoritism to the "established guilds" and ignoring the newer players, oppression against the majority to say only the castle owning guilds matter. (and, what if a guild just happen to successfully steal a castle last minute in one particular woe, and that suddenly "qualify" them to be a decision maker?)
My POV: It's way too hard in a lose-lose situation to take back a castle unless someone comes up with a good proposal on how to do it. What we can do, is we make darn sure of ourselves never to add a new castle for a *LONG* time.