Nests make usually one, but sometimes two (that may have been a bug), combinations of anthrons and tritons. (Its risky, but if you find a nest that gives Tritons a good, expensive weapon, like an iconoblast, you may want to leave the nest alone to farm the gun).
Liars can spawn by themselves (if you aggressively smash nests) or they are created when two nests combine. If nests combine, the lair will spawn a combination of the mutations from each nest. Lairs spawn Sirens and Chirons.
Citadels spawn Scyllas. They can spawn by themselves if you crush all lairs immediately, or they are created when two lairs combine, mixing the mutations of each.
As far as nest difficulty, i guess, based on playing, that they have a currency, and different mutations cost different amounts. My evidence for this is that all the nests seem to randomly mutate, but are always lacking in some significant way. Something is always obviously weak. I have never rolled a nest that mutated with no weaknesses. This means that allowing nests to combine is dangerous, because you may get stacked anthrons and stacked tritons spawning from the same place, instead of only one template being stacked.
As far as difficulty in other aspects, it seems linear. You may notice that every diplomatic mission you do is harder than the last. It seems that every time you complete one of these missions, all future missions of the same type increase in difficulty. It seems like story missions and diplomatic missions share a difficulty amplifier.
This is true also of scavenge missions it seems. You can fight way harder enemies on scavenge missions than nests will ever produce until much later in the game. Part of the problem with the game is this isn’t clear until you play too much, and some people understandably don’t have the patience.
For example, a player may believe, reasonably, that the first diplomatic mission of New Jericho should be relatively as difficult as the first diplomatic mission for Anu, but the game says no, this is your third diplomatic mission, and cranks the spawns up a few notches.
edit: This seems to be different since the patch in regards to diplomacy missions at least. This at least was true pre-patch