First, having uniquely powerful materia would mess with "actual gear progression". Moreover, I'd keep things simple where possible; avoid overcomplication. I think it's fine, for instance, if progressively difficult nightmare runs give progressively higher chances for (or move you nearer to your next weekly guaranteed drop of) something, say, just 5 ilvls above the pre-tier BiS and may hit 5 ilvls short of post-tier BiS by the time most raiders are farming the last floor of that raid. Mostly, you just want to avoid it being a way for raiders to get ahead on their mains or it providing so much random but rare junk that it badly devalues other means of, say, acquiring non-craftable vanity items. But you don't need to offer power in some totally unique way. You want the gameplay to be the point, and the rewards merely (or just short of) commensurate to the gear and skill required.
Speaking of keeping things simple, the basic idea of a dungeon getting harder as you go already seems pretty great for midcore side-content, especially if you plan to shake it up further with little added challenges. I feel your greatest strength in that concept is that you can use the Nightmare stacks to naturally fit reward to difficulty. For instance, maybe you could have a bar you build to acquire additional treasure chests and each boss and mob type would give a certain amount multiplied by your level of Nightmare at that time (with nightmare gradually increasing or fading with mob count, etc., so it doesn't oblige cheese strats like pulling the whole room before finishing off the first pack just to get more reward from them). In this way, even against normal, relatively weak mobs, less skilled/geared parties could pull smaller and be challenged while more skilled/geared parties could just pull larger, and each would be rewarded proportionately: the greater the nominal danger, the greater the reward.
For more core content, especially if the gap between early- and late-dungeon difficulty is fairly wide, I suspect having only a small span of a dungeon that actually suits the difficulty you want -- before that being too easy and thereafter like testing the hardness of one's skull against concrete -- could be irritating. However, having some initial difficulty modifiers available, such as to just increase all nightmare (and thus difficulty and reward) from the very start, could easily fix that. That be your first step towards "M+" territory, but I don't see why that should matter; what works, works. We needn't avoid whole spans of content over a stigmatized potential (and highly vague) affiliation.