okay after reading this i have to say I agree that the idea as it is now is a bit clunky, but you have the groundwork for a nice idea. Now i wouldn't say you need a major overhaul but this does need a lot more work to become a balanced job in the games current meta.
The first thing I will say that I think you did an awesome job doing is crafting the abilities in a listed and stylized manor, I can't even do that yet since I don't have the right programs... yet. Though I would suggest making the RDM symbol their iconic hat over a sword with flames, seems more fitting.
Secondly, the main thing to provide balance to the job is to keep things simple. Though, Yoshi-P said that he feels the HW rotations for each job are hard, they are quite simple to understand and execute, the only flaws are usually player error do to lag, time constraints, or controller vs Keyboard. DPS's and Tanks usually have a set rotation, or a standard one, that is flexible when the moment hits. Right now I see no "standard" rotation or excess abilities to be used as OGCD moves. Wanting the RDM to be primarily a spellweaver is fine but you have to make the skills/spells used have a main purpose. Sadly En-spells and debilitation abilities serve no purpose in this game in its current form. And, while I understand your reason for adding the En-spells the scaling with Int is usually build into a job, just look at DRK, a decent portion of their skills are spells. Also having 6 spells/skills that are basically the same thing is rather pointless, condensing them into one "En-spell" or ability, sounds like a better idea.
As for the combos you tries to create, having a "unique" branch as you currently have it seems okay, but i would recommend creating it similar to the mudra system where each finishing move has its own additional effect; adding too many effects can become confusing, especially if they are tied to main weapon skills. Plus you need to create a reason for the RDM to switch between using Magic or Melee attacks, push either a little too far in one direction and people will just spam that one set of skills, especially if they can augment them in other ways. The best suggestion I can give it is creating a job mechanic that encourages player to swap between the two like BLM does with Astral Fire and Umbral Ice. having a stacking mechanic that boosts the current arts power then allows for the use of a certain skill of the other art would encourage swapping. Start with a melee stance which increases str by 10% per stack, and once you gain, maybe 3 stacks, lets say a combo finish creates a stack, you can use a massive spell only available when you have those stacks, similar to warrior. Then once you use this spell it places you in mage stance, and the process repeats, build stacks by casting spells, then may chain or not, then once you have 3 again you use another skill and it places you in melee stance again. (this is my idea but feel free to use it in yours as well)
Hopefully everything that has been said encourages you to create the best RDM you can make, happy job-crafting, XD