All I know so far is if you trade in 5 of any materia of the same tier (i.e. 5 Fire IV materia), you have a chance of receiving any materia from the same tier (i.e. any tier IV materia). I also traded in 1 IV materia, 3 III, and 1 I, and got a mind IV, but I haven't tested much beyond that. If nothing else, it is system to transmute shitty elemental materia into something potentially worthwhile.