This is how I made gil for a big house.

[Reinha]

also it's better to put in 4-5 different types of junk/cheap materia because that increases the odds of getting red/crafter/gatherer materia.
please note that the outcome will always be different from the materia you used...

This is correct. I'd like to add that one should also only use materia of the same grade (I-VI) because the result will usually be the lowest grade that was used (sometimes a higher grade procs).

There are 24 types of materia. If 5 different types are used in the transmutation, that leaves 19 possible materia types as a result. The chance to get a specific type of materia is therefore 1/19. Multiply this with the price of each materia on your server and add up those values to estimate your total winnings.

Example from my server:

Dex VI, Int VI, Mind VI, Pie VI and Quickarm VI are 20k gil each. Total cost is 100k gil, so we want to win more than that on average from transmutations. The best resulting grade VI materias and their prices are listed below.

Savage Aim Materia: 130k
Savage Might Materia: 100k
Heaven's Eye Materia: 120k
Craftsman's Command Materia: 714k
Craftsman's Cunning Materia: 660k
Craftsman's Competence Materia: 650k
Gatherer's Guerdon Materia: 370k
Gatherer's Guile Materia: 410k

Multiply the prices by 1/19=0.05263 and add them up.

0.05263 * (130k+100k+120k+714k+660k+650k+370k+410k) = 166k

From these expensive materia types alone one would gain (on average!) 166k for every 100k worth one puts in the transmutation. The small amount of gil from the cheaper types of materia is extra on top of the 166k. If you want to be precise you can do the calculation using all 19 types of materia you may receive from the transmutation.