All it will take is a GM checking transaction logs between two people. Especially if you see something like "Bone chip Bisasdfasdf -> Matata 10,000,000 G". 'Bis' = Bismarck server, for example.
Because the easiest way to suspect a player for participating in possible RMT activities is via monetary and trade transaction logs, GMs and the STF will flag those players and investigate them. But, remember: It has to go from RMT player to legitimate player or legitimate player to RMT player. If they find that you were trading large amounts of Gil to/from an RMT player within a day or regularly and repeatedly, you will possibly be banned permanently. And, they can check IP addresses of each player. It would make sense since Square-Enix has banned IP ranges in FFXI. I am sure Square-Enix has a list of IP addresses belonging to known RMT companies, possibly known proxy and VPN servers used by them, and possibly country of origin that have high RMT activity like China.