You'll not lose any gained attribute points ever. If you desire so you can never spend any of them, although there is little to no reason to hoard them.