I wouldn't advise the manual esm selection simply because it's too easy to lose track of the mods you're using. Some don't have an esm/esp at all, or some mod the same thing. The mod manager will ensure that you can install/uninstall a mod without breaking the others (IE: if you install a mod that overwrites some file of another mod, if you disinstall the first mod, the manager will actually reinstate the files you previously overwritten, ensuring that the first mod will work).
It's a very sophisticated piece of software and makes your life a LOT easier without drawbacks.
"need" no, but the right selection of mods will make the game so much better that it's really not worth it to run without.
