Instructions are in the bottom of nvm --help - "to remove, delete, or uninstall nvm - just remove the $NVM_DIR folder (usually ~/.nvm)"
You can also remove the two relevant lines from your profile file, but they'll be a noop when $NVM_DIR doesn't exist.
brew uninstall nvmbrew doctorbrew cleanup