Fail2ban and Port Knocking should address most of your needs.
Changing your SSH port and only allowing Key-based authentication are also recommended.
It can be argued that you may reach a point of diminishing returns in adding additional security measures, but then again, it's up to you to decide when you're "secure enough".
It's also a good idea to disallow root login.