First make sure your key login works, in a second terminal, while the first one stays open. Then edit /etc/ssh/sshd_config:
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
On Ubuntu there may be a file in /etc/ssh/sshd_config.d/ that sets PasswordAuthentication yes and overrides you. Check what is in effect with:
sudo sshd -T | grep -i passwordauthentication
Then reload and test from yet another terminal:
sudo systemctl reload ssh
If ssh -o PubkeyAuthentication=no user@server gets refused, you're done.