The issue is if you are a) targeted, and b)involved in multiple breaches. If they can get the pattern, they potentially get everything.
Is it worth it? That depends. Are you willing to risk it NOT being worth it to a random guy in Africa earning a few $ a day?
Even if you don’t use it as a password manager, bitwarden has an excellent pass phrase generator. The only annoyance is when I run into maximum password lengths at times.