Setting up an email server used to be a bit easier, back when all you had was plain old SMTP. Programs like Postfix may have been hard to configure, but the actual concept of email wasn't that complicated.

Now instead of plain old SMTP, you not have encrypted SMTP, so you need certificates. Next, you have SPF records to say which IPs are allowed to send email on a domains behalf. Then you have DKIM, which signs outgoing emails, and thus anything sending email on your behalf needs to have this configured in DNS (you can have multiple keys so you don't need to share keys, but it still needs to be setup in DNS). Then you have DMARC to tell other servers how to enforce your SPF/DKIM settings, and where to send delivery reports so you can kind of sort of monitor what's going on on remote servers.

Then you have blacklists, where if anyone on your netblock misbehaves, you'll get blacklisted too and then your emails will start to get rejected.

You can get this all setup and configured; I have in the past. But even if you get it all setup properly for incoming email and having outgoing mail authenticated and whatnot, you'll still get bit by the blacklists. You can get around that by relaying all your outgoing email through another provider like SendGrid.

And after you get all that setup, what exactly do you gain out of it? The satisfaction of running your own email server? It's not exactly any more secure either; depending on who you're trying to keep your data from, if it's a government entity they can just go to the VPS provider and take a snapshot of the running VM, and you'd be none the wiser. Even if you encrypt data on-disk, the snapshot will have the encryption key in memory, and they'll be able to recover the data.

Therefore, I've found it worth it just to use a third-party provider. Now, I wouldn't recommend major providers like Google or Microsoft, but smaller ones like ProtonMail, FastMail (my current provider), Kolab Now, etc.; i.e. providers that are in the business of providing email service, not ones that are trying to get ads in front of your eyeballs.

That said, I'd also recommend going with an email provider that allows you to upload custom Sieve filters, so that you can take your email filters with you when you inevitably change email providers. Both ProtonMail and FastMail support this. I personally have hundreds of rules in my custom Sieve filter, filtering mailing lists, bills, etc., into their appropriate places. There's no way I'd be able to replicate these rules in something like Google's interface; it would probably take me days if I tried, and even if I did there'd be no way to replicate some of the more complicated rules I have.