Self-Hosted Email with Postfix and Dovecot: Honest Sysadmin's Guide

I spent three days debugging a silent email blackhole. SPF, DKIM, DMARC all passing. Logs clean. Yet every message to Outlook.com vanished. The culprit? My PTR record pointed to mail.example.com but the reverse DNS zone for my IP had a trailing space in the hostname. A single invisible character cost me a week of reputation with Microsoft’s delivery team.

Self-hosting email is not a weekend project. It’s a career‑long commitment to DNS hygiene, reputation management, and knowing that one misconfiguration can land your IP on every RBL in existence. This guide is for the sysadmin who already runs a mail server and is tired of “just works” tutorials that omit the hard parts. By the end, you will have a production‑hardened Postfix + Dovecot setup—and the war stories to know why every decision matters.

TL;DR

• Self‑hosted email is a ton of work; you must own a dedicated IP with a clean reputation.
• SPF, DKIM, and DMARC are non‑negotiable—skip them and your email will be spam.
• Use Maildir, not mbox. Use virtual users, not system accounts.
• TLS everywhere: require it for inbound and outbound. No exceptions.
• Monitor logs with pflogsumm and automate certificate renewal—your Saturday nights depend on it.

Prerequisites

• A Linux server (Debian 12 or RHEL 9) with root access and a public IP—not a shared residential IP.
• A domain name with DNS control. You’ll add A, MX, PTR, SPF, DKIM, DMARC records.
• Basic familiarity with systemd and firewalld or iptables.
• Open ports: 25 (SMTP), 465 (SMTPS), 587 (submission), 993 (IMAPS), 143 (STARTTLS), 995 (POP3S). Do not open 110 or 143 without TLS forced.

Why Self‑Hosting Email Is (Usually) a Bad Idea — But Here We Are

The True Cost of Control vs. Convenience

The illusion of full control evaporates the first time you realize deliverability is 90% of the problem. Google and Microsoft have spent billions on reputation systems; your $5 VM can’t compete on trust. You trade convenience for a constant fire‑fighting loop: SPF alignment, IP warming, feedback loop registration, and hourly dives into your RBL status.

Comparison: Self‑Hosted vs. Google vs. Exchange

Factor	Self‑Hosted	Google Workspace	Exchange Online
Monthly cost (10 users)	$5‑10 (VPS)	$120+	$120+
Control	Full (you or your pager)	Limited (UI policies)	Moderate (admin tools)
Deliverability	You own the reputation	Baked‑in reputation	Baked‑in reputation
Maintenance effort	4‑8 hours/week	30 min/month	1 hour/month
IP reputation risk	Your entire IP’s neighbors matter	Immutable	Immutable

I’ve run mail servers for 15 years. If you aren’t ready to read RBL listings on a Sunday morning, buy a service. This guide assumes you are that kind of masochist.

Your Email Will End Up in Spam Without This Checklist

Start with SPF, DKIM, and DMARC or don’t start at all. I’ve seen perfectly configured Postfix boxes get flagged because the SPF record used +all or the DKIM selector was missing. These are not optional. They are the foundation of trust in 2026.

What This Guide Covers (and What It Doesn’t)

This guide covers the minimal viable Postfix + Dovecot stack with SASL and TLS, plus the deliverability trinity. It explicitly excludes webmail clients (Roundcube, SnappyMail), antivirus/antispam (Rspamd, Amavis, postfwd)—those war stories come in a separate guide.

Q: Is self‑hosting email worth it in 2025?
Short answer: only if you have a dedicated IP, a clean reputation, and the willingness to maintain it. “Worth it” means you learn DNS, reputation, and protocol internals. If you just want email that works, pay for it.

The Minimal Viable Setup (And Why Anything Less Is a Waste of Time)

Server Requirements: Your $5 VM Is Not Enough

Minimum specs: 2 vCPU, 4GB RAM, dedicated IP with clean reputation. A 1‑core 1GB box will swap under load; a shared IP will get you blacklisted by association. I once ran Postfix on a VPS whose neighbor was a spam botnet. Spamhaus listed the entire /24—my IP was collateral damage. You need a dedicated IP, and you need to check its history before provisioning.

# Open firewall ports (firewalld example)
firewall-cmd --add-service=smtp --permanent
firewall-cmd --add-service=smtps --permanent
firewall-cmd --add-service=submission --permanent
firewall-cmd --add-service=imaps --permanent
firewall-cmd --add-service=pop3s --permanent
firewall-cmd --reload
# Verify
firewall-cmd --list-services

DNS Records: The Foundation You Can’t Skip

A, MX, PTR, SPF, DKIM, DMARC—in that order. Here’s a zone file snippet that proves you care:

; example.com zone
@   IN  A   203.0.113.10
@   IN  MX  10 mail.example.com.
mail    IN  A   203.0.113.10
; SPF
@   IN  TXT "v=spf1 mx ip4:203.0.113.10 -all"
; DKIM (default._domainkey)
default._domainkey  IN  TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4...="
; DMARC
_dmarc  IN  TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc-reports@example.com"

Comparison: Required vs. Nice‑to‑Have Records

Record	Required?	Why
A	Yes	Mail server address
MX	Yes	Route incoming mail
PTR	Yes	Reverse validation
SPF	Yes	Authorize senders
DKIM	Yes	Sign messages
DMARC	Yes	Policy for failure
MTA‑STS	Nice	TLS enforcement
TLS‑RPT	Nice	Failure reports
BIMI	Nice	Brand icon in inbox

Reverse DNS and IP Reputation: The Gatekeepers

PTR must match the hostname used in the HELO/EHLO greeting. If your hostname is mail.example.com, your PTR must be mail.example.com. Request rDNS from your provider—most give you a control panel entry.

# Verify PTR
dig -x 203.0.113.10 +short
# Should output: mail.example.com

If it doesn’t match, fix it before sending a single email.

Postfix Configuration for Real Mail Delivery (Not Just Local Delivery)

main.cf Essentials: What to Set and What to Leave Alone

Don’t touch default values for obscure parameters like smtpd_helo_required unless you understand why a CVE exists. Focus on these:

/etc/postfix/main.cf

myhostname = mail.example.com
myorigin = $myhostname
mydomain = example.com
mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain
mynetworks = 127.0.0.0/8, [::1]/128   # NEVER add 10.0.0.0/8 here
inet_interfaces = all
relay_domains = $mydestination            # only relay for our domain

The mynetworks directive is critical: only localhost and your internal network should be trusted. Adding your VPS private IP is a recipe for open relay.

TLS Encryption for Both Inbound and Outbound (No Excuses)

# TLS for inbound
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_security_level = may   # opportunistic but better than "none"
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# TLS for outbound
smtp_tls_security_level = may
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

I once set smtpd_tls_security_level = encrypt without thinking. It rejected all connections from MX hosts that only supported opportunistic TLS (like some old Exchange servers). Use may and let the client choose.

Restricting Relays and Rejecting Unauthorized Use

The most common misconfiguration: ordering of restrictions. Postfix 3.0+ uses smtpd_relay_restrictions as the first filter. Here’s my production setup:

smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client zen.spamhaus.org, permit

Comparison: smtpd_relay_restrictions vs smtpd_recipient_restrictions

Directive	Purpose	Order
relay_restrictions	Control who can relay (outbound)	First check before recipient_restrictions
recipient_restrictions	Control who can receive (inbound + outbound)	After relay check; additional filtering

Using defer_unauth_destination in relay restrictions buys you a clean reject message. Don’t use reject there—defer gives temporary errors, which are easier to debug.

Dovecot — Painless Authentication and Mailbox Access

Setting Up Mailboxes (Maildir vs. mbox — Don’t Pick mbox)

Maildir is the only sane choice: one file per message, no locking nightmares, easy to backup. mbox puts everything in one file—corrupt it and lose all mail. dbox offers performance but adds complexity.

/etc/dovecot/dovecot.conf

mail_location = maildir:~/Maildir
namespace inbox {
  inbox = yes
}

Comparison: Maildir vs. mbox vs. dbox

Feature	Maildir	mbox	dbox
Corruption risk	Very low	High (single file)	Low
Backup	Per‑message files	One file per folder	Per‑folder files
Performance (IMAP)	Fast	Slow with many messages	Fast
Disk space	More inodes	Space efficient	Moderate

SASL Authentication with Dovecot (Postfix Will Thank You)

Dovecot handles authentication and exposes a Unix socket for Postfix. This avoids running a separate SASL daemon.

/etc/dovecot/conf.d/10-auth.conf

auth_mechanisms = plain login
!include auth-system.conf.ext   # for system users, but we'll switch to virtual

If you use PLAIN without TLS, you deserve the pwnage. Dovecot’s ssl = required will reject any non‑TLS connection.

/etc/dovecot/conf.d/10-ssl.conf

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem

Securing Dovecot with TLS and Authentication Rules

auth_failure_delay = 2 secs    # slows brute force
auth_worker_max_count = 30     # allow enough workers

I once set auth_failure_delay to 10 seconds—legit users got locked out during peak load. Keep it low (2 seconds) and monitor logs instead.

War story: A client’s mail server was being brute‑forced; I added auth_failure_delay 5 and auth_worker_max_count 10. Problem solved, but then admin accounts started timing out because the auth workers were all busy. Watch your logs and adjust.

The Handshake: SASL and Auth Socket Config

Setting Up the Unix Socket for Postfix to Use Dovecot Auth

Dovecot creates a Unix socket that Postfix reads for SASL authentication. Permissions must be tight.

# /etc/dovecot/conf.d/10-master.conf (in section service auth)
unix_listener /var/spool/postfix/private/auth {
  mode = 0660
  user = postfix
  group = postfix
}

Then tell Postfix where to find it:

/etc/postfix/main.cf

smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth   # relative to postfix chroot
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous

Password Storage: Plaintext Is a Sin (Use SHA512‑CRYPT or ARGON2)

/etc/dovecot/conf.d/10-auth.conf

password_scheme = SHA512-CRYPT
passdb {
  driver = passwd-file
  args = /etc/dovecot/users
}

Comparison: Hashes

Scheme	Security	Performance	Use Case
PLAIN	None	Fast	Only for testing
MD5‑CRYPT	Weak	Fast	Legacy
SHA512‑CRYPT	Strong	Moderate	Recommended default
ARGON2	Strongest	Slow	High‑security environments

Generate hashes with doveadm pw -s SHA512-CRYPT.

Virtual Users vs. System Users — Why Virtual Wins for Scale

System users (uid/gid) become unmanageable beyond ~200 mailboxes. Virtual users use a single uid (e.g., vmail) and store account info in a file or database.

# /etc/dovecot/conf.d/10-auth.conf (virtual users)
passdb {
  driver = passwd-file
  args = /etc/dovecot/users
  # each line: user@example.com:{SHA512-CRYPT}hashed_password:uid:gid
}
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/%d/%n
}

War story: A sysadmin created system accounts for 500 mailboxes. /etc/passwd became a nightmare, and UID exhaustion caused Dovecot to refuse connections. Virtual users saved his career.

Deliverability — The Hardest Part of Self‑Hosted Email

SPF, DKIM, and DMARC: The Trinity of Email Authentication

SPF authorizes sending IPs. DKIM cryptographically signs messages. DMARC tells receivers what to do when both fail.

# SPF record (TXT)
v=spf1 mx ip4:203.0.113.10 -all

# Generate DKIM key
opendkim-genkey -D /etc/opendkim/keys/ -d example.com -s default
# Publishes default._domainkey TXT record

# DMARC policy (TXT)
_dmarc.example.com.  IN TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com"

Comparison: Mandatory vs. Optional

Protocol	Mandatory	Notes
SPF	Yes	Prevents spoofing of your domain
DKIM	Yes	Prevents message tampering
DMARC	Yes	Controls failure policy
ARC	Optional	Preserves DKIM across forwarding
MTA‑STS	Optional	Enforces TLS when receiving
TLS‑RPT	Optional	Reports on TLS failures

Setting Up OpenDKIM and Postfix Integration

OpenDKIM signs outgoing mail and verifies incoming.

/etc/opendkim.conf

Domain                  example.com
KeyFile                 /etc/opendkim/keys/default.private
Selector                default
Socket                  inet:8891@localhost

Postfix milters:

/etc/postfix/main.cf

smtpd_milters = inet:localhost:8891
non_smtpd_milters = inet:localhost:8891
milter_default_action = accept

War story: I used the selector mail for years. When I added a second domain with the same selector, Postfix loaded the first key it found—DKIM broke for three days. Always use unique selectors per domain.

Monitoring Blacklists and Heating Up a New IP

Start slowly: throttle outbound to 50 emails/hour per domain for the first week. Use a warm‑up service like Mailwarm or SendGrid’s free tier for initial sending.

# Cron job to check RBLs
0 */6 * * * /usr/bin/dig +short 203.0.113.10.zen.spamhaus.org | \
  grep -q "127.0.0." && echo "Listed on Spamhaus!" | mail -s "RBL Alert" admin@example.com

Q: My email still lands in spam even with SPF/DKIM/DMARC — what now?
Check: 1) PTR matches HELO. 2) IP reputation (use https://multirbl.valli.org). 3) Content—avoid spammy words, include plaintext alternative. 4) Register for feedback loops (Outlook, Yahoo). 5) Use a dedicated IP with a warm sending pattern.

Keeping the Ship Afloat — Logs, Metrics, and Regular Chores

Logging with pflogsumm and Dovecot Stats

Pipe pflogsumm to a weekly report:

# Weekly cron
@weekly pflogsumm /var/log/maillog 2>/dev/null | mail -s "Postfix Weekly Report" admin@example.com

Enable Dovecot stats plugin for more granularity.

Queue Management: Clearing Backlogs Before They Become Emergencies

# See queue
postqueue -p
# Delete all deferred (be careful)
postsuper -d ALL deferred
# Flush queue
postqueue -f

War story: A misconfigured SPF caused a bounce loop—10,000 messages stuck in the deferred queue. postsuper -d ALL cleared it, but I lost legitimate mail. Prune carefully, then fix the root cause.

Certificate Renewal and Security Updates (Automate or Suffer)

/etc/systemd/system/certbot-renew.timer

# Systemd timer for certbot
[Unit]
Description=Certbot renewal timer

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=timers.target

With a post‑hook to reload services:

# Run after renewal
certbot renew --post-hook "systemctl reload postfix dovecot"

If your TLS certificate expires on a Saturday night, you have no one to blame but yourself.

Q: How do I handle a queue that refuses to drain?

1. mailq to see deferred messages.
2. postcat -q [queue-id] to inspect a sample. If they’re all the same recipient domain, that domain is rejecting you—check your logs and stop sending until resolved.
3. Use postsuper -d ALL only if you’re sure. Otherwise, postsuper -r ALL retries immediately.

Common Mistakes and How to Avoid Them

1. Open relay. Forgetting to set mynetworks correctly. The first spammer who finds you will destroy your reputation. Test with dig +short myip.opendns.com @resolver1.opendns.com | nc -w2 yourserver 25.
2. PTR mismatch. HELO hostname doesn’t match PTR record. Most mailbox providers will reject with 550 5.7.1.
3. Using mbox. One corrupted message destroys the whole inbox. Use Maildir.
4. Ignoring DMARC. Without a policy, receivers treat your mail as untrusted. Start with p=none, monitor, then move to p=quarantine.
5. Not monitoring RBLs. You won’t know you’re blacklisted until your mail disappears. Set up alerts.

FAQ

Q: Can I use a shared IP for self‑hosted email?
Technically yes, but you’re at the mercy of your neighbor’s reputation. One spammy neighbor and your IP is listed. Pay for a dedicated IP.

Q: Do I need antivirus scanning on my mail server?
Not strictly for deliverability, but for security. Amavis + ClamAV is the standard stack. That’s a separate guide—don’t skip it on production.

Q: Why is my outgoing email delayed by minutes?
Likely DNS lookup time or a slow RBL check. Check postfix/smtp logs for connection timeouts. Reduce smtp_connection_cache_destinations and consider running a recursive resolver locally.

Q: How often should I rotate DKIM keys?
At least every 6 months. Old keys should be removed from DNS after a 30‑day overlap.

Q: Can I use Let’s Encrypt for my mail server?
Yes. The certificates work perfectly for SMTP TLS and IMAPS. Just automate renewal.

Key Takeaways

Key Takeaways

• Self‑hosting email demands a dedicated IP, proper DNS hygiene, and continuous monitoring. It’s not cheaper—it’s a trade of money for time.
• SPF, DKIM, DMARC are non‑negotiable. Without them, you are spamming.
• Use Postfix + Dovecot with SASL over a Unix socket. Virtual users over system users. Maildir over mbox.
• Automate certificate renewal and RBL checks—your future self will thank you.
• If you can’t dedicate 4 hours a week to maintenance, buy a service. Your mailserver will thank you.