Running your own mail server has a reputation for being a masochistic hobby. "Just use Google Workspace," people say. And honestly, for most businesses, they’re right. But there’s a real case for self-hosting: full control over your data, no per-seat costs, custom filtering rules, and the satisfaction of not feeding another $6/month per mailbox to the SaaS machine.
The problem is that most guides are either five years out of date or stop right before the part that actually matters — getting your mail delivered. SPF alone won’t cut it anymore. Gmail and Microsoft 365 have become increasingly aggressive gatekeepers, and your fresh VPS IP is one hop away from someone’s blocklist.
This guide covers the full stack: Postfix as your MTA, Dovecot for IMAP, Rspamd for spam filtering and DKIM signing, and everything you need configured at the DNS level to not end up in junk. We’ll run this on Debian 12 (Bookworm), but the configs translate to Ubuntu 22.04/24.04 with no real changes.
What You Need Before You Start
- A VPS with a dedicated IPv4 (and ideally IPv6). Shared IPs from certain cheap providers are pre-burned on blocklists. Check your IP at mxtoolbox.com/blacklists.aspx before you do anything else.
- A domain you control with access to DNS records.
- A reverse DNS (PTR) record for your IP pointing to your mail hostname (
mail.yourdomain.com). This is set at your hosting provider’s panel, not in your DNS registrar. No PTR = instant rejection from major providers. - Port 25 must be open. Some cloud providers (AWS, GCP, Azure) block outbound port 25 by default and require a support request to unblock it. Check this first.
Our target hostname throughout this guide: mail.yourdomain.com
DNS Records: Do This First
Mail delivery lives or dies by DNS. Get this wrong and nothing else matters.
MX record — tells other servers where to deliver mail for your domain:
yourdomain.com. MX 10 mail.yourdomain.com.
A record — your mail server’s IP:
mail.yourdomain.com. A YOUR_SERVER_IP
PTR record — set at your hosting provider. Must resolve YOUR_SERVER_IP → mail.yourdomain.com.
SPF record — authorizes your server to send mail for your domain:
yourdomain.com. TXT "v=spf1 mx ~all"
Use ~all (softfail) instead of -all (hardfail) initially. Once you’ve verified everything works, tighten to -all.
DMARC record — we’ll add the DKIM selector after setup, but create a relaxed DMARC policy now:
_dmarc.yourdomain.com. TXT "v=DMARC1; p=none; rua=mailto:[email protected]"
Start with p=none (monitor only). Escalate to quarantine then reject after a few weeks of clean reports.
DKIM record — we’ll add this after Rspamd generates the keypair. Hold this slot in your DNS editor.
TLS Certificate with Certbot
Everything runs over TLS. Install Certbot and get a cert for your mail hostname:
apt install certbot -y
certbot certonly --standalone \
--agree-tos \
--email [email protected] \
-d mail.yourdomain.com
Certs land in /etc/letsencrypt/live/mail.yourdomain.com/. Set up auto-renewal:
systemctl enable --now certbot.timer
Gotcha: Postfix and Dovecot run as their own users and can’t read the privkey directly. Fix with a deploy hook that copies certs after renewal. Create
/etc/letsencrypt/renewal-hooks/deploy/copy-certs.sh:
#!/bin/bash
DOMAIN="mail.yourdomain.com"
DEST="/etc/ssl/mail"
mkdir -p $DEST
cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem $DEST/
cp /etc/letsencrypt/live/$DOMAIN/privkey.pem $DEST/
chmod 640 $DEST/privkey.pem
chown root:ssl-cert $DEST/privkey.pem
# Reload services
postfix reload
dovecot reload
chmod +x /etc/letsencrypt/renewal-hooks/deploy/copy-certs.sh
bash /etc/letsencrypt/renewal-hooks/deploy/copy-certs.sh # run once now
Installing the Stack
apt update && apt install -y \
postfix postfix-mysql \
dovecot-core dovecot-imapd dovecot-lmtpd \
rspamd redis-server \
opendkim opendkim-tools
During Postfix installation, select Internet Site and enter mail.yourdomain.com as the mail name.
Postfix Configuration
Postfix config lives in /etc/postfix/. The two files that matter most are main.cf and master.cf.
/etc/postfix/main.cf
Replace the default with this:
# Core identity
myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
# Listen on all interfaces
inet_interfaces = all
inet_protocols = all
# What we consider "local" — don't add $mydomain here if it's handled virtually
mydestination = localhost
# Relay config — we handle mail for virtual domains, not relay
mynetworks = 127.0.0.0/8 [::1]/128
# Virtual mailbox setup (we'll use Dovecot LMTP for delivery)
virtual_mailbox_domains = yourdomain.com
virtual_transport = lmtp:unix:private/dovecot-lmtp
# Virtual alias and mailbox maps — flat file for simplicity
virtual_alias_maps = hash:/etc/postfix/virtual_aliases
virtual_mailbox_maps = hash:/etc/postfix/virtual_mailboxes
# TLS — inbound (receiving)
smtpd_tls_cert_file = /etc/ssl/mail/fullchain.pem
smtpd_tls_key_file = /etc/ssl/mail/privkey.pem
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
# TLS — outbound (sending)
smtp_tls_security_level = may
smtp_tls_cert_file = /etc/ssl/mail/fullchain.pem
smtp_tls_key_file = /etc/ssl/mail/privkey.pem
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
# Spam filtering via Rspamd milter
smtpd_milters = unix:/run/rspamd/milter.sock
non_smtpd_milters = unix:/run/rspamd/milter.sock
milter_default_action = accept
milter_protocol = 6
# Restrictions on incoming mail
smtpd_helo_required = yes
smtpd_recipient_restrictions =
permit_mynetworks,
reject_unauth_destination,
reject_invalid_helo_hostname,
reject_non_fqdn_helo_hostname,
reject_unknown_sender_domain
# Message size limit: 50MB
message_size_limit = 52428800
# Mailbox size limit: 1GB per user
mailbox_size_limit = 1073741824
Add virtual mailbox maps
# /etc/postfix/virtual_mailboxes — one entry per mailbox
echo "[email protected] yourdomain.com/nikita/" > /etc/postfix/virtual_mailboxes
postmap /etc/postfix/virtual_mailboxes
# /etc/postfix/virtual_aliases — optional aliases
echo "[email protected] [email protected]" > /etc/postfix/virtual_aliases
postmap /etc/postfix/virtual_aliases
Enable submission port (587) in /etc/postfix/master.cf
Uncomment or add the submission entry:
submission inet n - y - - smtpd
-o syslog_name=postfix/submission
-o smtpd_tls_security_level=encrypt
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
-o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject
Port 465 (SMTPS/implicit TLS) — also useful:
smtps inet n - y - - smtpd
-o syslog_name=postfix/smtps
-o smtpd_tls_wrappermode=yes
-o smtpd_sasl_auth_enable=yes
-o smtpd_sasl_type=dovecot
-o smtpd_sasl_path=private/auth
-o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
Dovecot Configuration
Dovecot handles IMAP access and local delivery via LMTP. The default config is split across a bunch of files under /etc/dovecot/conf.d/. We’ll override the key ones.
/etc/dovecot/dovecot.conf
protocols = imap lmtp
/etc/dovecot/conf.d/10-auth.conf
disable_plaintext_auth = yes
auth_mechanisms = plain login
!include auth-passwdfile.conf.ext
/etc/dovecot/conf.d/auth-passwdfile.conf.ext
passdb {
driver = passwd-file
args = scheme=ARGON2ID username_format=%u /etc/dovecot/users
}
userdb {
driver = passwd-file
args = username_format=%u /etc/dovecot/users
default_fields = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
}
/etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:/var/mail/vhosts/%d/%n/Maildir
mail_uid = vmail
mail_gid = vmail
/etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_cert = </etc/ssl/mail/fullchain.pem
ssl_key = </etc/ssl/mail/privkey.pem
ssl_min_protocol = TLSv1.2
ssl_cipher_list = ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384
ssl_prefer_server_ciphers = yes
/etc/dovecot/conf.d/10-master.conf — LMTP and auth sockets
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
mode = 0600
user = postfix
group = postfix
}
}
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0666
user = postfix
group = postfix
}
unix_listener auth-userdb {
mode = 0600
user = vmail
}
}
Create the vmail user and directories
groupadd -g 5000 vmail
useradd -g vmail -u 5000 vmail -d /var/mail/vhosts -m
mkdir -p /var/mail/vhosts/yourdomain.com/nikita/Maildir
chown -R vmail:vmail /var/mail/vhosts
Create user credentials
Use Dovecot’s doveadm to hash the password:
doveadm pw -s ARGON2ID -p 'your-strong-password'
Copy the output hash and add to /etc/dovecot/users:
[email protected]:{ARGON2ID}$argon2id$v=19$...
Rspamd: Spam Filtering and DKIM Signing
Rspamd is what separates a modern mail server from a 2015 tutorial. It’s fast, modular, and handles DKIM signing natively — no need for a separate OpenDKIM daemon.
Configure Rspamd milter socket
Create /etc/rspamd/local.d/worker-proxy.inc:
milter = yes;
timeout = 120s;
upstream "local" {
default = yes;
self_scan = yes;
}
bind_socket = "/run/rspamd/milter.sock mode=0660 owner=_rspamd group=postfix";
Enable Redis for Rspamd statistics
systemctl enable --now redis-server
Create /etc/rspamd/local.d/redis.conf:
servers = "127.0.0.1";
Configure DKIM signing
Generate a keypair:
mkdir -p /etc/rspamd/dkim
rspamadm dkim_keygen -s mail2026 -d yourdomain.com \
-k /etc/rspamd/dkim/yourdomain.com.mail2026.key \
> /etc/rspamd/dkim/yourdomain.com.mail2026.txt
chown -R _rspamd:_rspamd /etc/rspamd/dkim
chmod 640 /etc/rspamd/dkim/*.key
The .txt file contains your DNS TXT record. Add it to DNS:
cat /etc/rspamd/dkim/yourdomain.com.mail2026.txt
Output will look like:
mail2026._domainkey.yourdomain.com IN TXT "v=DKIM1; k=rsa; p=MIIBIjAN..."
Create /etc/rspamd/local.d/dkim_signing.conf:
enabled = true;
sign_authenticated = true;
sign_local = true;
use_domain = "header";
use_redis = false;
use_esld = true;
path = "/etc/rspamd/dkim/$domain.$selector.key";
selector = "mail2026";
Rspamd actions config
Create /etc/rspamd/local.d/actions.conf:
reject = 15; # hard reject, clear spam
add_header = 6; # add X-Spam headers
greylist = 4; # soft delay for unknown senders
Enable and start everything
systemctl enable --now rspamd
systemctl enable --now postfix
systemctl enable --now dovecot
Testing Your Setup
Check logs in real time:
journalctl -f -u postfix
journalctl -f -u dovecot
journalctl -f -u rspamd
Send a test email to an external address:
echo "Test from self-hosted" | mail -s "Test" [email protected]
Watch the Postfix log for status=sent. If you see status=deferred or connection refused, check your PTR record and port 25 accessibility first.
Check your score at mail-tester.com: Send an email to the address they provide. A score of 9+/10 means you’re clean. Below 7 and something is misconfigured.
Verify DKIM, SPF, DMARC alignment at mxtoolbox.com/emailhealth.
Test IMAP with openssl:
openssl s_client -connect mail.yourdomain.com:993 -quiet
# Should show the Dovecot IMAP banner
Gotchas That Will Burn You
IP reputation is the real boss. A fresh IP from a datacenter is inherently suspicious. Services like Postmark, Mailgun, and SendGrid have been warming their IPs for years. Expect some false positives from aggressive recipients in your first weeks. Send low volumes and monitor your DMARC reports.
mydestination vs virtual domains. If you add yourdomain.com to both mydestination and virtual_mailbox_domains, Postfix will try to deliver locally using system accounts, not virtual mailboxes. Keep mydestination = localhost only.
Rspamd milter socket permissions. If Postfix can’t connect to the Rspamd socket, it will accept mail without filtering (because milter_default_action = accept). Verify the socket exists and that both _rspamd and postfix can access it. Check /run/rspamd/milter.sock ownership.
Greylisting bites newsletter providers. Rspamd’s greylisting will delay first-time senders. Legitimate mail servers retry; sketchy ones don’t. But some bulk senders rotate IPs per message, so greylisting never passes them. Tune accordingly or whitelist specific IPs.
DMARC aggregate reports need a mailbox. The rua=mailto:[email protected] address must exist or you’ll get NDRs. Create an alias or a real mailbox for it.
SMTP smuggling. In late 2023, a class of vulnerabilities was discovered in how MTAs handle bare <CR><LF> sequences. Postfix 3.8.4+ has mitigations. Debian 12 ships a patched version, but always run postfix -d to check your version and postconf smtp_body_checks to understand your exposure.
Production Hardening
Fail2ban to block brute-force auth attempts:
apt install fail2ban -y
# Enable dovecot and postfix jails in /etc/fail2ban/jail.local
Postscreen as a first-line bot filter — add to master.cf to run before smtpd and reject obvious junk before it hits your filtering stack.
Rspamd Web UI — Rspamd ships a web dashboard on port 11334. Don’t expose it publicly. Use an SSH tunnel: ssh -L 11334:localhost:11334 [email protected]. Default password is empty; set one in /etc/rspamd/local.d/worker-controller.inc.
Backup MX — a single server has no redundancy. For personal use, services like mailspons.com provide cheap backup MX that hold mail and retry when your server comes back up. Worth the $5/year.
Monitor your blacklist status. Set up a cron job or a service like HetrixTools to alert you if your IP lands on a blocklist. You want to know immediately, not after a client calls you.
IMAP quota — configure per-user quotas in Dovecot to prevent one user from filling your disk. Add to 10-mail.conf:
mail_plugins = $mail_plugins quota
And configure /etc/dovecot/conf.d/90-quota.conf with appropriate limits.
Running your own mail server in 2026 isn’t for everyone. The ongoing maintenance burden is real — you need to watch your IP reputation, keep Postfix and Dovecot patched, and occasionally fight with delivery issues for specific recipients who use overzealous spam policies. But if you’re self-hosting other services already and want email under your own roof, this stack is genuinely solid. Rspamd in particular is a step change from the old SpamAssassin days — the Lua scripting interface, the Redis-backed Bayesian filters, and the built-in DKIM signer make it a joy to run.
The config above will get you to a working, deliverable server. The next step is to set up DMARC reporting review (parsedmarc is excellent for this), automate your certificate renewal tests, and start tuning Rspamd’s classifier with real mail samples after a few weeks of traffic. Mail is one of those things where the first 80% takes a weekend and the last 20% takes years of iteration — and that’s what makes it interesting.