Im writing this post if someone come across same situation to have Postfix (SNI Support) or Version >= 3.4.

I tasked to  troubleshoot and fix the existing postfix setup and have users to send/receive email's through client software within short period of time.

When i tookover this task, i thought it could be intermediate issue due to port or auth level problems. but i never expected  this going to drag me for 2 days to complete the work.  Lets jump into this and you will learn realistic problems.

The existing setup was working well with RC(Roundcube - Webportal) but not with email clients.

Issue : Completely misconfigured with different instructions that messed the entire setup,  not satisfying the SNI requirements  and not working with email clients.

Before jumping into this, have a tip learned from internet to share.


TIPS : Understand Email Types

First, you need to understand which type of email you are going to send. For the purpose of this article, I will divide email messages into 6 categories:

  • Personal email
  • Transactional email
  • Marketing email/newsletter
  • Group discussion email
  • Cold email
  • Spam

Personal emails are usually sent from a person to his/her friends, family members, co-workers snd so on. There is often only one recipient.

Transactional emails are sent from websites or web applications to their users. For example, a visitor receives an email after creating an account at a website, or request resetting the password. There is only one recipient per email.

Marketing email/newsletter is sent by website owners to their subscribers to inform about new blog post, webinar, or promote products and services. There are many recipients per email. A subscriber can only reply to the list owner, but can not send email to other subscribers.

Group discussion emails are commonly seen in open-source software development. For instance, the Linux kernel mailing list allows its subscribers to discuss Linux kernel development. A subscriber can send an email to all other subscribers in the mailing list.

Cold email and spam are both unsolicited emails. The difference is that cold emails are often targeted at a specific group of people to sell a service and the sender will provide real service if the recipient responds. Spam are sent to random people and they are often scams.

Email Deliverability Factors

There are mainly 3 factors mailbox providers look into when they decide if your email is spam or not.

  • Email standard compliance
  • IP address reputation
  • Domain name reputation

We discuss about delivery factors in my another blog as its big subject.


Lets jump back to our goal.

Here i'm going to cover and approach the plan of install and migrate the old mails boxes with fresh setup.

Postfix

Postfix is a Mail Transfer Agent(MTA) for routing and delivering electronic mail(email). Dovecot is a secure IMAP, LMTP and POP3 Mail Delivery Agent(MDA). These two open-source applications work well with Roundcube. In this guide, you'll install Postfix, Dovecot, and Roundcube on Ubuntu 18.04. This guide uses the domain ksociety.com and the server name mail.ksociety.com

Prerequisites

Before you begin, make sure you have the following:

  • An Ubuntu 18.04 server configured with a Fully Qualified Domain Name (FQDN).
  • A non-root user with sudo privileges.
  • A LAMP stack with an SSL certificate installed
  • I used a famous Let's Encrypt certificate.
  • Verify the required outbound port status and its not blocked.
$ sudo ufw allow 80,443,587,465,143,993/tcp  110,995/tcp 

1: Installation Postfix

1, Install & Configure Postfix

SSH to your server and install the Postfix server by running the commands below.

$ sudo apt update -y
$ sudo apt upgrade -y

Note : by default ubuntu 18 installs postfix version 3.3, where SNI is not supported,  

I stuck over here and spend many hours to have postfix 3.4, Tried several alternate options but none went with successful completion. After many hours spending over the research.  Here is the solution to have postfix 3.4 over ubuntu 18 server. Use this only if you required SNI Support else you can ignore below two commands.

$ sudo add-apt-repository ppa:carsten-uppenbrink-net/postfix 
$ sudo apt-get update

Note : Hatsoff to carsten to having the PPA repo. for more details about the build/repo

$ sudo apt install -y postfix postfix-mysql

You'll get the Postfix configuration screen, use  TAB and ENTER to continue.

A , select Internet Site

B, Enter the system mail name, which is your domain name. For instance, the server name is mail.ksociety.com, so you'll enter ksociety.com here.

Thats all postfix install is over, the talent work starts now.

To verify postfix version

$ postfix --version

2, Update main.cf file

Edit the  /etc/postfix/main.cf file. [ you use nano or some other editor as your wish]

$ sudo vi /etc/postfix/main.cf

Enter the information below to the new file. Replace ksociety.com with your domain name throughout the file. we need to specify the location of TLS certificate and private key in Postfix configuration,  smtpd_tls_cert_file and smtpd_tls_key_file point to your SSL certificates generated by lets encrypt or your certfile location

$ sudo vi /etc/postfix/main.cf

# See /usr/share/postfix/main.cf.dist for a commented, more complete version
# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# See http://www.postfix.org/COMPATIBILITY_README.html -- default to 2 on
# fresh installs.
compatibility_level = 2



# TLS parameters
#smtpd_tls_cert_file=/etc/letsencrypt/live/mail.ksociety.com/cert.pem
#smtpd_tls_key_file=/etc/letsencrypt/live/mail.ksociety.com/privkey.pem
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mail.ksociety.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = $myhostname, localhost.$mydomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 85.235.65.245
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all

smtpd_tls_security_level=may
smtpd_tls_loglevel = 1
smtp_tls_security_level = may
smtp_tls_loglevel = 1
#Enforce TLSv1.3 or TLSv1.2
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

#####
virtual_mailbox_domains = proxy:mysql:/etc/postfix/sql/mysql_virtual_domains_maps.cf
virtual_mailbox_maps =
   proxy:mysql:/etc/postfix/sql/mysql_virtual_mailbox_maps.cf,
   proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_mailbox_maps.cf
virtual_alias_maps =
   proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_maps.cf,
   proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_maps.cf,
   proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_catchall_maps.cf



virtual_transport = lmtp:unix:private/dovecot-lmtp
smtputf8_enable = no

########
virtual_mailbox_base = /var/vmail
virtual_minimum_uid = 5000
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000


# Milter configuration
milter_default_action = accept
milter_protocol = 6
smtpd_milters = local:/opendkim/opendkim.sock
non_smtpd_milters = $smtpd_milters



### SNI ###
smtpd_tls_chain_files =
    /etc/letsencrypt/live/mail.ksociety.com/privkey.pem,
    /etc/letsencrypt/live/mail.ksociety.com/fullchain.pem

tls_server_sni_maps = hash:/etc/postfix/vhosts_ssl.map

3,  Enabling Submission Service

To send emails from a desktop email client, we need to enable the submission service of Postfix so that the email client can submit emails to Postfix SMTP server. Edit the master.cf file.

$ sudo vi /etc/postfix/master.cf

In submission section, uncomment or add the following lines. By default the submission section is commented out. You can copy the following lines and paste them into the file, so you don’t have to manually uncomment or add new text.The submission daemon listens on TCP port 587. STARTTLS is used to encrypt communications between email client and the submission daemon.

submission     inet     n    -    y    -    -    smtpd 
-o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt 
-o smtpd_tls_wrappermode=no -o smtpd_sasl_auth_enable=yes 
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject 
-o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject 
-o smtpd_sasl_type=dovecot 
-o smtpd_sasl_path=private/auth

Microsoft Outlook mail client only supports submission over port 465. If you are going to use Microsoft Outlook, then you also need to enable submission service on port 465 by adding the following lines in the file.

smtps     inet  n       -       y       -       -       smtpd 
-o syslog_name=postfix/smtps 
-o smtpd_tls_wrappermode=yes 
-o smtpd_sasl_auth_enable=yes 
-o smtpd_relay_restrictions=permit_sasl_authenticated,reject 
-o smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject -o smtpd_sasl_type=dovecot 
-o smtpd_sasl_path=private/auth
$ sudo service postfix restart
$ sudo ss -lnpt | grep master

2: Installing Dovecot IMAP, POP3, LMTP Server

Enter the following command to install Dovecot core package and the IMAP daemon package on Ubuntu server.

$ sudo apt install dovecot-core dovecot-imapd dovecot-pop3d 
dovecot-mysql dovecot-lmtpd

to verify the dovecot version

$ dovecot --version

1,  Enabling IMAP/POP3 Protocol

$ sudo vi /etc/dovecot/dovecot.conf

Add the following line to enable IMAP LMTP & POP3 protocol.

protocols = imap pop3 lmtp

2, Configuring Mailbox Location

By default, Postfix and Dovecot use mbox format to store emails. Each user’s emails are stored in a single file /var/mail/username as unix format  but we going to  use the Maildir format to store email messages for better.

$ sudo vi /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir

Since we are using virtual mailbox domain now, we need to enable mail_home for the virtual users by adding the following line in the file, because virtual users don’t have home directories by default.

mail_home = /var/vmail/%d/%n

We need to add the following line in the file and close.

mail_privileged_group = mail

Then add dovecot to the mail group so that Dovecot can read the INBOX.

$ sudo adduser dovecot mail

3, Configuring Authentication Mechanism

Edit the authentication config file.

$ sudo vi /etc/dovecot/conf.d/10-auth.conf

Uncomment the following line.

disable_plaintext_auth = yes

It will disable plaintext authentication when there’s no SSL/TLS encryption.

Now we are using virtual mailbox domains, which means the username of every email address includes the domain part, so we need to change the auth_username_format as follows. %u won’t drop away the domain (username@your-domain.com) This allows users to login with the full email address.

auth_username_format = %u

LOGIN is another authentication mechanism you probably want to add to support older email clients.search this line and add as below

auth_mechanisms = plain login

Uncomment the following line so Dovecot can query user information from the database.

!include auth-sql.conf.ext

4, Configuring SSL/TLS Encryption

$ sudo vi/etc/dovecot/conf.d/10-ssl.conf

Change ssl = no to ssl = yes to enforce encryption.

ssl = yes

Then find the following lines.  ssl_cert  &  ssl_key

Replace them with the following values, which specify the location of your Let’s Encrypt TLS certificate and private key. Don’t leave out the < character. It’s necessary.

ssl_cert = </etc/letsencrypt/live/mail.your-domain.com/fullchain.pem 
ssl_key = </etc/letsencrypt/live/mail.your-domain.com/privkey.pem

Find the following line and change as below

It’s a good practice to prefer the server’s order of ciphers over client’s. So uncomment this line and change the value to yes.

ssl_prefer_server_ciphers = yes

Also disable inscure  TLSv1, TLSv1.1 and SSLv3, by adding the following line.

Note: If you using Dovecot version 2.3.x or above (as in Ubuntu 20.04), then you should add the following line instead, which will force Dovecot to use TLSv1.2 or TLSv1.3. Please don’t add this line if you use Dovecot version 2.2.x.

ssl_protocols = !SSLv3 !TLSv1 !TLSv1.1
ssl_min_protocol = TLSv1.2

5. Configuring SASL Auth

Postfix supports two SASL implementations: Cyrus SASL and Dovecot SASL.

$ sudo vi /etc/dovecot/conf.d/10-master.conf

Change service auth section to the following so that Postfix can find the Dovecot authentication server. Please be careful about the syntax. Every opening bracket should be terminated by a closing bracket.

service auth {
	unix_listener /var/spool/postfix/private/auth { 
	mode = 0660 
	user = postfix
	group = postfix 
	} 
}

6, Dovecot to Deliver Email to Message Store

By default, Postfix uses its builtin local delivery agent (LDA) to move inbound emails to the message store (inbox, sent, trash, Junk, etc). We can configure it to use Dovecot to deliver emails, via the LMTP protocol, which is a simplified version of SMTP. LMTP allows for a highly scalable and reliable mail system. This step is required if you want to use the sieve plugin to filter inbound messages to different folders.

Since we already installed this plugin as part of step 2 and 2.1, now we directly edit the Dovecot 10-master.conf file.

$ sudo vi /etc/dovecot/conf.d/10-master.conf

Change the lmtp service definition to the following.

service lmtp {  
	unix_listener /var/spool/postfix/private/dovecot-lmtp {    
	mode = 0600   
 	user = postfix 
	group = postfix  
 	} 
 }

Add the following lines at the end of the file. The first line tells Postfix to deliver emails to local message store via the dovecot LMTP server.  The second line disables SMTPUTF8 in Postfix, because Dovecot-LMTP doesn’t support this email extension.

sudo vi /etc/postfix/main.cf

mailbox_transport = lmtp:unix:private/dovecot-lmtp 
smtputf8_enable = no
$ sudo systemctl restart dovecot postfix
$ sudo ss -lnpt | grep master
$ sudo ss -lnpt | grep dovecot

3:  Configure Postfix to Use MySQL/MariaDB Database

By default, Postfix delivers emails only to users with a local Unix account. To make it deliver emails to virtual users whose information is stored in the database, we need to configure Postfix to use virtual mailbox domains.

Since postfix-mysql package we already installed, we directly edit main.cf file

$ sudo vi /etc/postfix/main.cf

Add the following lines at the end of this file.

virtual_mailbox_domains = 	       	
	proxy:mysql:/etc/postfix/sql/mysql_virtual_domains_maps.cf
virtual_mailbox_maps = 		     
	proxy:mysql:/etc/postfix/sql/mysql_virtual_mailbox_maps.cf, 
    proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_mailbox_maps.cf virtual_alias_maps = 
    proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_maps.cf, 
    proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_maps.cf, 
    proxy:mysql:/etc/postfix/sql/mysql_virtual_alias_domain_catchall_maps.cf

Where:

  • virtual_mailbox_domains points to a file that will tell Postfix how to look up domain information from the database.
  • virtual_mailbox_maps points to files that will tell Postfix how to look up email addresses from the database.
  • virtual_alias_maps points to files that will tell Postfix how to look up aliases from the database.

Now create files and folder under /etc/postfix

sudo mkdir /etc/postfix/sql/
touch /etc/postfix/sql/mysql_virtual_domains_maps.cf /etc/postfix/sql/mysql_virtual_mailbox_maps.cf /etc/postfix/sql/mysql_virtual_alias_domain_mailbox_maps.cf /etc/postfix/sql/mysql_virtual_alias_maps.cf /etc/postfix/sql/mysql_virtual_alias_domain_maps.cf /etc/postfix/sql/mysql_virtual_alias_domain_catchall_maps.cf

Create the mysql_virtual_domains_maps.cf file.

$ sudo vi /etc/postfix/sql/mysql_virtual_domains_maps.cf

Add the following content. Replace password with the postfixadmin password you set in Step 1.

user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT domain FROM domain WHERE domain='%s' AND active = '1' 
#query = SELECT domain FROM domain WHERE domain='%s' 
#optional query to use when relaying for backup MX 
#query = SELECT domain FROM domain WHERE domain='%s' AND backupmx = '0' AND active = '1' 
#expansion_limit = 100

Create the mysql_virtual_mailbox_maps.cf file.

$ sudo vi /etc/postfix/sql/mysql_virtual_mailbox_maps.cf

Add the following content.

user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT maildir FROM mailbox WHERE username='%s' AND active = '1' #expansion_limit = 100

Create the mysql_virtual_alias_domain_mailbox_maps.cf file.

$ sudo vi /etc/postfix/sql/mysql_virtual_alias_domain_mailbox_maps.cf

Add the following content.

user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT maildir FROM mailbox,alias_domain WHERE alias_domain.alias_domain = '%d' and mailbox.username = CONCAT('%u', '@', alias_domain.target_domain) AND mailbox.active = 1 AND alias_domain.active='1'

Create the mysql_virtual_alias_maps.cf  file.

sudo vi /etc/postfix/sql/mysql_virtual_alias_maps.cf

Add the following content.

user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT goto FROM alias WHERE address='%s' AND active = '1' #expansion_limit = 100

Create the mysql_virtual_alias_domain_maps.cf file.

$ sudo vi /etc/postfix/sql/mysql_virtual_alias_domain_maps.cf

Add the following content.

user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT goto FROM alias,alias_domain WHERE alias_domain.alias_domain = '%d' and alias.address = CONCAT('%u', '@', alias_domain.target_domain) AND alias.active = 1 AND alias_domain.active='1'

Create the mysql_virtual_alias_domain_catchall_maps.cf file.

sudo vi /etc/postfix/sql/mysql_virtual_alias_domain_catchall_maps.cf

Add the following content.

# handles catch-all settings of target-domain 
user = postfixadmin 
password = password
hosts = localhost 
dbname = postfixadmin 
query = SELECT goto FROM alias,alias_domain WHERE alias_domain.alias_domain = '%d' and alias.address = CONCAT('@', alias_domain.target_domain) AND alias.active = 1 AND alias_domain.active='1'

Since the database passwords are stored in plain text so they should be readable only by user postfix and root, which is done by executing the following two commands.

$ sudo chmod 0640 /etc/postfix/sql/* 
$ sudo setfacl -R -m u:postfix:rx /etc/postfix/sql/

Next, we need to create a user named vmail with ID 5000 and a group with ID 5000.

sudo adduser vmail --system --group --uid 5000 --disabled-login --no-create-home

Create the mail base location.

sudo mkdir /var/vmail/

Make vmail as the owner.

sudo chown vmail:vmail /var/vmail/ -R

Now let’s open the Postfix main configuration file again.

sudo vi /etc/postfix/main.cf

Add the following lines at the end of this file.

virtual_mailbox_base = /var/vmail 
virtual_minimum_uid = 5000 
virtual_uid_maps = static:5000 
virtual_gid_maps = static:5000

The first line defines the base location of mail files. The remaining 3 lines define which user ID and group ID Postfix will use when delivering incoming emails to the mailbox. We use the user ID 5000 and group ID 5000.

Save and close the file. Restart Postfix for the changes to take effect.

sudo systemctl restart postfix

4:  Optional Configurations and Troubleshooting

Troubleshooting Optional

1, It can be helpful to add the following two lines in this file to debug login issues. The login errors would be logged into /var/log/mail.log file. (Once users can login without problems, you can comment out the following two lines.)

$ sudo vi /etc/dovecot/conf.d/10-auth.conf
auth_debug = yes auth_debug_passwords = yes

2. When a user tries to log in, Dovecot would use the Argon2 algorithm to generate a password hash from the password entered by the user, then compare it with the password hash stored in the database. Replace password with the postfixadmin password you set in previous steps.

sudo vi /etc/dovecot/dovecot-sql.conf.ext
driver = mysql 
connect = 
host=localhost 
dbname=postfixadmin 
user=postfixadmin 
password=password 
default_pass_scheme = ARGON2I 
password_query = SELECT username AS user,password FROM mailbox WHERE username = '%u' AND active='1' 
user_query = SELECT maildir, 5000 AS uid, 5000 AS gid FROM mailbox WHERE username = '%u' AND active='1' 
iterate_query = SELECT username AS user FROM mailbox

Save and close the file. Restart Dovecot.

sudo systemctl restart dovecot

5. Roundcube for Webmail Interface

You can literally use email client which supports smtp and pop/imap. So webmail part is completely optional.

apt-get install roundcube roundcube-plugins roundcube-plugins-extra

Above install roundcube inside /usr/share/roundcube

Roundcube config files are present in: /etc/roundcube

Open vim /etc/roundcube/main.inc.php

Add/change following:

$rcmail_config['default_host'] = 'localhost'; 
$rcmail_config['imap_cache'] = memcache; 
$rcmail_config['messages_cache'] = db

6. Nginx config

server { 
server_name mail.ksociety.com; 
access_log   /var/log/nginx/mail.ksociety.com.access.log; 
error_log    /var/log/nginx/mail.ksociety.com.error.log; 
root /usr/share/roundcube; 
index index.php; 
location / { 
	try_files $uri $uri/ /index.php?$args; 
    } 
location ~ \.php$ {
	try_files $uri =404; 
    include fastcgi_params;
    fastcgi_pass 127.0.0.1:9000; 
    } 
}

You can open mail.ksociety.com in browser and login using a virtual user-email and password.

Conclude

At this point, we have SMTP (via Postfix), POP/IMAP/LMTP (via dovecot) and a web-interface (via ROUNDCUBE) to handle emails.