Merge pull request #2 from chillout2k/devel

master init

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+    "python.pythonPath": "/home/dominik/src/git/ExOTA-Milter/venv/bin/python3"
+}

README.md

@@ -1,2 +1,93 @@
-# ExOTA-Milter
-Exchange Online Tenant Autorisation Milter (Mail-Filter) 
+# ExOTA-Milter - Exchange Online Tenant Authorisation Milter (Mail-Filter) 
+
+The **ExOTA-[Milter](https://en.wikipedia.org/wiki/Milter)** application is written in python3 and derives from **[sdgathman´s pymilter](https://github.com/sdgathman/pymilter)**.
+
+# Abstract/problem/motivation
+Fact is that more and more companies are migrating their Outlook/Exchange environments to the [Microsoft cloud](https://www.microsoft.com/microsoft-365).
+
+The **ExOTA-Milter** is kind of an authentication helper which plays an essential as well as security-related role in the so called [*smarthost connector for Microsoft Exchange-Online*](https://docs.microsoft.com/de-de/exchange/mail-flow-best-practices/use-connectors-to-configure-mail-flow/set-up-connectors-to-route-mail) scenario. The milter tries to close the gap of missing [client SMTP-Auth capability](https://tools.ietf.org/html/rfc4954) of MS Exchange-Online when there´s a need to relay emails over a third party smarthost.  According to this the MS Exchange-Online platform expects the smarthost to accept emails without any further authentication of users/tenants, which can lead to many misconfigured SMTP-relays!
+
+So, the question is: *How can an Exchange-Online user/tenant be identified by a smarthost?*
+
+# Identification possibilities provided by Microsoft
+## Client-IP ranges specified by Microsoft
+Microsoft provides an ACL as [JSON file (ID: 10)](https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a7), which looks like this. 
+```
+[...]
+{
+    "id": 10,
+    "serviceArea": "Exchange",
+    "serviceAreaDisplayName": "Exchange Online",
+    "urls": [
+      "*.mail.protection.outlook.com"
+    ],
+    "ips": [
+      "40.92.0.0/15",
+      "40.107.0.0/16",
+      "52.100.0.0/14",
+      "104.47.0.0/17",
+      "2a01:111:f400::/48",
+      "2a01:111:f403::/48"
+    ],
+    "tcpPorts": "25",
+    "expressRoute": true,
+    "category": "Allow",
+    "required": true
+  }
+[...]
+```
+
+The problem of this IP based ACL is that many other Exchange-Online customers/tenants are sending from the same IP-ranges as well! **This means that many smarthost configured to relay mails comming from Exchange-Online tends to be an open relay (for Microsoft customers) unless additional authentication mechanism on a higher layer than IP takes place! IP-address based ACLs are definitely not the right way to achieve this!**
+
+## x509 client certificate presented by Exchange-Online
+The Exchange-Online platform also *presents* a x509 client certificate to identitfy onself to the smarthost. Taking a closer look at the received header we´ll notice that the certificates common name (CN) *mail.protection.outlook.com* is not realy tenant specific. Although the certificate provides additional security regarding the identity of the client system, it does not provide identity regarding the tenant. **IMHO that´s stil not enough to permit relaying!**
+```
+Received: from DEU01-FR2-obe.outbound.protection.outlook.com (mail-fr2deu01lp2173.outbound.protection.outlook.com [104.47.11.173])
+	(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))
+	(Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK))
+	by some.secure.smarthost.example.com (Postfix) with ESMTPS id 4CjqCQ2WRCzGjg6
+	for <blah.blubb@example.com>; Sat, 28 Nov 2020 12:34:26 +0100 (CET)
+```
+Client certificate verification is the job of the underlying MTA. So the **ExOTA-Milter** does not validate the client certificate itself, but it can be enabled (disabled per default) to match for the *expected* client CN. The emailserver [Postfix](http://postfix.org), for example, only provides the client CN to the milter API if the client certificate was successfully validated. Otherwise the **ExOTA-Milter** will not *see* the client CN over the milter protocol ([postfix milter macro](http://www.postfix.org/MILTER_README.html) `{cert_subject}`) which results in a milter reject action.
+
+## DKIM - DomainKey Identified Message
+Nevertheless, as [Microsoft supports DKIM-signing for outbound email traffic](https://docs.microsoft.com/de-de/microsoft-365/security/office-365-security/use-dkim-to-validate-outbound-email?view=o365-worldwide) the **ExOTA-Milter** can be used to authenticate sending tenants, respectively their sender domains, based on the cryptographic capabilities of [DKIM](https://tools.ietf.org/html/rfc6376). In fact the **ExOTA-Milter** does not validate the DKIM-signatures itself. Instead it simply parses DKIM-specific *Authentication-Results* headers produced by any previously DKIM-validating milter (like [OpenDKIM](http://www.opendkim.org/), [Rspamd](https://rspamd.com/) or [AMavis](https://www.ijs.si/software/amavisd/)) in the chain. I personally prefer OpenDKIM as it´s lightweight and fully focused on DKIM.
+
+*DKIM-Signature* headers appended by the Exchange-Online platform look like this:
+```
+[...]
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+ d=tenantdomain.onmicrosoft.com; s=selector1-tenantdomain-onmicrosoft-com;
+ h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
+ bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=;
+ b=DYTLJtLFjvVrSZtZQagTwuEe5PQYqrNGi7hR5bkhO[...snip...]
+[...]
+```
+*Authentication-Results* headers provided by OpenDKIM (signature validated) look like this:
+```
+[...]
+Authentication-Results: trusted.dkim.validating.relay;  dkim=pass header.d=tenantdomain.onmicrosoft.com header.s=selector1-tenantdomain-onmicrosoft-com header.b=mmmjFpv8"
+[...]
+```
+
+## X-MS-Exchange-CrossTenant-Id header
+Further each Microsoft Exchange-Online tenant has a unique tenant-ID in form of a UUID ([RFC 4122](https://tools.ietf.org/html/rfc4122)). **ExOTA-Milter** determines the tenant-ID from the *X-MS-Exchange-CrossTenant-Id* email header and uses it as a *mandatory* authentication factor.
+```
+[...]
+X-MS-Exchange-CrossTenant-Id: <UUID-of-tenant>
+[...]
+```
+
+# The solution
+So, *how can an Exchange-Online user/tenant be identified by a third party smarthost?*
+
+Finally it´s the combination of all of the above discussed aspects which may result in a robust-enough smarthost setup used by the Exchange-Online platform:
+* restriction of client IPs via ACL (MTA)
+* verification of Microsoft´s x509 client certificate (MTA)
+* matching for client certificate´s CN (ExOTA-Milter)
+* verification of DKIM signatures providing *Authentication-Results* header (another milter)
+* consideration of DKIM verification results per sender domain (ExOTA-Milter)
+* matching for tenant-id provided in *X-MS-Exchange-CrossTenant-Id* header (ExOTA-Milter)
+
+# How to start?
+First of all please take a look at how to set up the testing environment, which is described [here](tests/README.md)

app/exota-milter.py

@@ -0,0 +1,349 @@
+import Milter
+import sys
+import traceback
+import os
+import logging
+import string
+import random
+import re
+import email.utils
+import authres
+import json
+from policy import (
+  ExOTAPolicyException, ExOTAPolicyNotFoundException, 
+  ExOTAPolicyBackendJSON, ExOTAPolicy
+)
+
+# Globals with mostly senseless defaults ;)
+g_milter_name = 'exota-milter'
+g_milter_socket = '/socket/' + g_milter_name
+g_milter_reject_message = 'Security policy violation!'
+g_milter_tmpfail_message = 'Service temporarily not available! Please try again later.'
+g_re_domain = re.compile(r'^.*@(\S+)$', re.IGNORECASE)
+g_loglevel = logging.INFO
+g_milter_dkim_enabled = False
+g_milter_trusted_authservid = 'invalid'
+g_milter_policy_source = 'file'
+g_milter_policy_file = None
+g_milter_policy_backend = None
+g_milter_x509_enabled = False
+g_milter_x509_trusted_cn = 'mail.protection.outlook.com'
+
+class ExOTAMilter(Milter.Base):
+  # Each new connection is handled in an own thread
+  def __init__(self):
+    self.reset_milter()
+
+  def reset_milter(self):
+    self.client_ip = None
+    self.hdr_from = None
+    self.hdr_from_domain = None
+    self.hdr_tenant_id = None
+    self.hdr_tenant_id_count = 0
+    self.dkim_results = []
+    self.dkim_valid = False
+    self.x509_client_valid = False
+    # https://stackoverflow.com/a/2257449
+    self.mconn_id = g_milter_name + ': ' + ''.join(
+      random.choice(string.ascii_lowercase + string.digits) for _ in range(8)
+    )
+    logging.debug(self.mconn_id + " reset_milter()")
+
+  # Not registered/used callbacks
+  @Milter.nocallback
+  def connect(self, IPname, family, hostaddr):
+    return Milter.CONTINUE
+  @Milter.nocallback
+  def hello(self, heloname):
+    return Milter.CONTINUE
+  @Milter.nocallback
+  def eoh(self):
+    return Milter.CONTINUE
+  @Milter.nocallback
+  def body(self, chunk):
+    return Milter.CONTINUE
+
+  # Mandatory callback
+  def envfrom(self, mailfrom, *str):
+    # Instance member values remain within reused SMTP-connections!
+    if self.client_ip is not None:
+      # Milter connection reused!
+      logging.debug(self.mconn_id + "/FROM connection reused!")
+      self.reset_milter()
+    self.client_ip = self.getsymval('{client_addr}')
+    if self.client_ip is None:
+      logging.error(self.mconn_id + " FROM exception: could not retrieve milter-macro ({client_addr})!")
+      self.setreply('550','5.7.1', g_milter_tmpfail_message)
+      return Milter.REJECT
+    else:
+      logging.debug(self.mconn_id + "/FROM client_ip={0}".format(self.client_ip))
+    return Milter.CONTINUE
+
+  # Mandatory callback
+  def envrcpt(self, to, *str):
+    logging.debug(self.mconn_id + "/RCPT 5321.rcpt={0}".format(to))
+    return Milter.CONTINUE
+
+  def header(self, name, hval):
+    logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
+      "/HDR: Header: {0}, Value: {1}".format(name, hval)
+    )
+
+    # Parse RFC-5322-From header
+    if(name == "From"):
+      hdr_5322_from = email.utils.parseaddr(hval)
+      self.hdr_from = hdr_5322_from[1].lower()
+      m = re.match(g_re_domain, self.hdr_from)
+      if m is None:
+        logging.error(self.mconn_id  + "/" + str(self.getsymval('i')) + "/HDR " +
+          "Could not determine domain-part of 5322.from=" + self.hdr_from
+        )
+        self.setreply('450','4.7.1', g_milter_tmpfail_message)
+        return Milter.TEMPFAIL
+      self.hdr_from_domain = m.group(1)
+      logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
+        "/HDR: 5322.from={0}, 5322.from_domain={1}".format(
+          self.hdr_from, self.hdr_from_domain
+        )
+      )
+
+    # Parse non-standardized X-MS-Exchange-CrossTenant-Id header
+    elif(name == "X-MS-Exchange-CrossTenant-Id"):
+      self.hdr_tenant_id_count += 1
+      self.hdr_tenant_id = hval.lower()
+      logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
+        "/HDR: Tenant-ID: {0}".format(self.hdr_tenant_id)
+      )
+
+    # Parse RFC-7601 Authentication-Results header
+    elif(name == "Authentication-Results"):
+      if g_milter_dkim_enabled == True:
+        ar = None
+        try:
+          ar = authres.AuthenticationResultsHeader.parse(
+            "{0}: {1}".format(name, hval)
+          )
+          if ar.authserv_id == g_milter_trusted_authservid:
+            for ar_result in ar.results:
+              if ar_result.method == 'dkim':
+                self.dkim_results.append({
+                  "selector": str(ar_result.header_s),
+                  "from_domain": str(ar_result.header_d),
+                  "result": str(ar_result.result)
+                })
+          else:
+            logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
+              "/HDR: Ignoring authentication results of {0}".format(ar.authserv_id)
+            )
+        except:
+          logging.error(self.mconn_id + "/" + str(self.getsymval('i')) +
+            "/HDR: AR-parse exception: " + traceback.format_exc()
+          )
+    return Milter.CONTINUE
+
+  # EOM is mandatory as well and thus always called by MTA
+  def eom(self):
+    # Here in EoM the final policy logic happens.
+
+    # Check if client certificate CN matches trusted CN
+    if g_milter_x509_enabled:
+      cert_subject = self.getsymval('{cert_subject}') 
+      if cert_subject is None:
+        logging.info(self.mconn_id + "/" + self.getsymval('i') 
+          + "/EOM: No trusted x509 client CN found - action=reject"
+        )
+        self.setreply('550','5.7.1', g_milter_tmpfail_message)
+        return Milter.REJECT
+      else:
+        if g_milter_x509_trusted_cn.lower() == cert_subject.lower():
+          self.x509_client_valid = True
+          logging.info(self.mconn_id + "/" + self.getsymval('i') +
+            "/EOM: Trusted x509 client CN {0}".format(cert_subject)
+          )
+        else:
+          logging.info(self.mconn_id + "/" + self.getsymval('i') +
+            "/EOM Untrusted x509 client CN {0} - action=reject".format(cert_subject)
+          )
+          self.setreply('550','5.7.1', g_milter_tmpfail_message)
+          return Milter.REJECT
+
+    if self.hdr_from is None:
+      logging.error(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM exception: could not determine 5322.from header - action=reject"
+      )
+      self.setreply('550','5.7.1', g_milter_tmpfail_message)
+      return Milter.REJECT
+
+    # Get policy for 5322.from_domain
+    policy = None
+    try:
+      policy = g_milter_policy_backend.get(self.hdr_from_domain)
+      logging.debug(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM Policy for 5322.from_domain={0} fetched from backend".format(self.hdr_from_domain)
+      )
+    except (ExOTAPolicyException, ExOTAPolicyNotFoundException) as e:
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM {0}".format(e.message)
+      )
+      self.setreply('550','5.7.1', g_milter_tmpfail_message)
+      return Milter.REJECT
+
+    if self.hdr_tenant_id is None:
+      logging.error(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM exception: could not determine X-MS-Exchange-CrossTenant-Id - action=reject"
+      )
+      self.setreply('550','5.7.1', g_milter_reject_message)
+      return Milter.REJECT
+    if self.hdr_tenant_id_count > 1:
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: More than one tenant-IDs for {0} found - action=reject".format(
+          self.hdr_from_domain
+        )
+      )
+      self.setreply('550','5.7.1', g_milter_reject_message)
+      return Milter.REJECT
+    if self.hdr_tenant_id == policy.get_tenant_id():
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: tenant_id={0} status=match".format(self.hdr_tenant_id)
+      )
+    else:
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: tenant_id={0} status=no_match - action=reject".format(
+          self.hdr_tenant_id
+        )
+      )
+      self.setreply('550','5.7.1', g_milter_reject_message)
+      return Milter.REJECT
+
+    if g_milter_dkim_enabled and policy.is_dkim_enabled():
+      logging.debug(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: 5322.from_domain={0} dkim_auth=enabled".format(self.hdr_from_domain)
+      )
+      if len(self.dkim_results) > 0:
+        for dkim_result in self.dkim_results:
+          if dkim_result['from_domain'] == self.hdr_from_domain:
+            logging.debug(self.mconn_id + "/" + self.getsymval('i') +
+              "/EOM: Found DKIM authentication result for {0}/{1}".format(
+                self.hdr_from_domain, dkim_result['selector']
+              )
+            )
+            if dkim_result['result'] == 'pass':
+              logging.info(self.mconn_id + "/" + self.getsymval('i') +
+                "/EOM: dkim_selector={0} result=pass".format(dkim_result['selector'])
+              )
+              self.dkim_valid = True
+              continue
+            else:
+              logging.info(self.mconn_id + "/" + self.getsymval('i') +
+                "/EOM: dkim_selector={0} result=fail".format(dkim_result['selector'])
+              )
+      else:
+        logging.info(self.mconn_id + "/" + self.getsymval('i') +
+          "/EOM: No DKIM authentication results (AR headers) found - action=reject"
+        )
+        self.setreply('550','5.7.1', g_milter_reject_message)
+        return Milter.REJECT
+      if self.dkim_valid == False:
+        logging.info(self.mconn_id + "/" + self.getsymval('i') +
+          "/EOM: DKIM authentication failed - action=reject"
+        )
+        self.setreply('550','5.7.1', g_milter_reject_message)
+        return Milter.REJECT
+
+    if g_milter_dkim_enabled:
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: Tenant authentication successful (dkim_enabled={0})".format(
+          str(policy.is_dkim_enabled())
+        )
+      )
+    else:
+      logging.info(self.mconn_id + "/" + self.getsymval('i') +
+        "/EOM: Tenant successfully authenticated"
+      )
+    return Milter.CONTINUE
+
+  def abort(self):
+    # Client disconnected prematurely
+    logging.debug(self.mconn_id + "/ABORT")
+    return Milter.CONTINUE
+
+  def close(self):
+    # Always called, even when abort is called.
+    # Clean up any external resources here.
+    logging.debug(self.mconn_id + "/CLOSE")
+    return Milter.CONTINUE
+
+if __name__ == "__main__":
+  if 'LOG_LEVEL' in os.environ:
+    if re.match(r'^info$', os.environ['LOG_LEVEL'], re.IGNORECASE):
+      g_loglevel = logging.INFO
+    elif re.match(r'^warn|warning$', os.environ['LOG_LEVEL'], re.IGNORECASE):
+      g_loglevel = logging.WARN
+    elif re.match(r'^error$', os.environ['LOG_LEVEL'], re.IGNORECASE):
+      g_loglevel = logging.ERROR
+    elif re.match(r'debug', os.environ['LOG_LEVEL'], re.IGNORECASE):
+      g_loglevel = logging.DEBUG
+  logging.basicConfig(
+    filename=None, # log to stdout
+    format='%(asctime)s: %(levelname)s %(message)s',
+    level=g_loglevel
+  )
+  if 'MILTER_NAME' in os.environ:
+    g_milter_name = os.environ['MILTER_NAME']
+  if 'MILTER_SOCKET' in os.environ:
+    g_milter_socket = os.environ['MILTER_SOCKET']
+  if 'MILTER_REJECT_MESSAGE' in os.environ:
+    g_milter_reject_message = os.environ['MILTER_REJECT_MESSAGE']
+  logging.info("ENV[MILTER_REJECT_MESSAGE]: {0}".format(g_milter_reject_message))
+  if 'MILTER_TMPFAIL_MESSAGE' in os.environ:
+    g_milter_tmpfail_message = os.environ['MILTER_TMPFAIL_MESSAGE']
+  logging.info("ENV[MILTER_TMPFAIL_MESSAGE]: {0}".format(g_milter_tmpfail_message))
+  if 'MILTER_DKIM_ENABLED' in os.environ:
+    g_milter_dkim_enabled = True
+    if 'MILTER_TRUSTED_AUTHSERVID' in os.environ:
+      g_milter_trusted_authservid = os.environ['MILTER_TRUSTED_AUTHSERVID'].lower()
+      logging.info("ENV[MILTER_TRUSTED_AUTHSERVID]: {0}".format(g_milter_trusted_authservid))
+    else:
+      logging.error("ENV[MILTER_TRUSTED_AUTHSERVID] is mandatory!")
+      sys.exit(1)
+  logging.info("ENV[MILTER_DKIM_ENABLED]: {0}".format(g_milter_dkim_enabled))
+  if 'MILTER_X509_ENABLED' in os.environ:
+    g_milter_x509_enabled = True
+    if 'MILTER_X509_TRUSTED_CN' in os.environ:
+      g_milter_x509_trusted_cn = os.environ['MILTER_X509_TRUSTED_CN']
+    logging.info("ENV[MILTER_X509_TRUSTED_CN]: {0}".format(g_milter_x509_trusted_cn))
+  logging.info("ENV[MILTER_X509_ENABLED]: {0}".format(g_milter_x509_enabled))
+  if 'MILTER_POLICY_SOURCE' in os.environ:
+    g_milter_policy_source = os.environ['MILTER_POLICY_SOURCE']
+  logging.info("ENV[MILTER_POLICY_SOURCE]: {0}".format(g_milter_policy_source))
+  if g_milter_policy_source == 'file':
+    if 'MILTER_POLICY_FILE' in os.environ:
+      g_milter_policy_file = os.environ['MILTER_POLICY_FILE']
+      logging.info("ENV[MILTER_POLICY_FILE]: {0}".format(g_milter_policy_file))
+      try:
+        g_milter_policy_backend = ExOTAPolicyBackendJSON(g_milter_policy_file)
+        logging.info("JSON policy backend initialized")
+      except ExOTAPolicyException as e:
+        logging.error("Policy backend error: {0}".format(e.message))
+        sys.exit(1)
+    else:
+      logging.error("ENV[MILTER_POLICY_FILE] is mandatory!")
+      sys.exit(1)
+  elif g_milter_policy_source == 'ldap':
+    logging.debug("LDAP-Backend not supported yet!")
+    sys.exit(1)
+  else:
+    logging.debug("Unsupported backend: {0}!".format(g_milter_policy_source))
+    sys.exit(1)
+  try:
+    timeout = 600
+    # Register to have the Milter factory create instances of your class:
+    Milter.factory = ExOTAMilter
+    logging.info("Startup " + g_milter_name +
+      "@socket: " + g_milter_socket
+    )
+    Milter.runmilter(g_milter_name,g_milter_socket,timeout,True)
+    logging.info("Shutdown " + g_milter_name)
+  except:
+    logging.error("MAIN-EXCEPTION: " + traceback.format_exc())
+    sys.exit(1)

app/policy.py

@@ -0,0 +1,101 @@
+import json
+import traceback
+import re
+
+class ExOTAPolicyException(Exception):
+  def __init__(self, message):
+    self.message = message
+
+class ExOTAPolicyNotFoundException(ExOTAPolicyException):
+  pass
+
+class ExOTAPolicyInvalidException(ExOTAPolicyException):
+  pass
+
+class ExOTAPolicy():
+  def __init__(self, policy_dict):
+    self.tenant_id = policy_dict['tenant_id']
+    self.dkim_enabled = policy_dict['dkim_enabled']
+
+  def get_tenant_id(self):
+    return self.tenant_id
+
+  def is_dkim_enabled(self):
+    return self.dkim_enabled
+
+  @staticmethod
+  def check_policy(policy_dict):
+    if 'tenant_id' not in policy_dict:
+      raise ExOTAPolicyInvalidException(
+        "Policy must have a 'tenant_id' attribute!"
+      )
+    else:
+      if policy_dict['tenant_id'] == '':
+        raise ExOTAPolicyInvalidException(
+          "'tenant_id' must not be empty!"
+        )
+      if re.match(r'^.*\s+.*$', policy_dict['tenant_id']):
+        raise ExOTAPolicyInvalidException(
+          "'tenant_id' must not contain whitespace characters!"
+        )
+    if 'dkim_enabled' not in policy_dict:
+      raise ExOTAPolicyInvalidException(
+        "Policy must have a 'dkim_enabled' attribute!"
+      )
+    else:
+      if not isinstance(policy_dict['dkim_enabled'], bool):
+        raise ExOTAPolicyInvalidException(
+          "'dkim_enabled'({0}) must be boolean!".format(policy_dict['dkim_enabled'])
+        )
+
+class ExOTAPolicyBackend():
+  type = None
+  def __init__(self):
+    pass
+  def get(self, from_domain):
+    pass
+
+########## JSON file
+class ExOTAPolicyBackendJSON(ExOTAPolicyBackend):
+  type = 'json'
+  def __init__(self, file_path):
+    self.policies = None
+    try:
+      with open(file_path, 'r') as policy_file:
+        self.policies = json.load(policy_file)
+        policy_file.close()
+        # validate policy
+        for policy in self.policies:
+          try:
+            ExOTAPolicy.check_policy(self.policies[policy])
+          except ExOTAPolicyInvalidException as e:
+            raise ExOTAPolicyException(
+              "Policy {0} is invalid: {1}".format(policy, e.message)
+            ) from e
+    except json.decoder.JSONDecodeError as e:
+      raise ExOTAPolicyException(
+        "JSON-error in policy file: " + str(e)
+      ) from e
+    except Exception as e:
+      raise ExOTAPolicyException(
+        "Error reading policy file: " + traceback.format_exc()
+      ) from e
+  
+  def get(self, from_domain):
+    try:
+      return ExOTAPolicy(self.policies[from_domain])
+    except KeyError as e:
+      raise ExOTAPolicyNotFoundException(
+        "Policy for from_domain={0} not found".format(from_domain)
+      ) from e
+    except Exception as e:
+      raise ExOTAPolicyException(
+        "Error fetching policy for {0}: {1}".format(
+          from_domain, traceback.format_exc()
+        )
+      ) from e
+
+########## LDAP
+class ExOTAPolicyBackendLDAP(ExOTAPolicyBackendJSON):
+  type = 'ldap'
+  pass

requirements.txt

@@ -0,0 +1,3 @@
+authres==1.2.0
+pkg-resources==0.0.0
+pymilter==1.0.4

samples/exo_validator.eml

@@ -0,0 +1,64 @@
+Return-Path: <>
+Received: from DEU01-FR2-obe.outbound.protection.outlook.com (mail-fr2deu01lp2173.outbound.protection.outlook.com [104.47.11.173])
+	(using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits))
+	(Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK))
+	by outbound.connector.blahblubb.de (Postfix) with ESMTPS id 4CjqCQ2WRCzGjg6
+	for <some.recipient@example.org>; Sat, 28 Nov 2020 12:34:26 +0100 (CET)
+ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none;
+ b=RuGSfIN1OzQHDqrF0erLAHZ3fyhtmoE5Sllj+Qp6CtbcNUkkmdhR44b8capz/J1mBpyb13udY1mhkPZCK1Cmt+mpg9yFXgkv5BxY+dV9647Fq+MboUE60Psn84d4vXFvyrWDrFW1jWZi7/NdXhjLcCqTHpAzDaRfAOfGhG/VWYJAXnD/EBpCzPfd8hh9ZOONI2UN2HQfRnx0P3WXyeVSGilP4RGPdmcCZV5ZzpjlQoKUshjq293+ZltXaeKfF/LHGX0yScHhKO2f9O+qY3hnH0P+NGwFvhIky3IyszfxpANaJnz2Jpp0sK1W16rwGSTI2gl9bpJsj+wzKLGkJV75+Q==
+ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com;
+ s=arcselector9901;
+ h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
+ bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=;
+ b=go8dFv6srV3NnETxQxaANld1if9BOsIgrhjefC4WkRrrgwEjZSNnm9DyO+GC2ZZo60At5JHOVLjqN9kjz2pFdAG0qnFEj3Wx/6NnuTfBUk0n4s32RoFuhADu8BC+aOU9Ec909uu2QQ9ucEMiVSjuyQ3QpGS5DR0yCAZLZ12B61hmoMgkXJ9ah6rluUV4GeMGKTsUn16u6mrJycXp0OoD4n19JomPpQo5o8gouK3Zz4F7DxX4lshNJ+VCsOznqS+FI4rQ2LSyU8Y0AZa9clyCSN94AJa6K0TiDgQ/gLZEWsZ1tZkgPrdMlyqi58ONW/dNQ7lyrEFz6deB4YmsusJPbQ==
+ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none
+ action=none header.from=lalalulu.onmicrosoft.com; dkim=none (message not
+ signed); arc=none
+DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
+ d=lalalulu.onmicrosoft.com; s=selector1-lalalulu-onmicrosoft-com;
+ h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
+ bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=;
+ b=DYTLJtLFjvVrSZtZQagTwuEe5PQYqrNGi7hR5bkhO+GYUV4dcQZnDO4hAPzJkOWhz8JCVJ+/yt5K8L/exegk80g9m0GJjZzJBxMy0ZE/7wg8yqiHNE+iQqWhJLtwsD23kx2+09G5dBSDI1QVqFKkL0YKBWVffSuXi+tjM4/BztffZ7ok7XZdKCFfKzK3TLdiAWYTRIp1214zdnIE0CLBhnOIWC4gnML2fXsVZsWb/CMgaW0vBsZGI/yaSivaNFPZloSb0/sEnMFMEbv2GXt9mN913M0thwCi/+NLwzaW6TNlw2Vz7l4SGRVvciGaa4s2sFnJ0ANMD2u5qBbJ8j8Z0w==
+Authentication-Results: blahblubb.de; dkim=none (message not signed)
+ header.d=none;blahblubb.de; dmarc=none action=none
+ header.from=lalalulu.onmicrosoft.com;
+Received: from AM6P193CA0087.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:88::28)
+ by BEXP281MB0216.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:6::12) with
+ Microsoft SMTP Server (version=TLS1_2,
+ cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.6; Sat, 28 Nov
+ 2020 11:34:25 +0000
+Received: from BE0P281MB0257.DEUP281.PROD.OUTLOOK.COM
+ (2603:10a6:209:88:cafe::a2) by AM6P193CA0087.outlook.office365.com
+ (2603:10a6:209:88::28) with Microsoft SMTP Server (version=TLS1_2,
+ cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20 via Frontend
+ Transport; Sat, 28 Nov 2020 11:34:24 +0000
+From: O365ConnectorValidation@lalalulu.onmicrosoft.com
+Date: Sat, 28 Nov 2020 11:34:24 +0000
+Message-Id: <b6d9c673-d0f3-4538-bb4e-9e099fb9a388@substrate-int.office.com>
+To: some.recipient@example.org
+Subject: Test email for connector validation
+MIME-Version: 1.0
+Content-Type: text/plain; charset=us-ascii
+X-MS-PublicTrafficType: Email
+X-MS-Office365-Filtering-Correlation-Id: abcd1234-abcd-471a-1234-08d893918edd
+X-MS-TrafficTypeDiagnostic: BEXP281MB0216:
+X-Microsoft-Antispam-PRVS:
+	<BEXP281MB021624EF3E3FC35524889C2AB8F70@BEXP281MB0216.DEUP281.PROD.OUTLOOK.COM>
+X-MS-Oob-TLC-OOBClassifiers: OLM:2733;
+X-MS-Exchange-SenderADCheck: 1
+X-Microsoft-Antispam: BCL:0;
+X-Microsoft-Antispam-Message-Info:
+	P2dut4iALZ4EsHFmDE6p0OBg/Q4PvbmhUGI6BnGbHo/u7Vza6tyXE6BPK0VrJQ8WnCYXNx7lEKtiZs8nakJ9EghgxvFRNuYyRBJcGAdlN2TJAb2/7Wp5m7vzuGp1JJhES0RC/hypLDL8miRoP1xYl/pQHZVUGczSddujsZT6im0EgDJvAB0L1vzyKvZJ1QH3vTWDKMAgetlQHiPvCfzZmUgY92g1+sfF9UwGTRXDj8cd83H+TLI7GL8kZF1H219l+DLDiZ3u+qUdprwMn9XDEBljZpczY8BhiFdmnbyJ26ePVNa5JluRboz2Gfaa6GZE+ar8FyKtepxFOyNlI+hyL/vcWNwmnjL+pyYFVPPHnODjxu8JixWg00ThTUiZbclJ
+X-Forefront-Antispam-Report:
+	CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BE0P281MB0257.DEUP281.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(376002)(346002)(39830400003)(34036004)(366004)(396003)(136003)(31686004)(78352004)(6916009)(508600001)(5660300002)(42882007)(2906002)(8936002)(558084003)(31696002)(17440700003)(316002)(9686003)(85236043)(68406010)(8676002)(83380400001)(16130700016)(100380200003)(20230700015);DIR:OUT;SFP:1501;
+X-OriginatorOrg: lalalulu.onmicrosoft.com
+X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2020 11:34:24.7460
+ (UTC)
+X-MS-Exchange-CrossTenant-Network-Message-Id: abcd1234-abcd-471a-1234-08d893918edd
+X-MS-Exchange-CrossTenant-AuthSource: AM6P193CA0087.EURP193.PROD.OUTLOOK.COM
+X-MS-Exchange-CrossTenant-AuthAs: Internal
+X-MS-Exchange-CrossTenant-Id: 1234abcd-18c5-45e8-88de-123456789abc
+X-MS-Exchange-CrossTenant-FromEntityHeader: Internet
+X-MS-Exchange-Transport-CrossTenantHeadersStamped: BEXP281MB0216
+
+This test email message was sent from Office 365 to check that email can be delivered to you using your new or modified connector. No need to reply.

samples/multiple_from.eml

@@ -0,0 +1,2 @@
+From: from2@example.com
+From: from1@example.org

snippets/parse_5322_headers.py

@@ -0,0 +1,16 @@
+import sys
+import email, email.header
+from email.utils import getaddresses
+
+f = open("../samples/exo_validator.eml", "r")
+email = email.message_from_file(f)
+from_hdr = email.get_all("From")
+print("from_hdr: " + str(from_hdr))
+if(len(from_hdr) > 1):
+  print("Multiple From-headers found!")
+  sys.exit(1)
+elif(len(from_hdr) == 1):
+  print("Exactly one From-header found :)")
+  print(from_hdr)
+  from_addr = getaddresses(from_hdr)
+  print(str(from_addr[0][1]))

tests/README.md

@@ -0,0 +1,35 @@
+# Prepare testing env
+First of all, please configure a python virtual environment and install all necessary python packages listed under `requirements.txt`. Go to the root-directory of this repo and
+1. `python3 -m venv venv`
+1. `. venv/bin/activate`
+1. `pip3 install -r requirements.txt`
+
+It´s not realy neccessary to configure a fully functional milter-aware MTA to see **ExOTA-Milter** in action. All you need is 
+* a binary called `miltertest`. Under debian based distros it´s located in the `opendkim-tools` package.
+* a lua-script for miltertest: `tests/miltertest.lue`
+* an **ExOTA-Milter** policy JSON-file: `tests/policy.json`
+
+Except for the `miltertest` binary you´ll find all mandatory resources to run a test in this repo.
+
+```
+export LOG_LEVEL=debug
+export MILTER_SOCKET=/tmp/exota-milter
+export MILTER_POLICY_FILE=tests/policy.json
+export MILTER_DKIM_ENABLED=yepp
+export MILTER_TRUSTED_AUTHSERVID=my-auth-serv-id
+export MILTER_X509_ENABLED=yepp
+export MILTER_X509_TRUSTED_CN=mail.protection.outlook.com
+```
+
+# Shell-1: start ExOTA-Milter
+```
+. venv/bin/activate
+python3 app/exota-milter.py
+```
+
+# Shell-2: execute `miltertest`
+This must be done only once: `export MILTER_SOCKET=/tmp/exota-milter`
+
+Execute miltertest pointing to the test script written in lua to feed the **ExOTA-Milter**:
+
+`miltertest -v -D socket="${MILTER_SOCKET}" -s tests/miltertest.lua`

tests/miltertest.lua

@@ -0,0 +1,73 @@
+-- https://mopano.github.io/sendmail-filter-api/constant-values.html#com.sendmail.milter.MilterConstants
+-- http://www.opendkim.org/miltertest.8.html
+
+-- socket must be defined as miltertest global variable (-D)
+conn = mt.connect(socket)
+if conn == nil then
+  error "mt.connect() failed"
+end
+
+mt.set_timeout(3)
+
+-- 5321.FROM + MACROS
+mt.macro(conn, SMFIC_MAIL, '{client_addr}', "127.128.129.130", "i", "4CgSNs5Q9sz7SllQ", '{cert_subject}', "mail.protection.outlook.com")
+if mt.mailfrom(conn, "envelope.sender@example.org") ~= nil then
+  error "mt.mailfrom() failed"
+end
+if mt.getreply(conn) ~= SMFIR_CONTINUE then
+  error "mt.mailfrom() unexpected reply"
+end
+
+-- 5321.RCPT
+if mt.rcptto(conn, "<envelope.recipient@example.com>") ~= nil then
+  error "mt.rcptto() failed"
+end
+if mt.getreply(conn) ~= SMFIR_CONTINUE then
+  error "mt.rcptto() unexpected reply"
+end
+
+-- HEADER
+if mt.header(conn, "From", '"Blah Blubb" <O365ConnectorValidation@lalalulu.onmicrosoft.com>') ~= nil then
+  error "mt.header(From) failed"  
+end
+if mt.header(conn, "X-MS-Exchange-CrossTenant-Id", "1234abcd-18c5-45e8-88de-123456789abc") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+--if mt.header(conn, "X-MS-Exchange-CrossTenant-Id", "4321abcd-18c5-45e8-88de-blahblubb") ~= nil then
+--  error "mt.header(Subject) failed"  
+--end
+if mt.header(conn, "Authentication-Results", "another-wrong-auth-serv-id;\n  dkim=fail header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+if mt.header(conn, "Authentication-Results", "wrong-auth-serv-id;\n  dkim=pass header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+if mt.header(conn, "Authentication-Results", "my-auth-serv-id;\n  dkim=pass header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+if mt.header(conn, "Authentication-Results", "my-auth-serv-id;\n  dkim=fail header.d=lalalulu.onmicrosoft.com header.s=selector2-asdf header.b=mmmjFpv8") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+if mt.header(conn, "Authentication-Results", "some-validating-host;\n dkim=pass header.d=paypal.de header.s=pp-dkim1 header.b=PmTtUzer;\n dmarc=pass (policy=reject) header.from=paypal.de;\n spf=pass (some-validating-host: domain of service@paypal.de designates 173.0.84.226 as permitted sender) smtp.mailfrom=service@paypal.de") ~= nil then
+  error "mt.header(Subject) failed"  
+end
+
+-- EOM
+if mt.eom(conn) ~= nil then
+  error "mt.eom() failed"
+end
+mt.echo("EOM: " .. mt.getreply(conn))
+if mt.getreply(conn) == SMFIR_CONTINUE then
+  mt.echo("EOM-continue")
+elseif mt.getreply(conn) == SMFIR_REPLYCODE then
+  mt.echo("EOM-reject")
+end
+
+if not mt.eom_check(conn, MT_HDRADD, "X-SOS-Milter") then
+  mt.echo("no header added")
+else
+  mt.echo("X-SOS-Milter header added -> LDAP-Domain with broken SPF")
+end
+
+-- DISCONNECT
+mt.disconnect(conn)

tests/policy.json

@@ -0,0 +1,10 @@
+{
+  "lalalulu.onmicrosoft.com": {
+    "tenant_id": "1234abcd-18c5-45e8-88de-123456789abc",
+    "dkim_enabled": true
+  },
+  "asdf2.onmicrosoft.com": {
+    "tenant_id": "asdftasdfa",
+    "dkim_enabled": true
+  }
+}