Merge pull request #13 from chillout2k/devel

Docs; forget about the dkim-selector

OCI/README.md

@@ -0,0 +1,78 @@
+# How to build and deploy ExOTA-Milter as an OCI container
+## Build with `docker-cli`
+Actually I´m going with docker-ce to build the container image, but same results should come out with e.g. [img](https://github.com/genuinetools/img) etc.
+
+Run following command in the root directory of this repo:
+```
+docker build -t exota-milter:local -f OCI/Dockerfile .
+[...]
+Successfully built 9cceb121f604
+Successfully tagged exota-milter:local
+```
+
+## Deploy with `docker-compose`
+Prerequisites: `docker-compose` installed
+* Create a deployment directory and jump into it. In my case it´s `/docker/containers/exota-milter`
+  * `install -d /docker/containers/exota-milter`
+  * `cd /docker/containers/exota-milter`
+* Create further directories in the deployment directory: 
+  * `install -d -m 777 data`. The application expects the policy file in `/data/policy.json` (path inside the container!).
+  * `install -d -m 777 socket`. The application places the milter socket file under `/socket/exota-milter` (path inside the container!)
+* Create the policy file `data/policy.json` with following content:
+```
+{
+  "yad.onmicrosoft.com": {
+    "tenant_id": "1234abcd-18c5-45e8-88de-123456789abc",
+    "dkim_enabled": true
+  },
+  "example.com": {
+    "tenant_id": "abcd1234-18c5-45e8-88de-987654321cba",
+    "dkim_enabled": false
+  }
+}
+```
+* Create a file named `docker-compose.yml` in the deployment directory with following content:
+```
+version: '2.4'
+
+services:
+  exota-milter:
+    image: exota-milter:local
+    environment:
+      LOG_LEVEL: 'debug'
+      MILTER_SOCKET: '/socket/exota-milter'
+      #MILTER_SOCKET: 'inet:123456@0.0.0.0'
+      MILTER_POLICY_FILE: '/data/policy.json'
+      MILTER_DKIM_ENABLED: 'some_value'
+      MILTER_TRUSTED_AUTHSERVID: 'my-auth-serv-id'
+      MILTER_X509_ENABLED: 'some_value'
+      MILTER_X509_TRUSTED_CN: 'mail.protection.outlook.com'
+      MILTER_ADD_HEADER: 'some_value'
+      MILTER_AUTHSERVID: 'my-auth-serv-id'
+    volumes:
+    - "./data/:/data/:ro"
+    - "./socket/:/socket/:rw"
+```
+If the milter should listen on a TCP-socket instead, just change the value of the `MILTER_SOCKET` ENV-variable to something like `inet:<port>@0.0.0.0`. As IPv6 is supported by the `libmilter` library too, a notation like `inet6:<port>@[::]` is also possible.
+
+* Deploy
+
+Execute `docker-compose up` and if nothing went wrong you shold see following output:
+```
+Creating network "exota-milter_default" with the default driver
+Creating exota-milter_exota-milter_1 ... done
+Attaching to exota-milter_exota-milter_1
+exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_SOCKET]: /socket/exota-milter
+exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_REJECT_MESSAGE]: Security policy violation!
+exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_TMPFAIL_MESSAGE]: Service temporarily not available! Please try again later.
+exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_TRUSTED_AUTHSERVID]: my-auth-serv-id
+exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_DKIM_ENABLED]: True
+exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_X509_TRUSTED_CN]: mail.protection.outlook.com
+exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_X509_ENABLED]: True
+exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_POLICY_SOURCE]: file
+exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_POLICY_FILE]: /data/policy.json
+exota-milter_1  | 2020-11-30 12:38:51,166: INFO JSON policy backend initialized
+exota-milter_1  | 2020-11-30 12:38:51,166: INFO Startup exota-milter@socket: /socket/exota-milter
+```
+
+Voila! The milter socket can be accessed on the host filesystem (in my case) under `/docker/containers/exota-milter/socket/exota-milter`.

README.md

@@ -53,6 +53,10 @@ Client certificate verification is the job of the underlying MTA. So the **ExOTA
 ## DKIM - DomainKey Identified Message
 Nevertheless, as [Microsoft supports DKIM-signing for outbound email traffic](https://docs.microsoft.com/de-de/microsoft-365/security/office-365-security/use-dkim-to-validate-outbound-email?view=o365-worldwide) the **ExOTA-Milter** can be used to authenticate sending tenants, respectively their sender domains, based on the cryptographic capabilities of [DKIM](https://tools.ietf.org/html/rfc6376). In fact the **ExOTA-Milter** does not validate the DKIM-signatures itself. Instead it simply parses DKIM-specific *Authentication-Results* headers produced by any previously DKIM-validating milter (like [OpenDKIM](http://www.opendkim.org/), [Rspamd](https://rspamd.com/) or [AMavis](https://www.ijs.si/software/amavisd/)) in the chain. I personally prefer OpenDKIM as it´s lightweight and fully focused on DKIM.
 
+**To use DKIM for tenant/sender domain authentication, DKIM must be enabled in the milter as well as in each policy!**
+
+**Worth to know when using OpenDKIM as AR provider:** As Microsoft already signs with 2kRSA keys be sure to use a version of OpenDKIM, which is linked against a DNS resolver library that is able to handle such large DNS responses! Further the resolver library should be aware of DNSSEC! **[libunbound](https://nlnetlabs.nl/documentation/unbound/libunbound/) meets all of these requirements :-)**. A libunbound-linked version of OpenDKIM is provided by [Debian](https://wiki.debian.org/opendkim#DNS_resolution).
+
 *DKIM-Signature* headers appended by the Exchange-Online platform look like this:
 ```
 [...]
@@ -63,10 +67,12 @@ DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
  b=DYTLJtLFjvVrSZtZQagTwuEe5PQYqrNGi7hR5bkhO[...snip...]
 [...]
 ```
-*Authentication-Results* headers provided by OpenDKIM (signature validated) look like this:
+*Authentication-Results* headers provided by OpenDKIM (signature valid, public key not DNSSEC signed) look like this:
 ```
 [...]
-Authentication-Results: trusted.dkim.validating.relay;  dkim=pass header.d=tenantdomain.onmicrosoft.com header.s=selector1-tenantdomain-onmicrosoft-com header.b=mmmjFpv8"
+Authentication-Results: trusted.dkim.validating.relay;
+  dkim=pass (2048-bit key; unprotected) header.d=tenantdomain.onmicrosoft.com header.i=@tenantdomain.onmicrosoft.com header.b=mmmjFpv8";
+  dkim-atps=neutral
 [...]
 ```
 
@@ -99,88 +105,14 @@ Finally it´s the combination of all of the above discussed aspects which may re
 * consideration of DKIM verification results per sender domain (ExOTA-Milter)
 * matching for tenant-id provided in *X-MS-Exchange-CrossTenant-Id* header (ExOTA-Milter)
 
+![Activity policy](http://www.plantuml.com/plantuml/png/5SKn3W8W30NGg-W1f8cZcuEZSN4tM8aq5ahAhyhjZMzvM-ciyIZXkgd0c0SYpv_q5DIunopErb4w4biZhg9gWVsBJj_BzRWxYw8ujJp_POQy1UisJ8LN6j7q1m00)
+
 # How about a docker/OCI image?
 ## Using prebuilt images from dockerhub.com
 **WIP ;-)**
 
 ## Build your own image
-Actually I´m going with docker-ce to build the container image, but same results should come out with e.g. [img](https://github.com/genuinetools/img) etc.
-
-Run following command in the root directory of this repo:
-```
-docker build -t exota-milter:local -f OCI/Dockerfile .
-[...]
-Successfully built 9cceb121f604
-Successfully tagged exota-milter:local
-```
-
-## Deploy the OCI image with `docker-compose`
-Prerequisites: `docker-compose` installed
-* Create a deployment directory and jump into it. In my case it´s `/docker/containers/exota-milter`
-  * `install -d /docker/containers/exota-milter`
-  * `cd /docker/containers/exota-milter`
-* Create further directories in the deployment directory: 
-  * `install -d -m 777 data`. The application expects the policy file in `/data/policy.json` (path inside the container!).
-  * `install -d -m 777 socket`. The application places the milter socket file under `/socket/exota-milter` (path inside the container!)
-* Create the policy file `data/policy.json` with following content:
-```
-{
-  "yad.onmicrosoft.com": {
-    "tenant_id": "1234abcd-18c5-45e8-88de-123456789abc",
-    "dkim_enabled": true
-  },
-  "example.com": {
-    "tenant_id": "abcd1234-18c5-45e8-88de-987654321cba",
-    "dkim_enabled": false
-  }
-}
-```
-* Create a file named `docker-compose.yml` in the deployment directory with following content:
-```
-version: '2.4'
-
-services:
-  exota-milter:
-    image: exota-milter:local
-    environment:
-      LOG_LEVEL: 'debug'
-      MILTER_SOCKET: '/socket/exota-milter'
-      #MILTER_SOCKET: 'inet:123456@0.0.0.0'
-      MILTER_POLICY_FILE: '/data/policy.json'
-      MILTER_DKIM_ENABLED: 'some_value'
-      MILTER_TRUSTED_AUTHSERVID: 'my-auth-serv-id'
-      MILTER_X509_ENABLED: 'some_value'
-      MILTER_X509_TRUSTED_CN: 'mail.protection.outlook.com'
-      MILTER_ADD_HEADER: 'some_value'
-      MILTER_AUTHSERVID: 'my-auth-serv-id'
-    volumes:
-    - "./data/:/data/:ro"
-    - "./socket/:/socket/:rw"
-```
-If the milter should listen on a TCP-socket instead, just change the value of the `MILTER_SOCKET` ENV-variable to something like `inet:<port>@0.0.0.0`. As IPv6 is supported by the `libmilter` library too, a notation like `inet6:<port>@[::]` is also possible.
-
-* Deploy
-
-Execute `docker-compose up` and if nothing went wrong you shold see following output:
-```
-Creating network "exota-milter_default" with the default driver
-Creating exota-milter_exota-milter_1 ... done
-Attaching to exota-milter_exota-milter_1
-exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_SOCKET]: /socket/exota-milter
-exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_REJECT_MESSAGE]: Security policy violation!
-exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_TMPFAIL_MESSAGE]: Service temporarily not available! Please try again later.
-exota-milter_1  | 2020-11-30 12:38:51,164: INFO ENV[MILTER_TRUSTED_AUTHSERVID]: my-auth-serv-id
-exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_DKIM_ENABLED]: True
-exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_X509_TRUSTED_CN]: mail.protection.outlook.com
-exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_X509_ENABLED]: True
-exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_POLICY_SOURCE]: file
-exota-milter_1  | 2020-11-30 12:38:51,165: INFO ENV[MILTER_POLICY_FILE]: /data/policy.json
-exota-milter_1  | 2020-11-30 12:38:51,166: INFO JSON policy backend initialized
-exota-milter_1  | 2020-11-30 12:38:51,166: INFO Startup exota-milter@socket: /socket/exota-milter
-```
-
-Voila! The milter socket can be accessed on the host filesystem (in my case) under `/docker/containers/exota-milter/socket/exota-milter`.
-
+Take a look [here](OCI/README.md)
 
 # How to test?
 First of all please take a look at how to set up the testing environment, which is described [here](tests/README.md)

activity_policy.puml

@@ -0,0 +1,59 @@
+@startuml
+
+title ExOTA-Milter security policy flow 
+start
+:MTA connected; 
+
+:HDR: Collecting all relevant headers;
+note left: From, Authentication-Results, X-MS-Exchange-CrossTenant-Id
+
+:HDR: Recognising sender domain;
+note left: Taken from RFC5322 From-header. RFC5321.mail (envelope) is NOT relevant!
+
+:EOM: Looking up policy in backend;
+note left: Based on RFC5322.from domain
+
+if (Policy found?) then (yes)
+  if (Milter: x509 client CN checking enabled?) then (yes)
+    :Looking up x509 client CN;
+    note left: ENV[MILTER_X509_TRUSTED_CN]
+    if (Found trusted x509 client CN?) then (yes)
+    else (no)
+      :REJECT;
+      stop
+    endif
+  else (no)
+  endif
+  if (Milter: DKIM checking enabled?) then (yes)
+    if (Policy has DKIM checking enabled?) then (yes)
+      :Looking up trusted Authentication-Results headers;
+      note left: ENV[MILTER_TRUSTED_AUTHSERVID]
+      if (Found trusted DKIM AR-headers?) then (yes)
+      else (no)
+        :REJECT;
+        stop
+      endif
+    else (no)
+    endif
+  else (no)
+  endif
+  :Looking up tenant-id in policy;
+  if (Found trusted tenant-ID?) then (no)
+    :REJECT;
+    stop
+  else (yes)
+  endif
+else (no)
+  :REJECT;
+  stop
+endif
+:Removing all X-ExOTA-Authentication-Results headers if present;
+if (Milter: add header?) then (yes)
+  :Adding X-ExOTA-Authentication-Results header;
+  note left: ENV[MILTER_ADD_HEADER]
+else (no)
+endif
+:CONTINUE;
+stop
+
+@enduml

app/exota-milter.py

@@ -51,22 +51,21 @@ class ExOTAMilter(Milter.Base):
   def __init__(self):
     self.x509_client_valid = False
     self.client_ip = None
-    self.reset_milter()
+    self.reset()
 
-  def reset_milter(self):
+  def reset(self):
     self.conn_reused = False
     self.hdr_from = None
     self.hdr_from_domain = None
     self.hdr_tenant_id = None
     self.hdr_tenant_id_count = 0
-    self.dkim_results = []
     self.dkim_valid = False
     self.xar_hdr_count = 0
     # https://stackoverflow.com/a/2257449
     self.mconn_id = g_milter_name + ': ' + ''.join(
       random.choice(string.ascii_lowercase + string.digits) for _ in range(8)
     )
-    logging.debug(self.mconn_id + " reset_milter()")
+    logging.debug(self.mconn_id + " reset()")
 
   def smfir_reject(self, **kwargs):
     message = g_milter_reject_message
@@ -76,6 +75,9 @@ class ExOTAMilter(Milter.Base):
       message = "queue_id: {0} - {1}".format(kwargs['queue_id'], message)
     if 'reason' in kwargs:
       message = "{0} - reason: {1}".format(message, kwargs['reason'])
+    logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
+      ": milter_action=reject"
+    )
     self.setreply('550','5.7.1', message)
     return Milter.REJECT
   
@@ -104,7 +106,7 @@ class ExOTAMilter(Milter.Base):
     if self.conn_reused:
       # Milter connection reused!
       logging.debug(self.mconn_id + "/FROM connection reused!")
-      self.reset_milter()
+      self.reset()
     else:
       self.conn_reused = True
       logging.debug(self.mconn_id + "/FROM client_ip={0}".format(self.client_ip))
@@ -156,11 +158,8 @@ class ExOTAMilter(Milter.Base):
           if ar.authserv_id == g_milter_trusted_authservid:
             for ar_result in ar.results:
               if ar_result.method == 'dkim':
-                self.dkim_results.append({
-                  "selector": str(ar_result.header_s),
-                  "from_domain": str(ar_result.header_d),
-                  "result": str(ar_result.result)
-                })
+                if ar_result.result == 'pass':
+                  self.dkim_valid = True
           else:
             logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
               "/HDR: Ignoring authentication results of {0}".format(ar.authserv_id)
@@ -267,50 +266,22 @@ class ExOTAMilter(Milter.Base):
       logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
         "/EOM: 5322.from_domain={0} dkim_auth=enabled".format(self.hdr_from_domain)
       )
-      if len(self.dkim_results) > 0:
-        for dkim_result in self.dkim_results:
-          if dkim_result['from_domain'] == self.hdr_from_domain:
-            logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
-              "/EOM: Found DKIM authentication result for {0}/{1}".format(
-                self.hdr_from_domain, dkim_result['selector']
-              )
-            )
-            if dkim_result['result'] == 'pass':
-              logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-                "/EOM: dkim_selector={0} result=pass".format(dkim_result['selector'])
-              )
-              self.dkim_valid = True
-              continue
-            else:
-              logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-                "/EOM: dkim_selector={0} result=fail".format(dkim_result['selector'])
-              )
+      if self.dkim_valid:
+        logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
+          "/EOM: Found valid DKIM authentication result for 5322.from_domain={0}".format(
+            self.hdr_from_domain
+          )
+        )
       else:
         logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-          "/EOM: No DKIM authentication results (AR headers) found - action=reject"
+          "/EOM: No valid DKIM authentication result found for 5322.from_domain={0}".format(
+            self.hdr_from_domain
+          )
         )
         return self.smfir_reject(
           queue_id = self.getsymval('i'),
-          reason = 'No DKIM authentication results found'
+          reason = 'No valid DKIM authentication results found'
         )
-      if self.dkim_valid == False:
-        logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-          "/EOM: DKIM authentication failed - action=reject"
-        )
-        return self.smfir_reject(
-          queue_id = self.getsymval('i'),
-          reason = 'DKIM failed'
-        )
-    if g_milter_dkim_enabled:
-      logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-        "/EOM: Tenant authentication successful (dkim_enabled={0})".format(
-          str(policy.is_dkim_enabled())
-        )
-      )
-    else:
-      logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-        "/EOM: Tenant successfully authenticated"
-      )
     
     # Delete all existing X-ExOTA-Authentication-Results headers
     for i in range(self.xar_hdr_count, 0, -1):
@@ -333,11 +304,22 @@ class ExOTAMilter(Milter.Base):
           )
         )
         logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) +
-        "/EOM: AR-header added"
+          "/EOM: AR-header added"
         )
       except Exception as e:
         logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
-        "/EOM: addheader(AR) failed: {0}".format(str(e))
+          "/EOM: addheader(AR) failed: {0}".format(str(e))
+        )
+
+    if g_milter_dkim_enabled:
+      logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
+        "/EOM: Tenant successfully authorized (dkim_enabled={0})".format(
+          str(policy.is_dkim_enabled())
+        )
+      )
+    else:
+      logging.info(self.mconn_id + "/" + str(self.getsymval('i')) +
+        "/EOM: Tenant successfully authorized"
       )
     return self.smfir_continue()