diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 0000000..6df829c --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "python.pythonPath": "/home/dominik/src/git/ExOTA-Milter/venv/bin/python3" +} \ No newline at end of file diff --git a/README.md b/README.md index 29b9bf7..1a585c8 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,93 @@ -# ExOTA-Milter -Exchange Online Tenant Autorisation Milter (Mail-Filter) +# ExOTA-Milter - Exchange Online Tenant Authorisation Milter (Mail-Filter) + +The **ExOTA-[Milter](https://en.wikipedia.org/wiki/Milter)** application is written in python3 and derives from **[sdgathman´s pymilter](https://github.com/sdgathman/pymilter)**. + +# Abstract/problem/motivation +Fact is that more and more companies are migrating their Outlook/Exchange environments to the [Microsoft cloud](https://www.microsoft.com/microsoft-365). + +The **ExOTA-Milter** is kind of an authentication helper which plays an essential as well as security-related role in the so called [*smarthost connector for Microsoft Exchange-Online*](https://docs.microsoft.com/de-de/exchange/mail-flow-best-practices/use-connectors-to-configure-mail-flow/set-up-connectors-to-route-mail) scenario. The milter tries to close the gap of missing [client SMTP-Auth capability](https://tools.ietf.org/html/rfc4954) of MS Exchange-Online when there´s a need to relay emails over a third party smarthost. According to this the MS Exchange-Online platform expects the smarthost to accept emails without any further authentication of users/tenants, which can lead to many misconfigured SMTP-relays! + +So, the question is: *How can an Exchange-Online user/tenant be identified by a smarthost?* + +# Identification possibilities provided by Microsoft +## Client-IP ranges specified by Microsoft +Microsoft provides an ACL as [JSON file (ID: 10)](https://endpoints.office.com/endpoints/worldwide?clientrequestid=b10c5ed1-bad1-445f-b386-b919946339a7), which looks like this. +``` +[...] +{ + "id": 10, + "serviceArea": "Exchange", + "serviceAreaDisplayName": "Exchange Online", + "urls": [ + "*.mail.protection.outlook.com" + ], + "ips": [ + "40.92.0.0/15", + "40.107.0.0/16", + "52.100.0.0/14", + "104.47.0.0/17", + "2a01:111:f400::/48", + "2a01:111:f403::/48" + ], + "tcpPorts": "25", + "expressRoute": true, + "category": "Allow", + "required": true + } +[...] +``` + +The problem of this IP based ACL is that many other Exchange-Online customers/tenants are sending from the same IP-ranges as well! **This means that many smarthost configured to relay mails comming from Exchange-Online tends to be an open relay (for Microsoft customers) unless additional authentication mechanism on a higher layer than IP takes place! IP-address based ACLs are definitely not the right way to achieve this!** + +## x509 client certificate presented by Exchange-Online +The Exchange-Online platform also *presents* a x509 client certificate to identitfy onself to the smarthost. Taking a closer look at the received header we´ll notice that the certificates common name (CN) *mail.protection.outlook.com* is not realy tenant specific. Although the certificate provides additional security regarding the identity of the client system, it does not provide identity regarding the tenant. **IMHO that´s stil not enough to permit relaying!** +``` +Received: from DEU01-FR2-obe.outbound.protection.outlook.com (mail-fr2deu01lp2173.outbound.protection.outlook.com [104.47.11.173]) + (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) + (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) + by some.secure.smarthost.example.com (Postfix) with ESMTPS id 4CjqCQ2WRCzGjg6 + for ; Sat, 28 Nov 2020 12:34:26 +0100 (CET) +``` +Client certificate verification is the job of the underlying MTA. So the **ExOTA-Milter** does not validate the client certificate itself, but it can be enabled (disabled per default) to match for the *expected* client CN. The emailserver [Postfix](http://postfix.org), for example, only provides the client CN to the milter API if the client certificate was successfully validated. Otherwise the **ExOTA-Milter** will not *see* the client CN over the milter protocol ([postfix milter macro](http://www.postfix.org/MILTER_README.html) `{cert_subject}`) which results in a milter reject action. + +## DKIM - DomainKey Identified Message +Nevertheless, as [Microsoft supports DKIM-signing for outbound email traffic](https://docs.microsoft.com/de-de/microsoft-365/security/office-365-security/use-dkim-to-validate-outbound-email?view=o365-worldwide) the **ExOTA-Milter** can be used to authenticate sending tenants, respectively their sender domains, based on the cryptographic capabilities of [DKIM](https://tools.ietf.org/html/rfc6376). In fact the **ExOTA-Milter** does not validate the DKIM-signatures itself. Instead it simply parses DKIM-specific *Authentication-Results* headers produced by any previously DKIM-validating milter (like [OpenDKIM](http://www.opendkim.org/), [Rspamd](https://rspamd.com/) or [AMavis](https://www.ijs.si/software/amavisd/)) in the chain. I personally prefer OpenDKIM as it´s lightweight and fully focused on DKIM. + +*DKIM-Signature* headers appended by the Exchange-Online platform look like this: +``` +[...] +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=tenantdomain.onmicrosoft.com; s=selector1-tenantdomain-onmicrosoft-com; + h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; + bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=; + b=DYTLJtLFjvVrSZtZQagTwuEe5PQYqrNGi7hR5bkhO[...snip...] +[...] +``` +*Authentication-Results* headers provided by OpenDKIM (signature validated) look like this: +``` +[...] +Authentication-Results: trusted.dkim.validating.relay; dkim=pass header.d=tenantdomain.onmicrosoft.com header.s=selector1-tenantdomain-onmicrosoft-com header.b=mmmjFpv8" +[...] +``` + +## X-MS-Exchange-CrossTenant-Id header +Further each Microsoft Exchange-Online tenant has a unique tenant-ID in form of a UUID ([RFC 4122](https://tools.ietf.org/html/rfc4122)). **ExOTA-Milter** determines the tenant-ID from the *X-MS-Exchange-CrossTenant-Id* email header and uses it as a *mandatory* authentication factor. +``` +[...] +X-MS-Exchange-CrossTenant-Id: +[...] +``` + +# The solution +So, *how can an Exchange-Online user/tenant be identified by a third party smarthost?* + +Finally it´s the combination of all of the above discussed aspects which may result in a robust-enough smarthost setup used by the Exchange-Online platform: +* restriction of client IPs via ACL (MTA) +* verification of Microsoft´s x509 client certificate (MTA) +* matching for client certificate´s CN (ExOTA-Milter) +* verification of DKIM signatures providing *Authentication-Results* header (another milter) +* consideration of DKIM verification results per sender domain (ExOTA-Milter) +* matching for tenant-id provided in *X-MS-Exchange-CrossTenant-Id* header (ExOTA-Milter) + +# How to start? +First of all please take a look at how to set up the testing environment, which is described [here](tests/README.md) \ No newline at end of file diff --git a/app/exota-milter.py b/app/exota-milter.py new file mode 100644 index 0000000..6c18d06 --- /dev/null +++ b/app/exota-milter.py @@ -0,0 +1,349 @@ +import Milter +import sys +import traceback +import os +import logging +import string +import random +import re +import email.utils +import authres +import json +from policy import ( + ExOTAPolicyException, ExOTAPolicyNotFoundException, + ExOTAPolicyBackendJSON, ExOTAPolicy +) + +# Globals with mostly senseless defaults ;) +g_milter_name = 'exota-milter' +g_milter_socket = '/socket/' + g_milter_name +g_milter_reject_message = 'Security policy violation!' +g_milter_tmpfail_message = 'Service temporarily not available! Please try again later.' +g_re_domain = re.compile(r'^.*@(\S+)$', re.IGNORECASE) +g_loglevel = logging.INFO +g_milter_dkim_enabled = False +g_milter_trusted_authservid = 'invalid' +g_milter_policy_source = 'file' +g_milter_policy_file = None +g_milter_policy_backend = None +g_milter_x509_enabled = False +g_milter_x509_trusted_cn = 'mail.protection.outlook.com' + +class ExOTAMilter(Milter.Base): + # Each new connection is handled in an own thread + def __init__(self): + self.reset_milter() + + def reset_milter(self): + self.client_ip = None + self.hdr_from = None + self.hdr_from_domain = None + self.hdr_tenant_id = None + self.hdr_tenant_id_count = 0 + self.dkim_results = [] + self.dkim_valid = False + self.x509_client_valid = False + # https://stackoverflow.com/a/2257449 + self.mconn_id = g_milter_name + ': ' + ''.join( + random.choice(string.ascii_lowercase + string.digits) for _ in range(8) + ) + logging.debug(self.mconn_id + " reset_milter()") + + # Not registered/used callbacks + @Milter.nocallback + def connect(self, IPname, family, hostaddr): + return Milter.CONTINUE + @Milter.nocallback + def hello(self, heloname): + return Milter.CONTINUE + @Milter.nocallback + def eoh(self): + return Milter.CONTINUE + @Milter.nocallback + def body(self, chunk): + return Milter.CONTINUE + + # Mandatory callback + def envfrom(self, mailfrom, *str): + # Instance member values remain within reused SMTP-connections! + if self.client_ip is not None: + # Milter connection reused! + logging.debug(self.mconn_id + "/FROM connection reused!") + self.reset_milter() + self.client_ip = self.getsymval('{client_addr}') + if self.client_ip is None: + logging.error(self.mconn_id + " FROM exception: could not retrieve milter-macro ({client_addr})!") + self.setreply('550','5.7.1', g_milter_tmpfail_message) + return Milter.REJECT + else: + logging.debug(self.mconn_id + "/FROM client_ip={0}".format(self.client_ip)) + return Milter.CONTINUE + + # Mandatory callback + def envrcpt(self, to, *str): + logging.debug(self.mconn_id + "/RCPT 5321.rcpt={0}".format(to)) + return Milter.CONTINUE + + def header(self, name, hval): + logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) + + "/HDR: Header: {0}, Value: {1}".format(name, hval) + ) + + # Parse RFC-5322-From header + if(name == "From"): + hdr_5322_from = email.utils.parseaddr(hval) + self.hdr_from = hdr_5322_from[1].lower() + m = re.match(g_re_domain, self.hdr_from) + if m is None: + logging.error(self.mconn_id + "/" + str(self.getsymval('i')) + "/HDR " + + "Could not determine domain-part of 5322.from=" + self.hdr_from + ) + self.setreply('450','4.7.1', g_milter_tmpfail_message) + return Milter.TEMPFAIL + self.hdr_from_domain = m.group(1) + logging.info(self.mconn_id + "/" + str(self.getsymval('i')) + + "/HDR: 5322.from={0}, 5322.from_domain={1}".format( + self.hdr_from, self.hdr_from_domain + ) + ) + + # Parse non-standardized X-MS-Exchange-CrossTenant-Id header + elif(name == "X-MS-Exchange-CrossTenant-Id"): + self.hdr_tenant_id_count += 1 + self.hdr_tenant_id = hval.lower() + logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) + + "/HDR: Tenant-ID: {0}".format(self.hdr_tenant_id) + ) + + # Parse RFC-7601 Authentication-Results header + elif(name == "Authentication-Results"): + if g_milter_dkim_enabled == True: + ar = None + try: + ar = authres.AuthenticationResultsHeader.parse( + "{0}: {1}".format(name, hval) + ) + if ar.authserv_id == g_milter_trusted_authservid: + for ar_result in ar.results: + if ar_result.method == 'dkim': + self.dkim_results.append({ + "selector": str(ar_result.header_s), + "from_domain": str(ar_result.header_d), + "result": str(ar_result.result) + }) + else: + logging.debug(self.mconn_id + "/" + str(self.getsymval('i')) + + "/HDR: Ignoring authentication results of {0}".format(ar.authserv_id) + ) + except: + logging.error(self.mconn_id + "/" + str(self.getsymval('i')) + + "/HDR: AR-parse exception: " + traceback.format_exc() + ) + return Milter.CONTINUE + + # EOM is mandatory as well and thus always called by MTA + def eom(self): + # Here in EoM the final policy logic happens. + + # Check if client certificate CN matches trusted CN + if g_milter_x509_enabled: + cert_subject = self.getsymval('{cert_subject}') + if cert_subject is None: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: No trusted x509 client CN found - action=reject" + ) + self.setreply('550','5.7.1', g_milter_tmpfail_message) + return Milter.REJECT + else: + if g_milter_x509_trusted_cn.lower() == cert_subject.lower(): + self.x509_client_valid = True + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: Trusted x509 client CN {0}".format(cert_subject) + ) + else: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM Untrusted x509 client CN {0} - action=reject".format(cert_subject) + ) + self.setreply('550','5.7.1', g_milter_tmpfail_message) + return Milter.REJECT + + if self.hdr_from is None: + logging.error(self.mconn_id + "/" + self.getsymval('i') + + "/EOM exception: could not determine 5322.from header - action=reject" + ) + self.setreply('550','5.7.1', g_milter_tmpfail_message) + return Milter.REJECT + + # Get policy for 5322.from_domain + policy = None + try: + policy = g_milter_policy_backend.get(self.hdr_from_domain) + logging.debug(self.mconn_id + "/" + self.getsymval('i') + + "/EOM Policy for 5322.from_domain={0} fetched from backend".format(self.hdr_from_domain) + ) + except (ExOTAPolicyException, ExOTAPolicyNotFoundException) as e: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM {0}".format(e.message) + ) + self.setreply('550','5.7.1', g_milter_tmpfail_message) + return Milter.REJECT + + if self.hdr_tenant_id is None: + logging.error(self.mconn_id + "/" + self.getsymval('i') + + "/EOM exception: could not determine X-MS-Exchange-CrossTenant-Id - action=reject" + ) + self.setreply('550','5.7.1', g_milter_reject_message) + return Milter.REJECT + if self.hdr_tenant_id_count > 1: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: More than one tenant-IDs for {0} found - action=reject".format( + self.hdr_from_domain + ) + ) + self.setreply('550','5.7.1', g_milter_reject_message) + return Milter.REJECT + if self.hdr_tenant_id == policy.get_tenant_id(): + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: tenant_id={0} status=match".format(self.hdr_tenant_id) + ) + else: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: tenant_id={0} status=no_match - action=reject".format( + self.hdr_tenant_id + ) + ) + self.setreply('550','5.7.1', g_milter_reject_message) + return Milter.REJECT + + if g_milter_dkim_enabled and policy.is_dkim_enabled(): + logging.debug(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: 5322.from_domain={0} dkim_auth=enabled".format(self.hdr_from_domain) + ) + if len(self.dkim_results) > 0: + for dkim_result in self.dkim_results: + if dkim_result['from_domain'] == self.hdr_from_domain: + logging.debug(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: Found DKIM authentication result for {0}/{1}".format( + self.hdr_from_domain, dkim_result['selector'] + ) + ) + if dkim_result['result'] == 'pass': + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: dkim_selector={0} result=pass".format(dkim_result['selector']) + ) + self.dkim_valid = True + continue + else: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: dkim_selector={0} result=fail".format(dkim_result['selector']) + ) + else: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: No DKIM authentication results (AR headers) found - action=reject" + ) + self.setreply('550','5.7.1', g_milter_reject_message) + return Milter.REJECT + if self.dkim_valid == False: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: DKIM authentication failed - action=reject" + ) + self.setreply('550','5.7.1', g_milter_reject_message) + return Milter.REJECT + + if g_milter_dkim_enabled: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: Tenant authentication successful (dkim_enabled={0})".format( + str(policy.is_dkim_enabled()) + ) + ) + else: + logging.info(self.mconn_id + "/" + self.getsymval('i') + + "/EOM: Tenant successfully authenticated" + ) + return Milter.CONTINUE + + def abort(self): + # Client disconnected prematurely + logging.debug(self.mconn_id + "/ABORT") + return Milter.CONTINUE + + def close(self): + # Always called, even when abort is called. + # Clean up any external resources here. + logging.debug(self.mconn_id + "/CLOSE") + return Milter.CONTINUE + +if __name__ == "__main__": + if 'LOG_LEVEL' in os.environ: + if re.match(r'^info$', os.environ['LOG_LEVEL'], re.IGNORECASE): + g_loglevel = logging.INFO + elif re.match(r'^warn|warning$', os.environ['LOG_LEVEL'], re.IGNORECASE): + g_loglevel = logging.WARN + elif re.match(r'^error$', os.environ['LOG_LEVEL'], re.IGNORECASE): + g_loglevel = logging.ERROR + elif re.match(r'debug', os.environ['LOG_LEVEL'], re.IGNORECASE): + g_loglevel = logging.DEBUG + logging.basicConfig( + filename=None, # log to stdout + format='%(asctime)s: %(levelname)s %(message)s', + level=g_loglevel + ) + if 'MILTER_NAME' in os.environ: + g_milter_name = os.environ['MILTER_NAME'] + if 'MILTER_SOCKET' in os.environ: + g_milter_socket = os.environ['MILTER_SOCKET'] + if 'MILTER_REJECT_MESSAGE' in os.environ: + g_milter_reject_message = os.environ['MILTER_REJECT_MESSAGE'] + logging.info("ENV[MILTER_REJECT_MESSAGE]: {0}".format(g_milter_reject_message)) + if 'MILTER_TMPFAIL_MESSAGE' in os.environ: + g_milter_tmpfail_message = os.environ['MILTER_TMPFAIL_MESSAGE'] + logging.info("ENV[MILTER_TMPFAIL_MESSAGE]: {0}".format(g_milter_tmpfail_message)) + if 'MILTER_DKIM_ENABLED' in os.environ: + g_milter_dkim_enabled = True + if 'MILTER_TRUSTED_AUTHSERVID' in os.environ: + g_milter_trusted_authservid = os.environ['MILTER_TRUSTED_AUTHSERVID'].lower() + logging.info("ENV[MILTER_TRUSTED_AUTHSERVID]: {0}".format(g_milter_trusted_authservid)) + else: + logging.error("ENV[MILTER_TRUSTED_AUTHSERVID] is mandatory!") + sys.exit(1) + logging.info("ENV[MILTER_DKIM_ENABLED]: {0}".format(g_milter_dkim_enabled)) + if 'MILTER_X509_ENABLED' in os.environ: + g_milter_x509_enabled = True + if 'MILTER_X509_TRUSTED_CN' in os.environ: + g_milter_x509_trusted_cn = os.environ['MILTER_X509_TRUSTED_CN'] + logging.info("ENV[MILTER_X509_TRUSTED_CN]: {0}".format(g_milter_x509_trusted_cn)) + logging.info("ENV[MILTER_X509_ENABLED]: {0}".format(g_milter_x509_enabled)) + if 'MILTER_POLICY_SOURCE' in os.environ: + g_milter_policy_source = os.environ['MILTER_POLICY_SOURCE'] + logging.info("ENV[MILTER_POLICY_SOURCE]: {0}".format(g_milter_policy_source)) + if g_milter_policy_source == 'file': + if 'MILTER_POLICY_FILE' in os.environ: + g_milter_policy_file = os.environ['MILTER_POLICY_FILE'] + logging.info("ENV[MILTER_POLICY_FILE]: {0}".format(g_milter_policy_file)) + try: + g_milter_policy_backend = ExOTAPolicyBackendJSON(g_milter_policy_file) + logging.info("JSON policy backend initialized") + except ExOTAPolicyException as e: + logging.error("Policy backend error: {0}".format(e.message)) + sys.exit(1) + else: + logging.error("ENV[MILTER_POLICY_FILE] is mandatory!") + sys.exit(1) + elif g_milter_policy_source == 'ldap': + logging.debug("LDAP-Backend not supported yet!") + sys.exit(1) + else: + logging.debug("Unsupported backend: {0}!".format(g_milter_policy_source)) + sys.exit(1) + try: + timeout = 600 + # Register to have the Milter factory create instances of your class: + Milter.factory = ExOTAMilter + logging.info("Startup " + g_milter_name + + "@socket: " + g_milter_socket + ) + Milter.runmilter(g_milter_name,g_milter_socket,timeout,True) + logging.info("Shutdown " + g_milter_name) + except: + logging.error("MAIN-EXCEPTION: " + traceback.format_exc()) + sys.exit(1) \ No newline at end of file diff --git a/app/policy.py b/app/policy.py new file mode 100644 index 0000000..356735b --- /dev/null +++ b/app/policy.py @@ -0,0 +1,101 @@ +import json +import traceback +import re + +class ExOTAPolicyException(Exception): + def __init__(self, message): + self.message = message + +class ExOTAPolicyNotFoundException(ExOTAPolicyException): + pass + +class ExOTAPolicyInvalidException(ExOTAPolicyException): + pass + +class ExOTAPolicy(): + def __init__(self, policy_dict): + self.tenant_id = policy_dict['tenant_id'] + self.dkim_enabled = policy_dict['dkim_enabled'] + + def get_tenant_id(self): + return self.tenant_id + + def is_dkim_enabled(self): + return self.dkim_enabled + + @staticmethod + def check_policy(policy_dict): + if 'tenant_id' not in policy_dict: + raise ExOTAPolicyInvalidException( + "Policy must have a 'tenant_id' attribute!" + ) + else: + if policy_dict['tenant_id'] == '': + raise ExOTAPolicyInvalidException( + "'tenant_id' must not be empty!" + ) + if re.match(r'^.*\s+.*$', policy_dict['tenant_id']): + raise ExOTAPolicyInvalidException( + "'tenant_id' must not contain whitespace characters!" + ) + if 'dkim_enabled' not in policy_dict: + raise ExOTAPolicyInvalidException( + "Policy must have a 'dkim_enabled' attribute!" + ) + else: + if not isinstance(policy_dict['dkim_enabled'], bool): + raise ExOTAPolicyInvalidException( + "'dkim_enabled'({0}) must be boolean!".format(policy_dict['dkim_enabled']) + ) + +class ExOTAPolicyBackend(): + type = None + def __init__(self): + pass + def get(self, from_domain): + pass + +########## JSON file +class ExOTAPolicyBackendJSON(ExOTAPolicyBackend): + type = 'json' + def __init__(self, file_path): + self.policies = None + try: + with open(file_path, 'r') as policy_file: + self.policies = json.load(policy_file) + policy_file.close() + # validate policy + for policy in self.policies: + try: + ExOTAPolicy.check_policy(self.policies[policy]) + except ExOTAPolicyInvalidException as e: + raise ExOTAPolicyException( + "Policy {0} is invalid: {1}".format(policy, e.message) + ) from e + except json.decoder.JSONDecodeError as e: + raise ExOTAPolicyException( + "JSON-error in policy file: " + str(e) + ) from e + except Exception as e: + raise ExOTAPolicyException( + "Error reading policy file: " + traceback.format_exc() + ) from e + + def get(self, from_domain): + try: + return ExOTAPolicy(self.policies[from_domain]) + except KeyError as e: + raise ExOTAPolicyNotFoundException( + "Policy for from_domain={0} not found".format(from_domain) + ) from e + except Exception as e: + raise ExOTAPolicyException( + "Error fetching policy for {0}: {1}".format( + from_domain, traceback.format_exc() + ) + ) from e + +########## LDAP +class ExOTAPolicyBackendLDAP(ExOTAPolicyBackendJSON): + type = 'ldap' + pass \ No newline at end of file diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..6d7adca --- /dev/null +++ b/requirements.txt @@ -0,0 +1,3 @@ +authres==1.2.0 +pkg-resources==0.0.0 +pymilter==1.0.4 diff --git a/samples/exo_validator.eml b/samples/exo_validator.eml new file mode 100644 index 0000000..8fa80bd --- /dev/null +++ b/samples/exo_validator.eml @@ -0,0 +1,64 @@ +Return-Path: <> +Received: from DEU01-FR2-obe.outbound.protection.outlook.com (mail-fr2deu01lp2173.outbound.protection.outlook.com [104.47.11.173]) + (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) + (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) + by outbound.connector.blahblubb.de (Postfix) with ESMTPS id 4CjqCQ2WRCzGjg6 + for ; Sat, 28 Nov 2020 12:34:26 +0100 (CET) +ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; + b=RuGSfIN1OzQHDqrF0erLAHZ3fyhtmoE5Sllj+Qp6CtbcNUkkmdhR44b8capz/J1mBpyb13udY1mhkPZCK1Cmt+mpg9yFXgkv5BxY+dV9647Fq+MboUE60Psn84d4vXFvyrWDrFW1jWZi7/NdXhjLcCqTHpAzDaRfAOfGhG/VWYJAXnD/EBpCzPfd8hh9ZOONI2UN2HQfRnx0P3WXyeVSGilP4RGPdmcCZV5ZzpjlQoKUshjq293+ZltXaeKfF/LHGX0yScHhKO2f9O+qY3hnH0P+NGwFvhIky3IyszfxpANaJnz2Jpp0sK1W16rwGSTI2gl9bpJsj+wzKLGkJV75+Q== +ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; + s=arcselector9901; + h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; + bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=; + b=go8dFv6srV3NnETxQxaANld1if9BOsIgrhjefC4WkRrrgwEjZSNnm9DyO+GC2ZZo60At5JHOVLjqN9kjz2pFdAG0qnFEj3Wx/6NnuTfBUk0n4s32RoFuhADu8BC+aOU9Ec909uu2QQ9ucEMiVSjuyQ3QpGS5DR0yCAZLZ12B61hmoMgkXJ9ah6rluUV4GeMGKTsUn16u6mrJycXp0OoD4n19JomPpQo5o8gouK3Zz4F7DxX4lshNJ+VCsOznqS+FI4rQ2LSyU8Y0AZa9clyCSN94AJa6K0TiDgQ/gLZEWsZ1tZkgPrdMlyqi58ONW/dNQ7lyrEFz6deB4YmsusJPbQ== +ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none + action=none header.from=lalalulu.onmicrosoft.com; dkim=none (message not + signed); arc=none +DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; + d=lalalulu.onmicrosoft.com; s=selector1-lalalulu-onmicrosoft-com; + h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; + bh=KWHUKEKZxeQyFbKe45TV1PJMC2XrVCMTFkdwSYWR6o0=; + b=DYTLJtLFjvVrSZtZQagTwuEe5PQYqrNGi7hR5bkhO+GYUV4dcQZnDO4hAPzJkOWhz8JCVJ+/yt5K8L/exegk80g9m0GJjZzJBxMy0ZE/7wg8yqiHNE+iQqWhJLtwsD23kx2+09G5dBSDI1QVqFKkL0YKBWVffSuXi+tjM4/BztffZ7ok7XZdKCFfKzK3TLdiAWYTRIp1214zdnIE0CLBhnOIWC4gnML2fXsVZsWb/CMgaW0vBsZGI/yaSivaNFPZloSb0/sEnMFMEbv2GXt9mN913M0thwCi/+NLwzaW6TNlw2Vz7l4SGRVvciGaa4s2sFnJ0ANMD2u5qBbJ8j8Z0w== +Authentication-Results: blahblubb.de; dkim=none (message not signed) + header.d=none;blahblubb.de; dmarc=none action=none + header.from=lalalulu.onmicrosoft.com; +Received: from AM6P193CA0087.EURP193.PROD.OUTLOOK.COM (2603:10a6:209:88::28) + by BEXP281MB0216.DEUP281.PROD.OUTLOOK.COM (2603:10a6:b10:6::12) with + Microsoft SMTP Server (version=TLS1_2, + cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.6; Sat, 28 Nov + 2020 11:34:25 +0000 +Received: from BE0P281MB0257.DEUP281.PROD.OUTLOOK.COM + (2603:10a6:209:88:cafe::a2) by AM6P193CA0087.outlook.office365.com + (2603:10a6:209:88::28) with Microsoft SMTP Server (version=TLS1_2, + cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3611.20 via Frontend + Transport; Sat, 28 Nov 2020 11:34:24 +0000 +From: O365ConnectorValidation@lalalulu.onmicrosoft.com +Date: Sat, 28 Nov 2020 11:34:24 +0000 +Message-Id: +To: some.recipient@example.org +Subject: Test email for connector validation +MIME-Version: 1.0 +Content-Type: text/plain; charset=us-ascii +X-MS-PublicTrafficType: Email +X-MS-Office365-Filtering-Correlation-Id: abcd1234-abcd-471a-1234-08d893918edd +X-MS-TrafficTypeDiagnostic: BEXP281MB0216: +X-Microsoft-Antispam-PRVS: + +X-MS-Oob-TLC-OOBClassifiers: OLM:2733; +X-MS-Exchange-SenderADCheck: 1 +X-Microsoft-Antispam: BCL:0; +X-Microsoft-Antispam-Message-Info: + P2dut4iALZ4EsHFmDE6p0OBg/Q4PvbmhUGI6BnGbHo/u7Vza6tyXE6BPK0VrJQ8WnCYXNx7lEKtiZs8nakJ9EghgxvFRNuYyRBJcGAdlN2TJAb2/7Wp5m7vzuGp1JJhES0RC/hypLDL8miRoP1xYl/pQHZVUGczSddujsZT6im0EgDJvAB0L1vzyKvZJ1QH3vTWDKMAgetlQHiPvCfzZmUgY92g1+sfF9UwGTRXDj8cd83H+TLI7GL8kZF1H219l+DLDiZ3u+qUdprwMn9XDEBljZpczY8BhiFdmnbyJ26ePVNa5JluRboz2Gfaa6GZE+ar8FyKtepxFOyNlI+hyL/vcWNwmnjL+pyYFVPPHnODjxu8JixWg00ThTUiZbclJ +X-Forefront-Antispam-Report: + CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BE0P281MB0257.DEUP281.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(376002)(346002)(39830400003)(34036004)(366004)(396003)(136003)(31686004)(78352004)(6916009)(508600001)(5660300002)(42882007)(2906002)(8936002)(558084003)(31696002)(17440700003)(316002)(9686003)(85236043)(68406010)(8676002)(83380400001)(16130700016)(100380200003)(20230700015);DIR:OUT;SFP:1501; +X-OriginatorOrg: lalalulu.onmicrosoft.com +X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2020 11:34:24.7460 + (UTC) +X-MS-Exchange-CrossTenant-Network-Message-Id: abcd1234-abcd-471a-1234-08d893918edd +X-MS-Exchange-CrossTenant-AuthSource: AM6P193CA0087.EURP193.PROD.OUTLOOK.COM +X-MS-Exchange-CrossTenant-AuthAs: Internal +X-MS-Exchange-CrossTenant-Id: 1234abcd-18c5-45e8-88de-123456789abc +X-MS-Exchange-CrossTenant-FromEntityHeader: Internet +X-MS-Exchange-Transport-CrossTenantHeadersStamped: BEXP281MB0216 + +This test email message was sent from Office 365 to check that email can be delivered to you using your new or modified connector. No need to reply. \ No newline at end of file diff --git a/samples/multiple_from.eml b/samples/multiple_from.eml new file mode 100644 index 0000000..1d24578 --- /dev/null +++ b/samples/multiple_from.eml @@ -0,0 +1,2 @@ +From: from2@example.com +From: from1@example.org \ No newline at end of file diff --git a/snippets/parse_5322_headers.py b/snippets/parse_5322_headers.py new file mode 100644 index 0000000..0e19435 --- /dev/null +++ b/snippets/parse_5322_headers.py @@ -0,0 +1,16 @@ +import sys +import email, email.header +from email.utils import getaddresses + +f = open("../samples/exo_validator.eml", "r") +email = email.message_from_file(f) +from_hdr = email.get_all("From") +print("from_hdr: " + str(from_hdr)) +if(len(from_hdr) > 1): + print("Multiple From-headers found!") + sys.exit(1) +elif(len(from_hdr) == 1): + print("Exactly one From-header found :)") + print(from_hdr) + from_addr = getaddresses(from_hdr) + print(str(from_addr[0][1])) diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..3f87602 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,35 @@ +# Prepare testing env +First of all, please configure a python virtual environment and install all necessary python packages listed under `requirements.txt`. Go to the root-directory of this repo and +1. `python3 -m venv venv` +1. `. venv/bin/activate` +1. `pip3 install -r requirements.txt` + +It´s not realy neccessary to configure a fully functional milter-aware MTA to see **ExOTA-Milter** in action. All you need is +* a binary called `miltertest`. Under debian based distros it´s located in the `opendkim-tools` package. +* a lua-script for miltertest: `tests/miltertest.lue` +* an **ExOTA-Milter** policy JSON-file: `tests/policy.json` + +Except for the `miltertest` binary you´ll find all mandatory resources to run a test in this repo. + +``` +export LOG_LEVEL=debug +export MILTER_SOCKET=/tmp/exota-milter +export MILTER_POLICY_FILE=tests/policy.json +export MILTER_DKIM_ENABLED=yepp +export MILTER_TRUSTED_AUTHSERVID=my-auth-serv-id +export MILTER_X509_ENABLED=yepp +export MILTER_X509_TRUSTED_CN=mail.protection.outlook.com +``` + +# Shell-1: start ExOTA-Milter +``` +. venv/bin/activate +python3 app/exota-milter.py +``` + +# Shell-2: execute `miltertest` +This must be done only once: `export MILTER_SOCKET=/tmp/exota-milter` + +Execute miltertest pointing to the test script written in lua to feed the **ExOTA-Milter**: + +`miltertest -v -D socket="${MILTER_SOCKET}" -s tests/miltertest.lua` \ No newline at end of file diff --git a/tests/miltertest.lua b/tests/miltertest.lua new file mode 100644 index 0000000..5243bfb --- /dev/null +++ b/tests/miltertest.lua @@ -0,0 +1,73 @@ +-- https://mopano.github.io/sendmail-filter-api/constant-values.html#com.sendmail.milter.MilterConstants +-- http://www.opendkim.org/miltertest.8.html + +-- socket must be defined as miltertest global variable (-D) +conn = mt.connect(socket) +if conn == nil then + error "mt.connect() failed" +end + +mt.set_timeout(3) + +-- 5321.FROM + MACROS +mt.macro(conn, SMFIC_MAIL, '{client_addr}', "127.128.129.130", "i", "4CgSNs5Q9sz7SllQ", '{cert_subject}', "mail.protection.outlook.com") +if mt.mailfrom(conn, "envelope.sender@example.org") ~= nil then + error "mt.mailfrom() failed" +end +if mt.getreply(conn) ~= SMFIR_CONTINUE then + error "mt.mailfrom() unexpected reply" +end + +-- 5321.RCPT +if mt.rcptto(conn, "") ~= nil then + error "mt.rcptto() failed" +end +if mt.getreply(conn) ~= SMFIR_CONTINUE then + error "mt.rcptto() unexpected reply" +end + +-- HEADER +if mt.header(conn, "From", '"Blah Blubb" ') ~= nil then + error "mt.header(From) failed" +end +if mt.header(conn, "X-MS-Exchange-CrossTenant-Id", "1234abcd-18c5-45e8-88de-123456789abc") ~= nil then + error "mt.header(Subject) failed" +end +--if mt.header(conn, "X-MS-Exchange-CrossTenant-Id", "4321abcd-18c5-45e8-88de-blahblubb") ~= nil then +-- error "mt.header(Subject) failed" +--end +if mt.header(conn, "Authentication-Results", "another-wrong-auth-serv-id;\n dkim=fail header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then + error "mt.header(Subject) failed" +end +if mt.header(conn, "Authentication-Results", "wrong-auth-serv-id;\n dkim=pass header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then + error "mt.header(Subject) failed" +end +if mt.header(conn, "Authentication-Results", "my-auth-serv-id;\n dkim=pass header.d=lalalulu.onmicrosoft.com header.s=selector1-lalalulu-onmicrosoft-com header.b=mmmjFpv8") ~= nil then + error "mt.header(Subject) failed" +end +if mt.header(conn, "Authentication-Results", "my-auth-serv-id;\n dkim=fail header.d=lalalulu.onmicrosoft.com header.s=selector2-asdf header.b=mmmjFpv8") ~= nil then + error "mt.header(Subject) failed" +end +if mt.header(conn, "Authentication-Results", "some-validating-host;\n dkim=pass header.d=paypal.de header.s=pp-dkim1 header.b=PmTtUzer;\n dmarc=pass (policy=reject) header.from=paypal.de;\n spf=pass (some-validating-host: domain of service@paypal.de designates 173.0.84.226 as permitted sender) smtp.mailfrom=service@paypal.de") ~= nil then + error "mt.header(Subject) failed" +end + +-- EOM +if mt.eom(conn) ~= nil then + error "mt.eom() failed" +end +mt.echo("EOM: " .. mt.getreply(conn)) +if mt.getreply(conn) == SMFIR_CONTINUE then + mt.echo("EOM-continue") +elseif mt.getreply(conn) == SMFIR_REPLYCODE then + mt.echo("EOM-reject") +end + +if not mt.eom_check(conn, MT_HDRADD, "X-SOS-Milter") then + mt.echo("no header added") +else + mt.echo("X-SOS-Milter header added -> LDAP-Domain with broken SPF") +end + +-- DISCONNECT +mt.disconnect(conn) \ No newline at end of file diff --git a/tests/policy.json b/tests/policy.json new file mode 100644 index 0000000..098f7ed --- /dev/null +++ b/tests/policy.json @@ -0,0 +1,10 @@ +{ + "lalalulu.onmicrosoft.com": { + "tenant_id": "1234abcd-18c5-45e8-88de-123456789abc", + "dkim_enabled": true + }, + "asdf2.onmicrosoft.com": { + "tenant_id": "asdftasdfa", + "dkim_enabled": true + } +} \ No newline at end of file