The code below worked but in the end I implemented a filter by https://www.hackitu.de/pyimapsrt/ which I modified to my preferences – see further down
Having completed the autoresponder I thought I’d have a go at an email noise filter to get rid of all those review or feedback requests that we get for seemingly everything we buy these days. The program moves the offending email into the specified folder and flags it as read. Like the autoresponder above, it only looks at emails that have arrived today. It will get rid of emails from specified senders (just part of the sender’s email is enough) and emails with strings of keywords in their subject.
The principle is the same as the autoresponder with a credentials file passed in the command line. It needs the folder name you want to move the email to; I’m using the Archive folder but it could be your Trash etc. Forgive the repeating code blocks below, I just needed a quick and simple solution without worrying too much about global variables etc.
#! /usr/bin/python3
# Noisecatcher - David Brewerton 2023
# v 20231128.0750
# ____________________________________
import configparser, os, sys
import time, socket, imaplib, traceback, datetime
from imap_tools import AND, OR, NOT, MailBox, MailboxLoginError, MailboxLogoutError, MailMessageFlags
from email.message import EmailMessage
def get_config_file_path():
# use the configparser to bring in the mailserver credentials
if "--help" in sys.argv or "-h" in sys.argv:
display_help_text()
if "--config-path" in sys.argv and len(sys.argv) >= 3:
global config_file_path
config_file_path = sys.argv[2]
if not os.path.isfile(config_file_path):
print("Configuration file not found. Expected it at '" + config_file_path + "'.")
exit(-1)
def display_help_text():
print("Options:")
print("\t--help: Display this help information")
print("\t--config-path <path/to/config/file>: "
"Override path to config file (defaults to same directory as the script is)")
exit(0)
# *** RUN ROUTINE ***
# Initialise and read in config
config_file_path = "noisecatcher.config.ini"
get_config_file_path()
# this is so the credentials config can be passed on the command line if required
config_file = configparser.ConfigParser()
config_file.read(config_file_path, encoding="UTF-8")
myname = (config_file["login credentials"]["mailserver.myname"])
username = (config_file["login credentials"]["mailserver.username"])
password = (config_file["login credentials"]["mailserver.password"])
mailserver = (config_file["login credentials"]["mailserver.server"])
# this allows the non secure config to be passed through a fixed filename
config_file_path = "noisecatcher.config.ini"
config_file = configparser.ConfigParser()
config_file.read(config_file_path, encoding="UTF-8")
archivefolder = (config_file["archive folder"]["email.archivefolder"]).strip('\"')
mixedcasesubjectlist = tuple((config_file["email subjects"]["email.subjectlist"]).split("\n"))
subjectlist = lowercase_tuple = tuple(string.lower() for string in mixedcasesubjectlist)
mixedcasesenderlist = tuple((config_file["email subjects"]["email.senderlist"]).split("\n"))
senderlist = lowercase_tuple = tuple(string.lower() for string in mixedcasesenderlist)
# Main Loop
print(time.strftime("%a %H:%M:%S"),"Monitoring")
done = False
while not done:
connection_start_time = time.monotonic()
connection_live_time = 0.0
try:
with MailBox(mailserver).login(username, password, 'INBOX') as mailbox:
# after first connection or reconnection
# print('@',end="") # prints @ to signify a new connection with the mailbox
while connection_live_time < 29 * 60: try: # deal with any unprocessed newmails before program start for today # do not mark them as read for msg in mailbox.fetch( AND(seen=False, date=datetime.date.today(), ), mark_seen=False ): # if newmail has a partial match with a string on the config subjectlist # first set the mail to read # then move it if any(ele in msg.subject.lower() for ele in subjectlist): print("\n",time.strftime("%a %H:%M:%S"), "1>", msg.uid, '"', msg.from_values.full, ' "',
msg.subject, '" ', msg.flags,end="")
flags = (MailMessageFlags.SEEN)
mailbox.flag(mailbox.uids(AND(seen=False,uid=msg.uid)), flags, True)
mailbox.move(msg.uid,archivefolder)
print(">",msg.uid,"archived")
else:
if any(ele in msg.from_.lower() for ele in senderlist):
print("\n",time.strftime("%a %H:%M:%S"), "1>", msg.uid, '"', msg.from_values.full, ' "',
msg.subject, '" ', msg.flags,end="")
flags = (MailMessageFlags.SEEN)
mailbox.flag(mailbox.uids(AND(seen=False,uid=msg.uid)), flags, True)
mailbox.move(msg.uid,archivefolder)
print(">",msg.uid,"archived")
# now go idle until the mailserver wakes me up
responses = mailbox.idle.wait(timeout=600)
# print(time.strftime("%H:%M:%S"), 'IDLE responses:', responses)
if responses:
# mailserver has woken up and alerted us to newmail to process
# do not mark them as read
for msg in mailbox.fetch(
AND(seen=False,
date=datetime.date.today(),
),
mark_seen=False
):
# if newmail has a partial match with a string on the config subjectlist
# first set the mail to read
# then move it
if any(ele in msg.subject.lower() for ele in subjectlist):
print("\n",time.strftime("%a %H:%M:%S"), "1>", msg.uid, '"', msg.from_values.full, ' "',
msg.subject, '" ', msg.flags,end="")
flags = (MailMessageFlags.SEEN)
mailbox.flag(mailbox.uids(AND(seen=False,uid=msg.uid)), flags, True)
mailbox.move(msg.uid,archivefolder)
print(">",msg.uid,"archived")
else:
if any(ele in msg.from_.lower() for ele in senderlist):
print("\n",time.strftime("%a %H:%M:%S"), "1>", msg.uid, '"', msg.from_values.full, ' "',
msg.subject, '" ', msg.flags,end="")
flags = (MailMessageFlags.SEEN)
mailbox.flag(mailbox.uids(AND(seen=False,uid=msg.uid)), flags, True)
mailbox.move(msg.uid,archivefolder)
print(">",msg.uid,"archived")
# stop with ctrl z
except KeyboardInterrupt:
print('~KeyboardInterrupt')
done = True
break
connection_live_time = time.monotonic() - connection_start_time
# deals primarily with timeouts on the mailserver which happen frequently
# so just dust yourself down and reconnect again
except (TimeoutError, ConnectionError,
imaplib.IMAP4.abort, MailboxLoginError, MailboxLogoutError,
socket.error, socket.gaierror, socket.timeout) as e:
# print(f'{e} reconnecting...')
time.sleep(15)
The Config file
[archive folder] email.archivefolder = "INBOX.Archive" [email subjects] email.subjectlist = Please Review Review your purchase Would you recommend love to get your thoughts Help us to improve our service How was your order how was it for you How did your courier Reminder - We appreciate your feedback about your delivery experience feedback request tell us what you think know what you think how did we do Thanks for your recent purchase scams Royal Mail Collection Please review Stay safe by Would you recommend Error Correction Received Tell us about your recent call Review your experience share your thoughts about reminder to review Rate your delivery Login from a new device Sign-in email.senderlist = newsletter@email.bonusprint.co.uk domesticandgeneral promotion@aliexpress.com
Run at boot time in the pi user
crontab -e
add at the bottom…
@reboot /usr/bin/python3 -u /home/pi/noisecatcher.py --config-path /home/pi/aardvark.ini 1>>/home/pi/noiselog.txt 2>&1 &
Email Noise Filter by Hackitude
Please see https://www.hackitu.de/pyimapsrt/ for configuration information. I made slight changes and then ran the script every 5 minutes in Cron:-
- Removed the checks for IMAP server capability as my server didn’t reply with MOVE but it still did it!
- Changed mail flags on emails that met my ‘noisy’ criteria to mark them as read and added a tag to grey them out in Thunderbird Mail.
#!/usr/bin/env python3
# Author: F. Schröder https://www.hackitu.de/pyimapsrt/
# Mods by David Brewerton (DJB) 2023
"""
Server-side search and move for IMAP mailboxes.
"""
import re
import sys
import logging.config
import configparser
import argparse
from locale import setlocale, LC_ALL
from datetime import datetime, timedelta
from getpass import getpass
from imaplib import IMAP4, IMAP4_SSL
from dataclasses import dataclass, field
from typing import Tuple, List, Optional, ClassVar
class ImapSearch:
"""
Parse matches as configured into individual search criteria.
"""
_now: ClassVar[datetime] = datetime.utcnow()
@dataclass(frozen=True)
class Criterion:
command: List[str]
arguments: List[str] = field(default_factory=list)
def __init__(self) -> None:
self._criteria: List[ImapSearch.Criterion] = []
def add(self, key: str, params: List[str]) -> None:
key = key.upper()
if key in {"CC", "FROM", "TO", "SUBJECT", "TEXT"} and len(params) >= 1:
self._criteria.append(self.Criterion([key], [" ".join(params)]))
elif key in {"HEADER"} and len(params) >= 1: # with name in IMAPv4 syntax
self._criteria.append(self.Criterion([key], [params[0], " ".join(params[1:])]))
elif key in {"ALL", "SEEN", "UNSEEN"} and len(params) == 0:
self._criteria.append(self.Criterion([key], []))
elif key in {"BEFORE", "SINCE", "SENTBEFORE", "SENTSINCE"} and len(params) >= 1:
self._criteria.append(self.Criterion([key], [self._parse_time_ago(" ".join(params))]))
else:
raise ValueError(f"Unsupported search criterion '{key}' with {len(params)} arguments")
def build(self) -> List[Criterion]:
return self._criteria if len(self._criteria) else [self.Criterion(["ALL"])]
@classmethod
def _parse_time_ago(cls, s: str) -> str:
match: Optional[re.Match] = re.match("^([0-9]+) (days) ago$", s)
if match is None:
raise ValueError(f"Cannot parse time ago: '{s}'")
try:
delta: timedelta = timedelta(**{match.group(2): int(match.group(1))})
except OverflowError as e:
raise ValueError(f"Cannot parse time ago: '{s}': {str(e)}") from None
else:
return (cls._now - delta).strftime("%d-%b-%Y")
@classmethod
def from_list(cls, criteria: List[List[str]]) -> 'ImapSearch':
search: ImapSearch = ImapSearch()
for criterion in criteria:
if len(criterion):
search.add(criterion[0], criterion[1:])
return search
class Imap:
"""
Wrap IMAP4 class with capability checks, response error handling, and convenience operations.
"""
def __init__(self, host: str, port: int, user: str, password: str, utf: bool, ro: bool) -> None:
self._logger: logging.Logger = logging.getLogger(self.__class__.__name__)
self._host: str = host
self._port: int = port
self._user: str = user
self._pass: str = password
self._utf: bool = utf
self._ro: bool = ro
try:
self._imap: IMAP4 = IMAP4_SSL(host=self._host, port=self._port)
except (IMAP4.error, OSError) as e:
raise RuntimeError(f"Cannot connect to {self._host}:{self._port}: {str(e)}") from None
def __str__(self) -> str:
return f"<{self.__class__.__name__} {self._user}@{self._host}:{self._port}>"
def _quote(self, s: str) -> str:
if not self._utf and re.match("^[ -~]*$", s) is None:
raise ValueError(f"String literals not supported, cannot escape '{s}'")
return "".join(['"', s.replace('\\', '\\\\').replace('"', '\\"'), '"'])
@classmethod
def _check_response(cls, response: Tuple[str, List[Optional[bytes]]], expected: str = "OK") -> str:
if not isinstance(response, tuple) or len(response) != 2 or not isinstance(response[0], str):
raise RuntimeError("Invalid response")
if response[0] != expected:
raise RuntimeError(f"Response '{response[0]}', expected '{expected}'")
if not isinstance(response[1], list) or len(response[1]) != 1:
raise RuntimeError("Invalid response value")
if response[1][0] is None:
return ''
elif isinstance(response[1][0], bytes):
return response[1][0].decode("utf-8", errors="replace")
else:
raise RuntimeError("Invalid response value value")
def handshake(self) -> None:
# DJB commented out the capabilities checks
#self._logger.debug(f"CAPABILITIES: {', '.join(self._imap.capabilities)}")
#if "AUTH=PLAIN" not in self._imap.capabilities:
# raise RuntimeError("No plaintext login capability")
#if "MOVE" not in self._imap.capabilities:
# raise RuntimeError("No 'MOVE' capability")
#if self._utf and "ENABLE" not in self._imap.capabilities:
# raise RuntimeError("No 'ENABLE' capability for UTF8")
#if self._utf and hasattr(self._imap, '_mode_utf8'):
# # ENABLE only valid in authenticated state, so already fake it to allow i.e. special chars in passwords
# self._imap._mode_utf8() # noqa
self.login()
if self._utf:
self._enable("UTF8=ACCEPT")
def login(self) -> None:
try:
self._logger.debug("LOGIN")
self._check_response(self._imap.login(user=self._quote(self._user), password=self._pass)) # type: ignore
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot login: {str(e)}") from None
def logout(self) -> None:
try:
self._logger.debug("LOGOUT")
self._check_response(self._imap.logout(), "BYE") # type: ignore
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot logout: {str(e)}") from None
def _enable(self, capability: str) -> None:
try:
self._logger.debug(f"ENABLE '{capability}'")
self._check_response(self._imap.enable(capability))
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot enable '{capability}': {str(e)}") from None
def _select(self, mailbox: str) -> None:
try:
self._logger.debug(f"SELECT '{mailbox}'")
self._check_response(self._imap.select(mailbox=self._quote(mailbox), readonly=self._ro))
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot select mailbox '{mailbox}': {str(e)}") from None
def _search(self, criteria: List[ImapSearch.Criterion]) -> List[str]:
conditions: List[str] = [" ".join(criterion.command + [self._quote(_) for _ in criterion.arguments])
for criterion in criteria]
try:
self._logger.debug(f"SEARCH {' '.join(conditions)}")
response: str = self._check_response(self._imap.uid("SEARCH", " ".join(conditions)))
return response.split(" ") if response else []
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot search: {str(e)}") from None
def _move(self, mailbox: str, mail_ids: List[str]) -> None:
if self._ro:
return
try:
ret = (self._imap.uid("STORE",'{} "{}"'.format(",".join(mail_ids),'+FLAGS'), '(\Seen $label1)')) # DJB mark as seen and set Thunderbird tag1, ignore the return code
#
self._logger.debug(f"MOVE #{len(mail_ids)}")
self._check_response(self._imap.uid("MOVE", '{} "{}"'.format(",".join(mail_ids), mailbox)))
except (IMAP4.error, RuntimeError) as e:
raise RuntimeError(f"Cannot move: {str(e)}") from None
def search_and_move(self, from_mailbox: str, to_mailbox: str, criteria: List[ImapSearch]) -> None:
self._select(from_mailbox)
for criterion in criteria: # TODO: support OR (IMAP4) and merge moves for same mailboxes
mail_ids: List[str] = self._search(criterion.build())
# self._logger.info(f"Found {len(mail_ids)} mails in '{from_mailbox}' for '{to_mailbox}'") #DJB don't log all
if len(mail_ids):
self._move(to_mailbox, mail_ids)
self._logger.info(f"Found {len(mail_ids)} mails in '{from_mailbox}' for '{to_mailbox}'") #DJB only log found
class Config:
"""
Parse ini-style config file into dataclasses.
"""
@dataclass(frozen=True)
class Account:
name: str
host: str
port: int
user: str
password: Optional[str]
utf: bool
@dataclass(frozen=True)
class Match:
name: str
account: str
match_any: bool
match: List[List[str]]
from_mailbox: List[str]
to_mailbox: str
def __init__(self, accounts: List[Account], matches: List[Match]) -> None:
self.accounts: List[Config.Account] = accounts
self.matches: List[Config.Match] = matches
@classmethod
def _getlist(cls, val: str) -> List[str]:
return [_ for _ in [line.strip() for line in val.split("\n")] if _]
@classmethod
def _getlistlist(cls, val: str) -> List[List[str]]:
return [line.split(None) for line in cls._getlist(val)]
@classmethod
def _parse(cls, config_parser: configparser.ConfigParser) -> 'Config':
accounts: List[Config.Account] = []
matches: List[Config.Match] = []
for section in config_parser.sections():
if '.' not in section:
raise ValueError(f"Invalid section '{section}'")
else:
section_type, name = section.split(".", maxsplit=1)
try:
if section_type == "account":
accounts.append(cls.Account(
name=name,
host=config_parser.get(section, "host"),
port=config_parser.getint(section, "port"),
user=config_parser.get(section, "user"),
password=config_parser.get(section, "password", fallback=None),
utf=config_parser.getboolean(section, "utf", fallback=False)
))
elif section_type == "match":
matches.append(cls.Match(
name=name,
account=config_parser.get(section, "account"),
match_any=config_parser.getboolean(section, "match_any", fallback=False),
match=config_parser.getlistlist(section, "match", fallback=[]), # type: ignore
from_mailbox=config_parser.getlist(section, "from_mailbox"), # type: ignore
to_mailbox=config_parser.get(section, "to_mailbox")
))
else:
raise ValueError("Unknown section")
except ValueError as e:
raise ValueError(f"Cannot parse {section_type} '{name}': {str(e)}") from None
return cls(accounts, matches)
@classmethod
def from_ini(cls, filename: str) -> 'Config':
try:
config_parser: configparser.ConfigParser = configparser.ConfigParser(
converters={'list': cls._getlist, 'listlist': cls._getlistlist}
)
with open(filename, "r") as fp:
config_parser.read_file(fp)
except (OSError, configparser.Error, UnicodeDecodeError) as e:
raise RuntimeError(f"Cannot read config '{filename}': {str(e)}") from None
try:
return cls._parse(config_parser)
except ValueError as e:
raise RuntimeError(f"Cannot parse config '{filename}': {str(e)}") from None
def main(config_file: str, dry_run: bool, account_name: Optional[str], match_name: Optional[str]) -> bool:
logger: logging.Logger = logging.getLogger(main.__name__)
# parse config
try:
config: Config = Config.from_ini(config_file)
except RuntimeError as e:
logger.error(str(e))
return False
else:
logger.debug(f"Loaded {len(config.matches)} operations on {len(config.accounts)} accounts")
# process defined accounts
success: bool = True
for account in config.accounts:
if account_name is not None and account.name != account_name:
continue
# ask for password if not specified in config
password: Optional[str] = account.password
if password is None:
try:
if account.utf:
password = input(f"Password for '{account.user}' on {account.name}: ")
else:
password = getpass(f"Password for '{account.user}' on {account.name}: ")
except (KeyboardInterrupt, Warning):
pass
except (OSError, UnicodeDecodeError) as e:
logger.error(str(e))
success = False
if password is None:
logger.warning(f"Skipping '{account.name}', no password given")
continue
try:
# connect, login, and set capabilities
logger.debug(f"Connecting to {account.host}:{account.port} as '{account.user}'")
imap: Imap = Imap(host=account.host, port=account.port,
user=account.user, password=password,
utf=account.utf, ro=dry_run)
imap.handshake()
for match in config.matches:
if match.account != account.name:
continue
if match_name is not None and match.name != match_name:
continue
logger.info(f"Checking {match.name} on {account.name}")
# parse search matches
try:
search: List[ImapSearch]
if match.match_any: # OR
search = [ImapSearch.from_list([match]) for match in match.match]
else: # AND
search = [ImapSearch.from_list(match.match)]
except ValueError as e:
logger.warning(f"Cannot parse match for '{match.name}': {str(e)}")
success = False
continue
# do the search and move
try:
for mailbox in match.from_mailbox:
imap.search_and_move(mailbox, match.to_mailbox, search)
except ValueError as e:
logger.warning(f"{str(imap)}: {str(e)}")
success = False
except RuntimeError as e:
logger.error(f"{str(imap)}: {str(e)}")
success = False
logger.debug(f"Logout at {account.host}:{account.port} as '{account.user}'")
imap.logout()
except RuntimeError as e:
logger.error(str(e))
success = False
return success
def setup_logging(debug: bool) -> None:
logging.raiseExceptions = True
logging.logThreads = False
logging.logMultiprocessing = False
logging.logProcesses = False
logging.config.dictConfig({
'version': 1,
'formatters': {'standard': {
'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s',
}},
'handlers': {'default': {
'formatter': 'standard',
'class': 'logging.StreamHandler',
'stream': 'ext://sys.stderr',
}},
'loggers': {'': {
'handlers': ['default'],
'level': 'DEBUG' if debug else 'ERROR', #DJB was else INFO
'propagate': False,
}},
})
if __name__ == "__main__":
parser = argparse.ArgumentParser(description=__doc__.strip(),
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--verbose', action='store_const', const=True, default=False,
help='enable debug logging')
parser.add_argument('--dry-run', action='store_const', const=True, default=False,
help='only search, do not actually move')
parser.add_argument('--config', metavar="CONFIG.INI", type=str, default='./imapsrt.ini',
help='path to configuration file')
parser.add_argument('--limit-account', metavar="NAME", type=str, default=None,
help='only process matches for the given account name')
parser.add_argument('--limit-match', metavar="NAME", type=str, default=None,
help='only process matches with the given name')
arguments = parser.parse_args()
setlocale(LC_ALL, "C") # for strptime/strftime %b
setup_logging(arguments.verbose)
sys.exit(0 if main(arguments.config, arguments.dry_run, arguments.limit_account, arguments.limit_match) else 1)
To schedule every 5 minutes
I added the process to Cron with
crontab -e
adding this entry at the bottom…
*/5 * * * * /usr/bin/python3 -u /home/pi/imapsrt.py --config /home/pi/imapsrt.ini
Notes from Hackitude’s page
An [account.NAME] section accepts the following settings:
host- IMAP server hostname to connect to
port- IMAP server port to connect to, typically 993
user- Login username, usually the email address
password- Login password, will be prompted for at startup when omitted
utf- Enable extended UTF-8 support
Please note that the login credentials have to be valid for the plain authentication method, which most servers should support and allow for encrypted connections. If that’s not the case, a so-called app password might be supported by the mail provider as fallback mechanism, and can be used just like in any generic mail client software.
Actual matches to perform can be defined by [match.NAME] sections:
account- The name of the account this match refers to (via it’s section header)
from_mailbox- Folder or list of folders to search in
to_mailbox- Where to move matched emails to
match_any- Per default, all given matches have to succeed (AND); if enabled, any match suffices (OR)
match- List of matches, a single one or one per line, see below for the list of supported search filters
IMAP search filters
Matches are more or less passed through unchanged to the remote server, using the IMAP search command. Depending on match_any, they are treated as single criterion (false: and) or as independent ones (true: or). No quoting or escaping in the configuration file is needed/allowed, tokens are concatenated with a single space character. The following matches are recognized:
CC|FROM|TO <address>- Search for email addresses (or parts thereof) in the respective field
SUBJECT|TEXT <string>- Match the subject field or whole header/body for string occurrences
SEEN|UNSEEN- Select only seen or unseen messages, respectively
HEADER <header> [<value>]- Arbitrary header lookup (IMAPv4 only), for exampleÂ
HEADER Reply-To foo@bar.net BEFORE|SINCE|SENTBEFORE|SENTSINCE <X> days ago- Allows restriction by time, for exampleÂ
SINCE 7 days ago