472,958 Members | 1,823 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 472,958 software developers and data experts.

Modify Python Code - no idea at all

I am trying to modify the Mailman Python code to stop mapping
MIME-types and use the extension of the attachment instead. I am
pretty much clueless as to what I need to do here, but I think I have
narrowed it down to the Scrubber.py file..

If this seems like a quick step me through, I would be very
appreciative, could get you something on your Amazon wish-list (that
is me on my knees begging)..

From just my basic understanding, it looks like it does a mimemapping,
and then uses the 'best' extension for that type. We are sending .pdf
files, but they are being classified as application/octet-stream from
the mail program, and this then gets mapped to a .obj file extension
when it is being archived. This causes problems when going through
the archive.

I am attaching the code below, if anyone could give me some guidance..


# Copyright (C) 2001-2003 by the Free Software Foundation, Inc.
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
02111-1307, USA.

"""Cleanse a message for archiving.

from __future__ import nested_scopes

import os
import re
import sha
import time
import errno
import binascii
import tempfile
from cStringIO import StringIO
from types import IntType

from email.Utils import parsedate
from email.Parser import HeaderParser
from email.Generator import Generator

from Mailman import mm_cfg
from Mailman import Utils
from Mailman import LockFile
from Mailman import Message
from Mailman.Errors import DiscardMessage
from Mailman.i18n import _
from Mailman.Logging.Syslog import syslog

# Path characters for common platforms
pre = re.compile(r'[/\\:]')
# All other characters to strip out of Content-Disposition: filenames
# (essentially anything that isn't an alphanum, dot, slash, or
sre = re.compile(r'[^-\w.]')
# Regexp to strip out leading dots
dre = re.compile(r'^\.*')

BR = '<br>\n'
SPACE = ' '

from mimetypes import guess_all_extensions
except ImportError:
import mimetypes
def guess_all_extensions(ctype, strict=1):
# BAW: sigh, guess_all_extensions() is new in Python 2.3
all = []
def check(map):
for e, t in map.items():
if t == ctype:
# Python 2.1 doesn't have common_types. Sigh, sigh.
if not strict and hasattr(mimetypes, 'common_types'):
return all

def guess_extension(ctype, ext):
# mimetypes maps multiple extensions to the same type, e.g. .doc,
# and .wiz are all mapped to application/msword. This sucks for
# the best reverse mapping. If the extension is one of the giving
# mappings, we'll trust that, otherwise we'll just guess. :/
all = guess_all_extensions(ctype, strict=0)
if ext in all:
return ext
return all and all[0]

# We're using a subclass of the standard Generator because we want to
# headers in the subparts of multiparts. We use a hack -- the ctor
# skipheaders to accomplish this. It's set to true for the outer
# object, but false for all internal objects. We recognize that
# sub-Generators will get created passing only mangle_from_ and
# to the ctors.
# This isn't perfect because we still get stuff like the multipart
# but see below for how we corrupt that to our nefarious goals.
class ScrubberGenerator(Generator):
def __init__(self, outfp, mangle_from_=1, maxheaderlen=78,
Generator.__init__(self, outfp, mangle_from_=0)
self.__skipheaders = skipheaders

def _write_headers(self, msg):
if not self.__skipheaders:
Generator._write_headers(self, msg)

def safe_strftime(fmt, floatsecs):
return time.strftime(fmt, floatsecs)
except (TypeError, ValueError):
return None
def calculate_attachments_dir(mlist, msg, msgdata):
# Calculate the directory that attachments for this message will
# under. To avoid inode limitations, the scheme will be:
# archives/private/<listname>/attachments/YYYYMMDD/<msgid-hash>/<files>
# Start by calculating the date-based and msgid-hash components.
fmt = '%Y%m%d'
datestr = msg.get('Date')
if datestr:
now = parsedate(datestr)
now = time.gmtime(msgdata.get('received_time', time.time()))
datedir = safe_strftime(fmt, now)
if not datedir:
datestr = msgdata.get('X-List-Received-Date')
if datestr:
datedir = safe_strftime(fmt, datestr)
if not datedir:
# What next? Unixfrom, I guess.
parts = msg.get_unixfrom().split()
month = {'Jan':1, 'Feb':2, 'Mar':3, 'Apr':4, 'May':5,
'Jul':7, 'Aug':8, 'Sep':9, 'Oct':10, 'Nov':11,
}.get(parts[3], 0)
day = int(parts[4])
year = int(parts[6])
except (IndexError, ValueError):
# Best we can do I think
month = day = year = 0
datedir = '%04d%02d%02d' % (year, month, day)
assert datedir
# As for the msgid hash, we'll base this part on the Message-ID:
so that
# all attachments for the same message end up in the same
directory (we'll
# uniquify the filenames in that directory as needed). We use the
first 2
# and last 2 bytes of the SHA1 hash of the message id as the basis
of the
# directory name. Clashes here don't really matter too much, and
# still gives us a 32-bit space to work with.
msgid = msg['message-id']
if msgid is None:
msgid = msg['Message-ID'] = Utils.unique_message_id(mlist)
# We assume that the message id actually /is/ unique!
digest = sha.new(msgid).hexdigest()
return os.path.join('attachments', datedir, digest[:4] +

def process(mlist, msg, msgdata=None):
sanitize = mm_cfg.ARCHIVE_HTML_SANITIZER
outer = 1
if msgdata is None:
msgdata = {}
dir = calculate_attachments_dir(mlist, msg, msgdata)
charset = None
lcset = Utils.GetCharSet(mlist.preferred_language)
# Now walk over all subparts of this message and scrub out various
for part in msg.walk():
ctype = part.get_type(part.get_default_type())
# If the part is text/plain, we leave it alone
if ctype == 'text/plain':
# We need to choose a charset for the scrubbed message, so
# arbitrarily pick the charset of the first text/plain
part in the
# message.
if charset is None:
charset = part.get_content_charset(lcset)
elif ctype == 'text/html' and isinstance(sanitize, IntType):
if sanitize == 0:
if outer:
raise DiscardMessage
del part['content-type']
part.set_payload(_('HTML attachment scrubbed and
# Adding charset arg and removing
# sets content-type to text/plain
elif sanitize == 2:
# By leaving it alone, Pipermail will automatically
escape it
elif sanitize == 3:
# Pull it out as an attachment but leave it unescaped.
# is dangerous, but perhaps useful for heavily
# lists.
omask = os.umask(002)
url = save_attachment(mlist, part, dir,
del part['content-type']
An HTML attachment was scrubbed...
URL: %(url)s
"""), lcset)
# HTML-escape it and store it as an attachment, but
make it
# look a /little/ bit prettier. :(
payload = Utils.websafe(part.get_payload(decode=1))
# For whitespace in the margin, change spaces into
# non-breaking spaces, and tabs into 8 of those. Then
use a
# mono-space font. Still looks hideous to me, but
then I'd
# just as soon discard them.
def doreplace(s):
return s.replace(' ', '&nbsp;').replace('\t',
lines = [doreplace(s) for s in payload.split('\n')]
payload = '<tt>\n' + BR.join(lines) + '\n</tt>\n'
# We're replacing the payload with the decoded payload
so this
# will just get in the way.
del part['content-transfer-encoding']
omask = os.umask(002)
url = save_attachment(mlist, part, dir,
del part['content-type']
An HTML attachment was scrubbed...
URL: %(url)s
"""), lcset)
elif ctype == 'message/rfc822':
# This part contains a submessage, so it too needs
submsg = part.get_payload(0)
omask = os.umask(002)
url = save_attachment(mlist, part, dir)
subject = submsg.get('subject', _('no subject'))
date = submsg.get('date', _('no date'))
who = submsg.get('from', _('unknown sender'))
size = len(str(submsg))
del part['content-type']
An embedded message was scrubbed...
From: %(who)s
Subject: %(subject)s
Date: %(date)s
Size: %(size)s
Url: %(url)s
"""), lcset)
# If the message isn't a multipart, then we'll strip it out as
# attachment that would have to be separately downloaded.
# will transform the url into a hyperlink.
elif not part.is_multipart():
payload = part.get_payload(decode=1)
ctype = part.get_type()
size = len(payload)
omask = os.umask(002)
url = save_attachment(mlist, part, dir)
desc = part.get('content-description', _('not available'))
filename = part.get_filename(_('not available'))
del part['content-type']
del part['content-transfer-encoding']
A non-text attachment was scrubbed...
Name: %(filename)s
Type: %(ctype)s
Size: %(size)d bytes
Desc: %(desc)s
Url : %(url)s
"""), lcset)
outer = 0
# We still have to sanitize multipart messages to flat text
# Pipermail can't handle messages with list payloads. This is a
# def (n) clever hack ;).
if msg.is_multipart():
# By default we take the charset of the first text/plain part
in the
# message, but if there was none, we'll use the list's
# language's charset.
if charset is None or charset == 'us-ascii':
charset = lcset
# We now want to concatenate all the parts which have been
scrubbed to
# text/plain, into a single text/plain payload. We need to
make sure
# all the characters in the concatenated string are in the
# encoding, so we'll use the 'replace' key in the coercion
# BAW: Martin's original patch suggested we might want to try
# generalizing to utf-8, and that's probably a good idea
text = []
for part in msg.get_payload():
# All parts should be scrubbed to text/plain by now.
partctype = part.get_content_type()
if partctype <> 'text/plain':
text.append(_('Skipped content of type
t = part.get_payload(decode=1)
except binascii.Error:
t = part.get_payload()
partcharset = part.get_content_charset()
if partcharset and partcharset <> charset:
t = unicode(t, partcharset, 'replace')
except (UnicodeError, LookupError):
# Replace funny characters
t = unicode(t, 'ascii', 'replace').encode('ascii')
# Should use HTML-Escape, or try generalizing to
t = t.encode(charset, 'replace')
except (UnicodeError, LookupError):
t = t.encode(lcset, 'replace')
# Separation is useful
if not t.endswith('\n'):
t += '\n'
# Now join the text and set the payload
sep = _('-------------- next part --------------\n')
del msg['content-type']
msg.set_payload(sep.join(text), charset)
del msg['content-transfer-encoding']
msg.add_header('Content-Transfer-Encoding', '8bit')
return msg

def makedirs(dir):
# Create all the directories to store this attachment in
os.makedirs(dir, 02775)
# Unfortunately, FreeBSD seems to be broken in that it doesn't
# the mode arg of mkdir().
def twiddle(arg, dirname, names):
os.chmod(dirname, 02775)
os.path.walk(dir, twiddle, None)
except OSError, e:
if e.errno <> errno.EEXIST: raise

def save_attachment(mlist, msg, dir, filter_html=1):
fsdir = os.path.join(mlist.archive_dir(), dir)
# Figure out the attachment type and get the decoded data
decodedpayload = msg.get_payload(decode=1)
# BAW: mimetypes ought to handle non-standard, but commonly found
# e.g. image/jpg (should be image/jpeg). For now we just store
# things as application/octet-streams since that seems the safest.
ctype = msg.get_content_type()
fnext = os.path.splitext(msg.get_filename(''))[1]
ext = guess_extension(ctype, fnext)
if not ext:
# We don't know what it is, so assume it's just a shapeless
# application/octet-stream, unless the Content-Type: is
# message/rfc822, in which case we know we'll coerce the type
# text/plain below.
if ctype == 'message/rfc822':
ext = '.txt'
ext = '.bin'
path = None
# We need a lock to calculate the next attachment number
lockfile = os.path.join(fsdir, 'attachments.lock')
lock = LockFile.LockFile(lockfile)
# Now base the filename on what's in the attachment,
uniquifying it if
# necessary.
filename = msg.get_filename()
if not filename:
filebase = 'attachment'
# Sanitize the filename given in the message headers
parts = pre.split(filename)
filename = parts[-1]
# Strip off leading dots
filename = dre.sub('', filename)
# Allow only alphanumerics, dash, underscore, and dot
filename = sre.sub('', filename)
# If the filename's extension doesn't match the type we
# which one should we go with? For now, let's go with the
one we
# guessed so attachments can't lie about their type.
Also, if the
# filename /has/ no extension, then tack on the one we
filebase, ignore = os.path.splitext(filename)
# Now we're looking for a unique name for this file on the
# system. If msgdir/filebase.ext isn't unique, we'll add a
# after filebase, e.g. msgdir/filebase-cnt.ext
counter = 0
extra = ''
while 1:
path = os.path.join(fsdir, filebase + extra + ext)
# Generally it is not a good idea to test for file
# before just trying to create it, but the alternatives
# wonderful (i.e. os.open(..., O_CREAT | O_EXCL) isn't
# NFS-safe). Besides, we have an exclusive lock now, so
# guaranteed that no other process will be racing with us.
if os.path.exists(path):
counter += 1
extra = '-%04d' % counter
# `path' now contains the unique filename for the attachment.
# just one more step we need to do. If the part is text/html and
# ARCHIVE_HTML_SANITIZER is a string (which it must be or we
wouldn't be
# here), then send the attachment through the filter program for
# sanitization
if filter_html and ctype == 'text/html':
base, ext = os.path.splitext(path)
tmppath = base + '-tmp' + ext
fp = open(tmppath, 'w')
cmd = mm_cfg.ARCHIVE_HTML_SANITIZER % {'filename' :
progfp = os.popen(cmd, 'r')
decodedpayload = progfp.read()
status = progfp.close()
if status:
'HTML sanitizer exited with non-zero status:
# BAW: Since we've now sanitized the document, it should be
# text. Blarg, we really want the sanitizer to tell us what
the type
# if the return data is. :(
ext = '.txt'
path = base + '.txt'
# Is it a message/rfc822 attachment?
elif ctype == 'message/rfc822':
submsg = msg.get_payload()
# BAW: I'm sure we can eventually do better than this. :(
decodedpayload = Utils.websafe(str(submsg))
fp = open(path, 'w')
# Now calculate the url
baseurl = mlist.GetBaseArchiveURL()
# Private archives will likely have a trailing slash. Normalize.
if baseurl[-1] <> '/':
baseurl += '/'
url = baseurl + '%s/%s%s%s' % (dir, filebase, extra, ext)
return url
Jul 18 '05 #1
0 2355

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

by: python | last post by:
hello and thanks for reading this, i have been a dos/windows user using some form of the basic language for 30 years now. i own and run a small programming company and there is one feature that...
by: Sandra-24 | last post by:
Is there a way in python to add the items of a dictionary to the local function scope? i.e. var_foo = dict. I don't know how many items are in this dictionary, or what they are until runtime. ...
by: tracyyun | last post by:
Hello everyone, I have a question and would like some advice on network connectivity. I have one computer connected to my router via WiFi, but I have two other computers that I want to be able to...
by: giovanniandrean | last post by:
The energy model is structured as follows and uses excel sheets to give input data: 1-Utility.py contains all the functions needed to calculate the variables and other minor things (mentions...
by: NeoPa | last post by:
Hello everyone. I find myself stuck trying to find the VBA way to get Access to create a PDF of the currently-selected (and open) object (Form or Report). I know it can be done by selecting :...
by: Teri B | last post by:
Hi, I have created a sub-form Roles. In my course form the user selects the roles assigned to the course. 0ne-to-many. One course many roles. Then I created a report based on the Course form and...
by: isladogs | last post by:
The next Access Europe meeting will be on Wednesday 1 Nov 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM) Please note that the UK and Europe revert to winter time on...
by: nia12 | last post by:
Hi there, I am very new to Access so apologies if any of this is obvious/not clear. I am creating a data collection tool for health care employees to complete. It consists of a number of...
by: NeoPa | last post by:
Introduction For this article I'll be focusing on the Report (clsReport) class. This simply handles making the calling Form invisible until all of the Reports opened by it have been closed, when it...
by: isladogs | last post by:
The next online meeting of the Access Europe User Group will be on Wednesday 6 Dec 2023 starting at 18:00 UK time (6PM UTC) and finishing at about 19:15 (7.15PM). In this month's session, Mike...
by: GKJR | last post by:
Does anyone have a recommendation to build a standalone application to replace an Access database? I have my bookkeeping software I developed in Access that I would like to make available to other...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.