I'm having a real hard time trying to do anything to a string (?)
returned by mailbox.MaildirMessage.get().

I'm extracting the Subject: header from a message and, if I write what
it returns to a log file using the python logging module what I see
in the log file (when the Subject: has non-ASCII characters in it) is:-

=?utf-8?Q?aka_Marne_=C3=A0_la_Sa=C3=B4ne_(Waterways_Continental_Europe)?=

Whatever I try I am unable to change the underscore characters in the
above string back to spaces.

So, what do those =?utf-8? and ?= sequences mean? Are they part of
the string or are they wrapped around the string on output as a way to
show that it's utf-8 encoded?

If I have the string in a variable how do I replace the underscores
with spaces? Simply doing "subject.replace('_', ' ')" doesn't work,
nothing happens at all.

All I really want to do is throw the non-ASCII characters away as the
string I'm trying to match in the subject is guaranteed to be ASCII.

--
Chris Green
·

Chris Green <cl@isbd.net> wrote:
> I'm having a real hard time trying to do anything to a string (?)
> returned by mailbox.MaildirMessage.get().
>
What a twit I am :-)

Strings are immutable, I have to do:-

newstring = oldstring.replace("_", " ")

Job done!

--
Chris Green
·

On Sat, 6 May 2023 14:50:40 +0100, Chris Green <cl@isbd.net> wrote:
[snip]
> So, what do those =?utf-8? and ?= sequences mean? Are they part of
> the string or are they wrapped around the string on output as a way to
> show that it's utf-8 encoded?

Yes, "=?utf-8?" signals "MIME header encoding".

I've only blundered about briefly in this area, but I think you
need to make sure that all header values you work with have been
converted to UTF-8 before proceeding.
Here's the code that seemed to work for me:

def mime_decode_single(pair):
"""Decode a single (bytestring, charset) pair.
"""
b, charset = pair
result = b if isinstance(b, str) else b.decode(
charset if charset else "utf-8")
return result

def mime_decode(s):
"""Decode a MIME-header-encoded character string.
"""
decoded_pairs = email.header.decode_header(s)
return "".join(mime_decode_single(d) for d in decoded_pairs)

--
To email me, substitute nowhere->runbox, invalid->com.

Peter Pearson ha scritto:
> On Sat, 6 May 2023 14:50:40 +0100, Chris Green <cl@isbd.net> wrote:
> [snip]
>> So, what do those =?utf-8? and ?= sequences mean? Are they part of
>> the string or are they wrapped around the string on output as a way to
>> show that it's utf-8 encoded?
>
> Yes, "=?utf-8?" signals "MIME header encoding".
>
> I've only blundered about briefly in this area, but I think you
> need to make sure that all header values you work with have been
> converted to UTF-8 before proceeding.
> Here's the code that seemed to work for me:
>
> def mime_decode_single(pair):
> """Decode a single (bytestring, charset) pair.
> """
> b, charset = pair
> result = b if isinstance(b, str) else b.decode(
> charset if charset else "utf-8")
> return result
>
> def mime_decode(s):
> """Decode a MIME-header-encoded character string.
> """
> decoded_pairs = email.header.decode_header(s)
> return "".join(mime_decode_single(d) for d in decoded_pairs)
>
>
>

HI,
You could also use make_header:

from email.header import decode_header, make_header

print(make_header(decode_header( subject )))

Chris Green <cl@isbd.net> writes:
> Chris Green <cl@isbd.net> wrote:
>> I'm having a real hard time trying to do anything to a string (?)
>> returned by mailbox.MaildirMessage.get().
>>
> What a twit I am :-)
>
> Strings are immutable, I have to do:-
>
> newstring = oldstring.replace("_", " ")
>
> Job done!

Not necessarily.

The subject in the original article was:
=?utf-8?Q?aka_Marne_=C3=A0_la_Sa=C3=B4ne_(Waterways_Continental_Europe)?=

That's some kind of MIME encoding. Just replacing underscores by spaces
won't necessarily give you anything meaningful. (What if there are
actual underscores in the original subject line?)

You should probably apply some kind of MIME-specific decoding. (I don't
have a specific suggestion for how to do that.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */

Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
> Chris Green <cl@isbd.net> writes:
> > Chris Green <cl@isbd.net> wrote:
> >> I'm having a real hard time trying to do anything to a string (?)
> >> returned by mailbox.MaildirMessage.get().
> >>
> > What a twit I am :-)
> >
> > Strings are immutable, I have to do:-
> >
> > newstring = oldstring.replace("_", " ")
> >
> > Job done!
>
> Not necessarily.
>
> The subject in the original article was:
> =?utf-8?Q?aka_Marne_=C3=A0_la_Sa=C3=B4ne_(Waterways_Continental_Europe)?=
>
> That's some kind of MIME encoding. Just replacing underscores by spaces
> won't necessarily give you anything meaningful. (What if there are
> actual underscores in the original subject line?)
>
> You should probably apply some kind of MIME-specific decoding. (I don't
> have a specific suggestion for how to do that.)
>
Yes, OK, but my problem was that my filter looks for the string
"Waterways Continental Europe" in the message Subject: to route the
message to the appropriate mailbox. When the Subject: has accents the
string becomes "Waterways_Continental_Europe" and thus the match
fails. Simply changing all underscores back to spaces makes my test
for "Waterways Continental Europe" work. The changed Subject: line
gets thrown away after the test so I don't care about anything else
getting changed.

(When there are no accented characters in the Subject: the string is
"Waterways Continental Europe" so I can't easily change the search
text. I guess I could use an RE.)

--
Chris Green
·

Chris Green ha scritto:
> Keith Thompson <Keith.S.Thompson+u@gmail.com> wrote:
>> Chris Green <cl@isbd.net> writes:
>>> Chris Green <cl@isbd.net> wrote:
>>>> I'm having a real hard time trying to do anything to a string (?)
>>>> returned by mailbox.MaildirMessage.get().
>>>>
>>> What a twit I am :-)
>>>
>>> Strings are immutable, I have to do:-
>>>
>>> newstring = oldstring.replace("_", " ")
>>>
>>> Job done!
>>
>> Not necessarily.
>>
>> The subject in the original article was:
>> =?utf-8?Q?aka_Marne_=C3=A0_la_Sa=C3=B4ne_(Waterways_Continental_Europe)?=
>>
>> That's some kind of MIME encoding. Just replacing underscores by spaces
>> won't necessarily give you anything meaningful. (What if there are
>> actual underscores in the original subject line?)
>>
>> You should probably apply some kind of MIME-specific decoding. (I don't
>> have a specific suggestion for how to do that.)
>>
> Yes, OK, but my problem was that my filter looks for the string
> "Waterways Continental Europe" in the message Subject: to route the
> message to the appropriate mailbox. When the Subject: has accents the
> string becomes "Waterways_Continental_Europe" and thus the match
> fails. Simply changing all underscores back to spaces makes my test
> for "Waterways Continental Europe" work. The changed Subject: line
> gets thrown away after the test so I don't care about anything else
> getting changed.
>
> (When there are no accented characters in the Subject: the string is
> "Waterways Continental Europe" so I can't easily change the search
> text. I guess I could use an RE.)
>

In reality you should also take into account the fact that if the header
contains a 'b' instead of a 'q' as a penultimate character, then the
rest of the package is converted on the basis64

"=?utf-8?Q?" --> "=?utf-8?B?"

Chris Green wrote at 2023-5-6 15:58 +0100:
>Chris Green <cl@isbd.net> wrote:
>> I'm having a real hard time trying to do anything to a string (?)
>> returned by mailbox.MaildirMessage.get().
>>
>What a twit I am :-)
>
>Strings are immutable, I have to do:-
>
> newstring = oldstring.replace("_", " ")

The solution based on `email.Header` proposed by `jak` is better.

On 08May2023 12:19, jak <nospam@please.ty> wrote:
>In reality you should also take into account the fact that if the
>header
>contains a 'b' instead of a 'q' as a penultimate character, then the
>rest of the package is converted on the basis64
>
>"=?utf-8?Q?" --> "=?utf-8?B?"

Aye. Specification:

https://datatracker.ietf.org/doc/html/rfc2047

You should reach for jak's suggested email.header suggestion _before_
parsing the subject line. Details:

https://docs.python.org/3/library/email.header.html#module-email.header

Cheers,
Cameron Simpson <cs@cskk.id.au>