Agent Regular Filters

Note: changes to reflect the release of Agent 1.9 have not yet been
made.

STOP! You may not need this much information!

This page covers [somewhat randomly and in excruciating detail]
'advanced' issues [primarily Agent {regular expression} filters]; for
this to make sense, you must be familiar with basic information in the
help file and the extensive list of (word) filter examples and related
advice/tips given in
Agent 'classic' (word) filters mini-FAQ

Generally, (word) filters are quicker to create, faster to run and
easier
to understand. Unless you absolutely have to filter on things that
(word)
filters can't handle [punctuation or sub-strings, for example] you
should
use (word) syntax if at all possible.

Credit where credit is due department: some expressions discussed
here
are copied from [or based on] suggestions by
bhkraft - gj.net
H.Reusch - ieee.org
homebody - mindspring.com
jimrtex - pipeline.com
Keith - Heinsohn.org
p_hantom - NoMail.Thks
phorty2 - super.zippo.com
m.pack - cableinet.co.uk
SirTwist - pobox.com
christof - paefken.westfalen.de
Dr.Peter.Giese - t-online.de

To point out an error, suggest an editorial correction, or submit a
new
filter [or expression] for inclusion, post to
news:alt.usenet.offline-reader.forte-agent
or use the guest book on my home page.

Changes in V1.809:
V1.809.1 Add discussion of the flood/HipCrime situation
Changes in V1.807:
V1.807.1 Add example to hit yet another morph of 'subject ends in
digits'
Changes in V1.806:
V1.806.1 Add example to hit new morph of 'subject ends in
specified characters'
Changes in V1.803:
V1.803.1 Add example to hit new morph of 'subject ends in digits'
Changes in V1.801:
V1.801.1 Add example to hit morphs of CABLE TV DESCRAMBLER
Changes in V1.75:
V1.75.01 Add note about anomaly with the = operator in global
searches
V1.75.02 Add note about anomaly with the = operator in global
searches
Changes in V1.73:
V1.73.01 Add another correction to the help file
V1.73.02 Recognize revision of the help file
Changes in V1.72:
V1.72.01 Add another mention of Nfilter
Changes in V1.60:
V1.60.06 Add example of global search for subject containing
'special' character
V1.60.05 Add another mention of Hamster
V1.60.04 Add usenet filter example for From: starting with a digit
V1.60.03 Add example of filtering subject on complex patterns
V1.60.02 Add examples of global search on Message-Id
V1.60.01 Add usenet filter example for Subject: ending with digits
Changes in V1.16:
V1.16.02 Add example of global search for any attachment
V1.16.01 Add caution on use of global search for Xref:
Changes in V1.11:
V1.11.05 Add examples of filtering subject on complex patterns
V1.11.04 Add examples of filtering subject on complex patterns
V1.11.03 Add examples of filtering subject on complex patterns
V1.11.02 Add example of filtering additional email headers
V1.11.01 Add example of filtering on Message-Id
Changes in V1.09:
V1.09.02 Add example of global search with complex criteria
V1.09.01 Add example of filtering subject containing punctuation
Changes in V1.08:
V1.08.03 Add example of filtering subject by complex criteria
V1.08.02 Add example of filtering subject containing punctuation
V1.08.01 Clarify explanation of one of the errors in the help file
Changes in V1.07:
V1.07.01 Suggest alternative to global search for killing
crossposts
Changes in V1.03:
V1.03.05 Add global search example for string containing <TAB>
V1.03.04 Add usenet filter example for Subject: ending with digits
V1.03.03 Add email filter example for Subject: ending with digits
V1.03.02 Add usenet filter example for Subject: beginning with
CAPS
V1.03.01 Add email filter example for blank/null Subject: and/or
To:
Changes in V1.02:
V1.02.08 Add clarification for null/blank cases
V1.02.07 Restate anti-U?E filter suggestion for convenience
V1.02.06 Add clarifying example to ALL CAPS discussion
V1.02.05 Suggest limited way to deal with pests using vcards
V1.02.04 Clarify global-search-as-filter strategy
V1.02.03 Suggest additional anti-U?E filter
V1.02.02 Add special case for 'newsgroups:' search
V1.02.01 Suggest limited way to deal with watch/kill sub-thread
Changes in V1.01:
V1.01.01 Describe small oddity with {} null expression

Table of contents:
I. What everyone must know before reading on.
II. Agent {regular expression} syntax is not all that regular.
III. Things to ignore in the help file.
IV. Detailed filter examples.
A. email filters
B. usenet From: filters
C. usenet Subject: filters
D. global search expressions

I. What everyone must know before reading on.

This mini-FAQ is intended to be presented in a Fixed Pitch font
[because
{regular expression} filters may be almost unintelligible viewed in a
proportional font]. Failing that, you must at least use a font that
distinguishes clearly between (parentheses) and {braces}.
Incidentally,
if you do a lot of work with {regular expression} filters in Agent you
you may find it useful to put FP [Fixed Pitch] buttons on your
toolbars.

Issues covered in the other mini-FAQ [filter priority, false
positives,
and many others] are not rehashed below. Watch out! Quite unexpected
false positives are more likely with {regular expression} filters than
with (word) filters, but detailed warnings may not be given for every
case. IMO the 'safe' way to proceed is to avoid using the "Delete"
action until it is _certain_ [to the extent that it may be possible]
that there are no undesirable side-effects.

Note: new filter expressions can be tested for 'reasonableness' using
the global search feature; that may not hit the same messages that the
final filter would [because of priority or scope differences], but it
may save you from installing an expression that is hopelessly flawed.

Also: provided that it is not necessary to match 'punctuation' [which
(word) filters cannot do], some filters given here could be translated
to the 'classic' form and still get [roughly speaking] the same result
[possibly in less time]. Doing that requires considerable familiarity
with the two 'languages', however. Just one example: if you are not
aware of the vast difference between
Bcc: {.*}
and
Bcc: * <- BAD
you should be _very_ careful when modifying these examples to avoid
accidentally using one form where the other is required.

II. Agent {regular expression} syntax is not all that regular.

If you are generally familiar with regular expressions, you need to
know that there are some significant differences between 'standard'
implementations and the Agent version.

A. In Agent, regular expressions must be enclosed in {braces}. They
can stand by themselves
subject: {string}
or be mixed with [or embedded in] Agent 'classic' (word) expressions
subject: word or from: {string}
subject: (word or {string})

B. Unless the % operator [an Agent extension] is in effect, Agent
{regular expression} filters are not case-sensitive and Agent (word)
filters follow rules given in the help file. In these two examples
subject: (%{STRING} or %word)
subject: (%{STRING} or word)
the first will match
STRING,aSTRINGent,STRING$(),...
word,MS-word,word-Perfect,...
but not
string,String,...
WORD,Word,MS-WORD,Word-Perfect,...
while the second will match
STRING,aSTRINGent,STRING$(),...
word,Word,WORD,MS-Word,Word-Perfect,...
Without the % operators they will both match
STRING,String,string,astringent,unstring$(),...
word,Word,WORD,MS-Word,Word-Perfect,...
[of course].
Note: the construction
subject: %({STRING} or word) <- BAD
is not valid.
Also: the % operator applies to an entire term, so in the filter
subject: % One two three
the last two words are constrained to match lower-case only; because
of the 'capitalization' rule for (word) filters, the first word would
match only
One
whether the % operator is present or not.

C. Another Agent extension is the = operator. It does two things:
1. it forces a match to the entire field; IOW, the 'standard' regular
expression
{^start.*finish$}
can be rendered in Agent as
={start.*finish}
with [except as noted in the next four items] exactly the same effect.
2. in the Subject: header only, it ignores follow-up prefixes ["re: "
or your defined list, if different]. IOW, when the subject: contains
Re: any topic
these expressions
{r}
{^Re: any topic$}
={any topic}
evaluate true, and these
={.*r.*}
={Re: any topic}
{^any topic$}
evaluate false.
+ +++ Added in V1.75
+ 3. the = operator causes global searches with (word) patterns and
+ the qualifiers Body: or Raw-Body: to apply to the entire body;
+ that is, it would take this expression
+ » Body: =This is line one "and" this is line two.
+ to hit a message with a body consisting entirely of these two lines
+ This is line one
+ and this is line two.
+ 4. the = operator causes global searches with {regular expression}
patterns and
+ the qualifiers Body: or Raw-Body: to fail absolutely; that is, these
expressions
+ body: ={.*} <- BAD
+ raw-body: ={.*} <- BAD
+ [which should hit every message] never hit anything.
+ 5. the operator pair {^$} [which in other fields is roughly
equivalent to ={}]
+ does not fail [as in item 4] or imply the entire body [as in item
3]; instead
+ it implies [as in most {regular expression} implementations] a
line-by-line match.
+ Thus, a message containing any line consisting entirely of the three
characters
+ foo
+ can be found by specifying either
+ » Body: {^foo$}
+ or
+ » Raw-Body: {^foo$}
+ +++