Dear all,

I am pleased to announce the availability of NaviServer 4.99.24 [1,2]. This release contains various fixes and enhancements. In short, it contains improved security features (reject invalid UTF-8), fallback charsets, Cookie support avoiding potential problems with more restrictive browsers, better network filtering support, improved Oracle support for new Oracle versions, etc.

The summary of changes is listed below. This release was tested with Ubuntu 20.04, Rocky Linux 8.5, FreeBSD 13.0, macOS 11.6.6.

The following people have contributed to this release:

Antonio Pisano
David Osborne
Gustaf Neumann
Zoran Vasiljevic

Many thanks to all contributors!

The upgrade to 4.99.24 is recommended. The installation script install-ns is updated to point to the new version.

-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.24/
[2] https://bitbucket.org/naviserver/naviserver/

======================================NaviServer 4.99.24, released 2022-06-14
======================================
77 files changed, 3242 insertions(+), 1100 deletions(-)

New Features:

- Improved security

* Added protection against certain attacks in ns_dbquotevalue

Due to the corrected conversion to external UTF-8 in db-output,
new potential attack vectors appeared that were protected earlier
via the Tcl-internal 'modified UTF-8'. E.g., the binary null
character is stored as an overlong (two-byte) encoding of null
(0xc0 0x80), so that an actual (embedded) null byte (0x00) never
appears in the string. Due to the conversion, the internal
representation is translated back to the binary null
character. Embedded null byte characters can lead to
non-terminated string literals via ns_dbquotevalue.

In the updated version of NaviServer, ns_dbquotevalue raises an
exception when this occurs. Therefore, the function can be used
as well as an input checker (together with "try").

* Raise an exception when trying to use "ns_urldecode" to
produce invalid UTF-8

Background: several (external) functions expect valid UTF-8 to be
passed in and crash if this is not the case. One such example is
tDOM. These nasty byte sequences are used more intensively by
vulnerability scanners.

Therefore, ns_urldecode raises now an exception, when it tries to
convert to invalid UTF-8. It is still possible to use
ns_urldecode to convert to other charsets.

ns_urldecode -charset iso8859-1 -part path "/mot%C3or"

When urldecode() is called internally and would produce invalid
UTF-8, it truncates the string (and writes a warning to the
system log).

Note that the new fallback charset feature might avoid such exceptions (see below).

- Fallback charsets

In case, a conversion to UTF-8 fails due to invalid byte sequences,
one can now provide a fallback charset for a second attempt of
decoding this data. This feature is useful for websites that
have to deal with requests containing invalid (form) data, typically
from legacy applications.

The fallback charset can be provided as optional parameter
"-fallbackcharset" for the command "ns_getform", "ns_parsequery"
and "ns_urldecode":

ns_getform ?-fallbackcharset fallbackcharset? ?charset?
ns_parsequery ?-charset charset? ?-fallbackcharset fallbackcharset? ?--? querystring
ns_urldecode ?-charset charset? ?-fallbackcharset fallbackcharset? ?-part part? ?--? string

In case, the parameter is not specified, it can be also be provided
to the form-processing commands "ns_getform" and "ns_parsequery" via
configuration variables:

* per-server configuration parameter "formFallbackCharset"
(in the section "ns/server/$server"), or as

* global server configuration parameter "formFallbackCharset"
(in the section "ns/parameters").

The highest precedence has the optional parameter, followed by the
per-server configuration parameter and the global configuration
parameter.

- Provide a hint when cache-entry is too large for caching

Background: the size of the entry is typically determined after the
execution of a potentially expensive query. During the eval of the
command, the cache entry is locked and forces a
serialization. However, this means that in these cases the
situation is worse than without a cache, where some queries can be
executed in parallel. We faced the situation of an unexpected
slowdown of the server with many "create entry collision" entries in the log file, where
due to application matters, an entry was becoming too large. This
situation is not easy to debug, especially in stress
situations. The log message would have helped a log to identify the
cause.

- Added support for multibyte numeric entities

This change supports the conversion of numeric entities representing
multibyte characters into HTML in "ns_striphtml" and
"ns_unquotehtml". Technically, the numeric entities represent
Unicode code points, which are transformed into UTF-8
serialization. Every entity represents a single code point; The
values can be provided in decimal or hexadecimal notation. Before
this change, only single byte numeric entities were
supported. ASCII control characters (decimal 0-31) are ignored as
before.

- New and extended commands:

* ns_unquotehtml /text/

This command is the inverse operation of "ns_quotehtml". It
replaces the named and numeric entities in the provided string
with the native values. The command is similar to "ns_striphtml",
but "ns_striphtml" removes as well other HTML markup which might
not be desired in all cases.

This change fixes as well a bug with numeric entities (the old
code assumed, these are starting directly with a number after the
ampersand) and it adds support for numeric entities with
hexadecimal values (so far with the same value range as for
decimal numeric entities).

* ns_subnetmatch /subnet/ /ipaddr/

Determine, if a provided IP address (IPv4 or IPv6) is included in
a subnet specification, which is provided in CIDR notation. The
command makes internal NaviServer functionality available at the
Tcl level. The regression test was extended to cover the
functionality.

The command ns_subnetmatch validates the provided subnet
specification (IPv4 or IPv6 address followed by slash and number
of significant bits) and the provided IP address and tests whether
the IP address is in the implied range. The command returns a
boolean value as the result. When comparing an IPv4 and IPv6
address/CIDR specification or vice versa, the result is always
false.

The function can be used when e.g. restricting access to certain
functionality to some subnets. The function can be used as well
to check, whether an IP address is an IPv4 or IPv6 address.

Examples:
% ns_subnetmatch 137.208.0.0/16 137.208.116.31
1

% ns_subnetmatch 137.208.0.0/16 112.207.16.33
0

% ns_subnetmatch 2001:628:404:74::31/64 [ns_conn peeraddr]
...

# Is IP address a valid IPv6 address?
% ns_subnetmatch ::/0 $ip

# Is IP address a valid IPv4 address?
% ns_subnetmatch 0.0.0.0/0 $ip

* ns_connchan: Added new subcommand "ns_connchan connect"

"ns_connchan connect" is similar to "ns_connchan open", except
that it does not send an HTTP request (HTTP method, URL, and
header fields) but just opens the connection. It can be used for
some non-HTTP communication over TCP and TLS over the ns_connchan
infrastructure.

* ns_parseheader, Ns_ParseHeader(): return the field
number (index) of the parsed entry

Previously, there was no explicit feedback, what field of an
"ns_set" has been parsed by "ns_parseheader". Now, in success
cases, the function returns the index of the new/modified
entry. This function made it possible to generalize and simplify
the Tcl-level parsing of "multipart/form-data"
significantly. Additionally, a new optional argument "-prefix"
was added. When specified, it adds the specified prefix to the
key.

* ns_setcookie, ns_deletecookie

Mozilla and Chrome changed the default value for SameSite of
cookies from "none" to "lax" in February and Aug 2020. Cookies
that explicitly set SameSite=None must also set the "Secure"
attribute.

In order to mirror this change of policy in NaviServer and to
reduce necessary code changes, the default behavior for setting
or deleting a cookie is now samesite "lax" (when "-samesite" is not
explicitly specified). When trying to set a cookie with
"-samesite none" without the "-secure" flag, a warning is
generated, and the "-samesite lax" is assumed, since major
browsers announced that they will reject these cookies soon.