On Tue, 23 Apr 2024 21:50:43 -0400, Hunter Goatley wrote:

> Not the source I would refer to.

You are certainly no “source” I would treat as credible.

On 24/4/24 12:34, Lawrence D'Oliveiro wrote:
> On Tue, 23 Apr 2024 21:50:43 -0400, Hunter Goatley wrote:
>
>> Not the source I would refer to.
>
> You are certainly no “source” I would treat as credible.

How I've missed you, usenet.

--
motk

In article <v04epp$jd4t$1@dont-email.me>, ldo@nz.invalid says...
>
> On Sun, 21 Apr 2024 19:14:59 -0400, Arne Vajhøj wrote:
>
> > On 4/21/2024 7:07 PM, Lawrence D'Oliveiro wrote:
> >>
> >> On Sun, 21 Apr 2024 11:20:03 +0200, Andreas Eder wrote:
> >>>
> >>> I think the problem is that they grew up in a Windows dominated world,
> >>> not like us greybeards.
> >>
> >> If only Windows had Linux-style service management, don?t you think?
> >> Imagine being able to add/remove, enable/disable and start/stop
> >> individual services without having to reboot the entire system!
> >
> > They don't need to imagine. They have been doing that for decades.

Indeed. If anything, Linux has acquired Windows NT style service
management and logging with systemd. There is no general requirement to
reboot for managing services. But some specific services can't always be
safely restarted on a running system.

Things like device drivers, filesystem drivers, the win32 environment
subsystem (of which csrss.exe is the usermode part), core system
libraries.

Linux is no different here. Want to upgrade your X server? Going to have
to at least restart all X11 apps. Want your update to Gtk or Qt to
actually take effect? All software using those is going to have to be
restarted. Rather than trying to figure out which software is or isn't
affected by some library update its easer to just reboot the whole
machine.

> Windows can?t even update a DLL that is in use by running processes. I
> suppose it inherited that file-locking mentality from VMS.
>
> Look at the long list of reasons why Windows needs a reboot here, from
> Microsoft itself:
> <https://learn.microsoft.com/en-us/troubleshoot/windows-server/installing-updates-features-roles/why-prompted-restart-computer>.

Those reasons all look pretty reasonable to me. You'd usually have to
reboot on Linux for most of those reasons too if you want the update to
actually take effect.

Remember that Windows NT doesn't have a monolithic kernel like Linux.
Just because its a .exe or .dll doesn't mean its some trivial user-space
thing that the system can continue running without.

For example, the csrss.exe mentioned in the article is the user-space
chunk of the Win32 environment subsystem - the thing that allows Windows
NT to run regular windows software.

And some services are actually hardware device drivers, or filesystem
drivers. Some drivers support being restarted online (graphics drivers
usually), but others are not so safely restarted on a running system.

On Wed, 24 Apr 2024 17:04:36 +1200, David Goodwin wrote:

> If anything, Linux has acquired Windows NT style service
> management and logging with systemd.

Did you know that systemd uses text-based config files in .INI format? The
same format Microsoft invented for Windows back in the 1980s, then junked
in favour of that stinking cesspool known as the Registry?

> Linux is no different here. Want to upgrade your X server? Going to have
> to at least restart all X11 apps.

That’s just logging out of a GUI session and logging in again.

And remember, you can have multiple GUI sessions logged in at once.

> For example, the csrss.exe mentioned in the article is the user-space
> chunk of the Win32 environment subsystem - the thing that allows Windows
> NT to run regular windows software.

There’s nothing like that needed on Linux.

In article <v0a5vu$25fmt$1@dont-email.me>, ldo@nz.invalid says...
>
> On Wed, 24 Apr 2024 17:04:36 +1200, David Goodwin wrote:
>
> > If anything, Linux has acquired Windows NT style service
> > management and logging with systemd.
>
> Did you know that systemd uses text-based config files in .INI format? The
> same format Microsoft invented for Windows back in the 1980s, then junked
> in favour of that stinking cesspool known as the Registry?

I do, I've written them before. But thats just the configuration storage
format - the actual service management capabilities at this point is
pretty similar. Here Linux is picking up features that Windows NT (and
commercial Unix) has had for decades.

> > Linux is no different here. Want to upgrade your X server? Going to have
> > to at least restart all X11 apps.
>
> That?s just logging out of a GUI session and logging in again.
>
> And remember, you can have multiple GUI sessions logged in at once.

Yes, but for most people (myself included), there isn't much difference
between logging off and on again and a full reboot. Either way I've
still got to open all my stuff again and having to re-open all my stuff
is the entire problem with rebooting.

Multiple sessions isn't a help here either unless you've got some
mechanism in place to move clients between X servers during the upgrade.
I guess you could do the same on Windows (it supports multiple sessions
too) but the benefits likely don't outweigh the costs of all the added
complexity.

> > For example, the csrss.exe mentioned in the article is the user-space
> > chunk of the Win32 environment subsystem - the thing that allows Windows
> > NT to run regular windows software.
>
> There?s nothing like that needed on Linux.

Linux still has a userspace API - the implementation of it is just all
statically linked into the kernel. If you want to update the parts that
make up its userspace API thats a kernel update and a reboot, just like
on Windows.

The difference here is that Windows NT isn't limited to a single
userspace API/personality, historically it provided three (Win32, OS/2
and POSIX) in addition to its own Native API. These lived entirely in
userspace rather than statically linked into the kernel with the first
user-space process (the Session Manager Subsystem) responsible for
starting them along with the Service Control Manager, the Local Security
Authority Subsystem Service, Windows Logon and other bits.

In article <v09vfb$2420c$1@dont-email.me>, motk <yep@yep.yep> wrote:
>On 24/4/24 12:34, Lawrence D'Oliveiro wrote:
>> On Tue, 23 Apr 2024 21:50:43 -0400, Hunter Goatley wrote:
>>
>>> Not the source I would refer to.
>>
>> You are certainly no “source” I would treat as credible.
>
>How I've missed you, usenet.

I'm always shocked and dismayed at how people many want to keep
arguing with the village idiot. Just plonk the guy and move on.
Life is too short to waste on the Lawrence's of the world.

- Dan C.

Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>The thing is, when you're working at scale, managing services
>across tens of thousands of machines, you quickly discover that
>shit happens. Things sometimes crash randomly; often this is
>due to a bug, but sometimes it's just because the OOM killer got
>greedy due to the delayed effects of a poor scheduling decision,
>or there was a dip on one of the voltage rails and a DIMM lost a
>bit, or a job landed on a machine that's got some latent
>hardware fault and it just happened to wiggle things in just the
>right way so that a 1 turned into a 0 (or vice versa), or any
>number of other things that may or may not have anything to do
>with the service itself.

Oh, I understand this completely. I have stood in the middle of a large
colocation facility and listened to Windows reboot sounds every second or
two coming from different places in the room each time.

What I don't necessarily understand is why people consider this acceptable.
People today just seem to think this is the normal way of doing business.
Surely we can do better.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

In article <v0bhum$laq$1@panix2.panix.com>,
Scott Dorsey <kludge@panix.com> wrote:
>Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>>The thing is, when you're working at scale, managing services
>>across tens of thousands of machines, you quickly discover that
>>shit happens. Things sometimes crash randomly; often this is
>>due to a bug, but sometimes it's just because the OOM killer got
>>greedy due to the delayed effects of a poor scheduling decision,
>>or there was a dip on one of the voltage rails and a DIMM lost a
>>bit, or a job landed on a machine that's got some latent
>>hardware fault and it just happened to wiggle things in just the
>>right way so that a 1 turned into a 0 (or vice versa), or any
>>number of other things that may or may not have anything to do
>>with the service itself.
>
>Oh, I understand this completely. I have stood in the middle of a large
>colocation facility and listened to Windows reboot sounds every second or
>two coming from different places in the room each time.
>
>What I don't necessarily understand is why people consider this acceptable.
>People today just seem to think this is the normal way of doing business.
>Surely we can do better.

Because it's the law of large numbers, not any particular fault
that can be reasonably addressed by an individual, organization,
etc.

If a marginal solder joint mechanically weakened by a bumpy ride
in a truck causes something to short, and that current draw on a
bus spikes over some threshold, pulling a voltage regulator out
of spec and causing voltage to sag by some nominal amount that
pulls another component on a different server below its marginal
threshold for a logic value and a bit flips, what are you, the
software engineer, supposed to do to tolerate that, let alone
recover from it? It's not a software bug, it's the confluence
of a large number of factors that only emerge when you run at a
scale with tens or hundreds of thousands of systems.

Google had a datacenter outage once, due to a diesel generator
catching on fire; there was a small leak and the ambient
temperature in the room with the generator exceeded the fuel's
flashpoint. "How does that happen?" you ask, since the auto
ignition temperatore of diesel fuel is actually pretty high. It
turns out that that is for liquids, and doesn't apply if the
fuel is aerosolized; the particular failure mode here was a
hairline fracture in the generator's fuel manifold. Diesel
forced through it was aerosolized and the ambient temperature
hit the fuel's flashpoint. Whoops. Cummins had never seen that
failure mode before; it was literally one in a million. The
solution, incidentally, was to wrap a flash-resistent cloth
around the manifold; basically putting a diaper on it, to keep
escaped fuel liquid.

Can we do better? Maybe. There were some lessons learned in
that failure; in part, making sure that the battery room doesn't
flood if the generator catches on fire (another part of the
story). But the reliability of hyperscalar operations is
already ridiculously high. They do it by using redundency and
designing in an _expectation_ of failure: multiple layers of
redundent load balancers, sharding traffic across multiple
backends, redundant storage in multiple geolocations, etc. But
a single computer failing and rebooting? That's expected. The
enterprise is, of course, much further behind, but I'd argue on
balance even they do all right, all things considered.

- Dan C.

On Wed, 24 Apr 2024 21:52:36 +1200, David Goodwin wrote:

> The difference here is that Windows NT isn't limited to a single
> userspace API/personality, historically it provided three (Win32, OS/2
> and POSIX) in addition to its own Native API.

That’s the theory. In practice, it doesn’t seem to have worked very well.
The POSIX “personality” for example, was essentially unusable.

When the Windows engineers were working on WSL1, emulating Linux kernel
APIs on Windows, you would think they would have used this “personality”
system. But they did not. I suspect it had already bitrotted into
nonfunctionality by that point.

In the end, they had to give up, and bring in an honest-to-goodness Linux
kernel, in WSL2.

On 4/24/2024 6:15 PM, Lawrence D'Oliveiro wrote:
> On Wed, 24 Apr 2024 21:52:36 +1200, David Goodwin wrote:
>> The difference here is that Windows NT isn't limited to a single
>> userspace API/personality, historically it provided three (Win32, OS/2
>> and POSIX) in addition to its own Native API.
>
> That’s the theory. In practice, it doesn’t seem to have worked very well.
> The POSIX “personality” for example, was essentially unusable.

My impression is that it worked fine.

But that customers were not interested.

Similar to VMS Posix kit.

> When the Windows engineers were working on WSL1, emulating Linux kernel
> APIs on Windows, you would think they would have used this “personality”
> system. But they did not.

Lowest common denonimator for Unix API's from the early 1990's was
probably not interesting.

> I suspect it had already bitrotted into
> nonfunctionality by that point.

Not keeping uptodate with evolution for 25 years.

> In the end, they had to give up, and bring in an honest-to-goodness Linux
> kernel, in WSL2.

Developers wanted the real thing.

Posix approach
--------------

If:

source--compiler--special executables--Posix API lib--Posix
subsystem--Windows

works then hope that:

source--standard Linux compiler--Linux executables--standard Linux
lib--standard Linux kernel

also works.

WSL1 approach
-------------

If:

source--Linux compiler--Linux executables--standard Linux lib--Linux
kernel emulation--Windows

works then hope that:

--Linux executables--standard Linux lib--standard Linux kernel

also works.

WSL1 approach
-------------

If:

source--Linux compiler--Linux executables--standard Linux lib--Linux
kernel--VM--Windows

works then know that:

--Linux executables--standard Linux lib--standard Linux kernel

also works.

Arne

On 4/22/2024 9:02 PM, Grant Taylor wrote:
> On 4/22/24 19:15, motk wrote:
>> I dunno, I'm not unintelligent but have you seen how much stress a
>> browser engine has to endure? Thousands of people with phds smash
>> these things to bits on the regular. Hundreds of thousands of people
>> use electron/react/whatever apps every day and never notice. Grousing
>> about this isn't a good look anymore.
>
> How much more productive work is done with a contemporary web browser in
> 2024 than in 2004 or even in 1998 (save for encryption)?

A lot I would say.

Most of email has moved to web.

A good chunk of office has moved to web (Google, MS in web mode).

Enabled by a combination of:
* faster PC's
* more standardized HTML & CSS - no more "requires IE x.0"
* faster JavaScript engines (JIT compiling)

> How much more productive work are computers doing in general in 2024
> than in 1994?

The world has become digitalized. A huge part of paper work
is now all electronic.

And also outside the administrative stuff. Try compare how a 2024
car works compared to a 1994 car.

> Have the frameworks and fancy things that are done in 2024 actually
> improved things?

They have enabled a lot of new stuff.

> I feel like there is massively disproportionately more computation power
> / resources consumed for very questionable things with not much to show
> for it.  Think what could have been done in the mid '90s with today's
> computing resources.

Software is changing due to hardware changes.

If you in 1994 could solve a problem in two ways:

A - super cool, takes 100 seconds
B - acceptable, takes 1 second

you picked B.

In 2024 it is:

A - super cool, takes 0.1 second
B - acceptable, takes 0.001 second

you pick A.

Arne

On 4/23/2024 9:03 AM, Dan Cross wrote:
> In article <v088m8$1juj9$1@dont-email.me>,
> Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>> On 2024-04-22, Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>>>
>>> Eh, JSON has its own problems; since the representation of
>>> numbers is specified to be compatible with floats, it's possible
>>> to lose data by translating it through JSON (I've seen people
>>> put e.g. machine addresses in JSON and then be surprised when
>>> the low bits disappear: floating point representations are not
>>> exact over the range of 64-bit integers!).
>>
>> I would consider that to be strictly a programmer error. That's the
>> kind of thing that should be stored as a string value unless you are
>> using a JSON library that preserves integer values unless decimal data
>> is present in the input data (and hence silently converts it to a float).
>>
>> I don't expect people to write their own JSON library (although I hope
>> they can given how relatively simple JSON is to parse), but I do expect
>> them to know what values they can use in libraries in general without
>> experiencing data loss.
>
> In modern languages, one can often derive JSON serialization and
> deserialization methods from the source data type, transparent
> to the programmer. Those may decide to use the JSON numeric
> type for numeric data; this has surprised a few people I know
> (who are extraordinarily competent programmers). Sure, the fix
> is generally easy (there's often a way to annotate a datum to
> say "serialize this as a string"), but that doesn't mean that
> even very senior people don't get caught out at times.
>
> But the problem is even more insideous than that; popular tools
> like `jq` can take properly serialized source data and silently
> make lossy conversions. So you might have properly written,
> value preserving libraries at both ends and still suffer loss
> due to some intermediate tool.
>
> JSON is dangerous. Caveat emptor.

This will be a relative long post. Sorry.

The problem at hand has nothing to do with JSON. It is
a string to numeric and data types problem.

JSON:

{ "v": 100000000000000001 }

XML:

<data>
<v>100000000000000001</v>
</data>

YAML:

v: 100000000000000001

All expose the same problem.

The value cannot be represented as is in some very common
data types like 32 bit integers and 64 bit floating point.

But selecting an appropriate data type for a given piece
of data based on its possible values and usage is
core responsibility for a developer.

The developer should understand possible values before
writing the code and understand the characteristics
of the available data types.

If not then that is a developer error.

But errors happen all the time. Not so good developers
make lots of errors. Good developers make occasionally
errors. The bad developers are not aware of data
requirements and how the data types work. The good developers
are aware, but because they are having a bad day or something
then it slips through anyway.

There is every indication that large JSON values and the
usage of 64 bit floating point has caused problems. There must
be a reason why this was added to the JSON RFC:

<quote>
This specification allows implementations to set limits on the range
and precision of numbers accepted. Since software that implements
IEEE 754 binary64 (double precision) numbers [IEEE754] is generally
available and widely used, good interoperability can be achieved by
implementations that expect no more precision or range than these
provide, in the sense that implementations will approximate JSON
numbers within the expected precision. A JSON number such as 1E400
or 3.141592653589793238462643383279 may indicate potential
interoperability problems, since it suggests that the software that
created it expects receiving software to have greater capabilities
for numeric magnitude and precision than is widely available.

Note that when such software is used, numbers that are integers and
are in the range [-(2**53)+1, (2**53)-1] are interoperable in the
sense that implementations will agree exactly on their numeric
values.
</quote>

Excusing the error with limitations in the programming language
and the JSON library is a poor excuse.

The developer (+ architect + other senior technology decision makers)
are responsible for selecting a programming language and a
JSON library that meet business requirements.

Mistakes can be made in all sorts of programming languages
no matter the typing system. But my guess is that it happens
more frequently when the type is not static and explicit declared.
Some doesn't like to type in that type, but it literally makes the
type visible.

The fact that it ultimately is the developers responsibility
to select proper data types does not mean that programming languages
and JSON libraries can not help catch errors.

If it is obvious that an unexpected/useless result is being
produced then it should be flagged (return error code or throw
exception depending on technology).

Let us go back to the example with 100000000000000001.

Trying to stuff that into a 32 bit integer by like parsing
it as a 64 bit integer and returning the lower 32 bits
is in my best opinion an error. Nobody wants to get an int
with 1569325057 from retrieving a 32 bit integer integer
from "100000000000000001". It should give an error.

The case with a 64 bit floating point is more tricky. One
could argue that 100000000000000001.0 is the expected
result and that 100000000000000000.0 should be considered
an error. And it probably would be an error in the majority
of cases. But there is actually the possibility that
someone that understand floating point are reading JSON
and expect what is happening and does not care because
there are some uncertainty in the underlying data. And
creating a false error for people that understand FP data
types to prevent those that do not understand FP data types
from shooting themself in the foot is not good.

Arne

On Wed, 24 Apr 2024 19:13:44 -0400, Arne Vajhøj wrote:

> On 4/24/2024 6:15 PM, Lawrence D'Oliveiro wrote:
>>
>> On Wed, 24 Apr 2024 21:52:36 +1200, David Goodwin wrote:
>>>
>>> The difference here is that Windows NT isn't limited to a single
>>> userspace API/personality, historically it provided three (Win32, OS/2
>>> and POSIX) in addition to its own Native API.
>>
>> That’s the theory. In practice, it doesn’t seem to have worked very
>> well. The POSIX “personality” for example, was essentially unusable.
>
> My impression is that it worked fine.

You got to be kidding <https://www.youtube.com/watch?v=BOeku3hDzrM>.

>> When the Windows engineers were working on WSL1, emulating Linux kernel
>> APIs on Windows, you would think they would have used this
>> “personality” system. But they did not.
>
> Lowest common denonimator for Unix API's from the early 1990's was
> probably not interesting.

They couldn’t get it to work.

On 4/24/2024 8:08 PM, Arne Vajhøj wrote:
> This will be a relative long post. Sorry.

And this one will be even longer!

> The problem at hand has nothing to do with JSON. It is
> a string to numeric and data types problem.
>
> JSON:
>
> { "v": 100000000000000001 }
>
> XML:
>
> <data>
>     <v>100000000000000001</v>
> </data>
>
> YAML:
>
> v: 100000000000000001
>
> All expose the same problem.
>
> The value cannot be represented as is in some very common
> data types like 32 bit integers and 64 bit floating point.

> The fact that it ultimately is the developers responsibility
> to select proper data types does not mean that programming languages
> and JSON libraries can not help catch errors.
>
> If it is obvious that an unexpected/useless result is being
> produced then it should be flagged (return error code or throw
> exception depending on technology).
>
> Let us go back to the example with 100000000000000001.
>
> Trying to stuff that into a 32 bit integer by like parsing
> it as a 64 bit integer and returning the lower 32 bits
> is in my best opinion an error. Nobody wants to get an int
> with 1569325057 from retrieving a 32 bit integer integer
> from "100000000000000001". It should give an error.
>
> The case with a 64 bit floating point is more tricky. One
> could argue that 100000000000000001.0 is the expected
> result and that 100000000000000000.0 should be considered
> an error. And it probably would be an error in the majority
> of cases. But there is actually the possibility that
> someone that understand floating point are reading JSON
> and expect what is happening and does not care because
> there are some uncertainty in the underlying data. And
> creating a false error for people that understand FP data
> types to prevent those that do not understand FP data types
> from shooting themself in the foot is not good.

Let us see some code.

I picked Groovy as demo language, because I am a J guy
and Groovy allows to demo a lot.

As a JVM language there are several possible data types to
pick from:
int
long
double
String
BigInteger
BigDecimal

And obviously int and double has problem with 100000000000000001
while the rest can store it OK.

To illustrate the option of different JSON libraries I will
test with both GSON (Google JSON library) and Jackson (probably
the most widely used JSON library in the Java wold).

Let us first look at the model where the JSON is parsed to
a tree.

GSON:

$ type Data1.groovy
import com.google.gson.*

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

String json = '{ "v": 100000000000000001 }'
println(json)
gson = new JsonParser()
o = gson.parse(json).getAsJsonObject()
wrap {
v1 = o.get("v").getAsInt()
printf("int: %d\n", v1)
} wrap {
v2 = o.get("v").getAsLong()
printf("long: %d (low 32 bit is %d)\n", v2, v2 & 0x00000000FFFFFFFFL)
} wrap {
v3 = o.get("v").getAsDouble()
printf("double: %f\n", v3)
} wrap {
v4 = o.get("v").getAsString()
printf("String: %s\n", v4)
} wrap {
v5 = o.get("v").getAsBigInteger()
printf("BigInteger: %s\n", v5)
} wrap {
v6 = o.get("v").getAsBigDecimal()
printf("BigDecimal: %s\n", v6)
} wrap {
v7 = o.get("v").getAsNumber()
printf("Number: %s (%s)\n", v7, v7.class.name)
} $ groovy Data1.groovy
{ "v": 100000000000000001 }
int: 1569325057
long: 100000000000000001 (low 32 bit is 1569325057)
double: 100000000000000000.000000
String: 100000000000000001
BigInteger: 100000000000000001
BigDecimal: 100000000000000001
Number: 100000000000000001 (com.google.gson.internal.LazilyParsedNumber)

Jackson:

$ type Data1X.groovy
import com.fasterxml.jackson.databind.*

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

String json = '{ "v": 100000000000000001 }'
println(json)
om = new ObjectMapper()
o = om.readTree(json)
wrap {
v1 = o.get("v").asInt()
printf("int: %d\n", v1)
} wrap {
v2 = o.get("v").asLong()
printf("long: %d (low 32 bit is %d)\n", v2, v2 & 0x00000000FFFFFFFFL)
} wrap {
v3 = o.get("v").asDouble()
printf("double: %f\n", v3)
} wrap {
v4 = o.get("v").asText()
printf("String: %s\n", v4)
} wrap {
v5 = o.get("v").bigIntegerValue()
printf("BigInteger: %s\n", v5)
} wrap {
v6 = o.get("v").decimalValue()
printf("BigDecimal: %s\n", v6)
} wrap {
v7 = o.get("v").numberValue()
printf("Number: %s (%s)\n", v7, v7.class.name)
} $ groovy Data1X.groovy
{ "v": 100000000000000001 }
int: 1569325057
long: 100000000000000001 (low 32 bit is 1569325057)
double: 100000000000000000.000000
String: 100000000000000001
BigInteger: 100000000000000001
BigDecimal: 100000000000000001
Number: 100000000000000001 (java.lang.Long)

We see:
* the expected slightly off value for double
* the crazy value for int (no exception)
* other data types are fine

Now mapping/binding to class.

GSON:

$ type Data2.groovy
import com.google.gson.*

class C1 {
int v
}

class C2 {
long v
}

class C3 {
double v
}

class C4 {
String v
}

class C5 {
BigInteger v
}

class C6 {
BigDecimal v
}

class C7 {
Number v
}

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

json = '{ "v": 100000000000000001 }'
println(json)
gson = new Gson()
wrap {
C1 v1 = gson.fromJson(json, C1.class)
printf("int: %d\n", v1.v)
} wrap {
C2 v2 = gson.fromJson(json, C2.class)
printf("long: %d\n", v2.v)
} wrap {
C3 v3 = gson.fromJson(json, C3.class)
printf("double: %f\n", v3.v)
} wrap {
C4 v4 = gson.fromJson(json, C4.class)
printf("String: %s\n", v4.v)
} wrap {
C5 v5 = gson.fromJson(json, C5.class)
printf("BigInteger: %s\n", v5.v)
} wrap {
C6 v6 = gson.fromJson(json, C6.class)
printf("BigDecimal: %s\n", v6.v)
} wrap {
C7 v7 = gson.fromJson(json, C7.class)
printf("Number: %s (%s)\n", v7.v, v7.v.class.name)
} $ groovy Data2.groovy
{ "v": 100000000000000001 }
com.google.gson.JsonSyntaxException: java.lang.NumberFormatException:
Expected an int but was 100000000000000001 at line 1 column 26
long: 100000000000000001
double: 100000000000000000.000000
String: 100000000000000001
BigInteger: 100000000000000001
BigDecimal: 100000000000000001
Number: 100000000000000001 (com.google.gson.internal.LazilyParsedNumber)

Jackson:

$ type Data2X.groovy
import com.fasterxml.jackson.databind.*

class C1 {
int v
}

class C2 {
long v
}

class C3 {
double v
}

class C4 {
String v
}

class C5 {
BigInteger v
}

class C6 {
BigDecimal v
}

class C7 {
Number v
}

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

json = '{ "v": 100000000000000001 }'
println(json)
om = new ObjectMapper()
wrap {
C1 v1 = om.readValue(json, C1.class)
printf("int: %d\n", v1.v)
} wrap {
C2 v2 = om.readValue(json, C2.class)
printf("long: %d\n", v2.v)
} wrap {
C3 v3 = om.readValue(json, C3.class)
printf("double: %f\n", v3.v)
} wrap {
C4 v4 = om.readValue(json, C4.class)
printf("String: %s\n", v4.v)
} wrap {
C5 v5 = om.readValue(json, C5.class)
printf("BigInteger: %s\n", v5.v)
} wrap {
C6 v6 = om.readValue(json, C6.class)
printf("BigDecimal: %s\n", v6.v)
} wrap {
C7 v7 = om.readValue(json, C7.class)
printf("Number: %s (%s)\n", v7.v, v7.v.class.name)
} $ groovy Data2X.groovy
{ "v": 100000000000000001 }
com.fasterxml.jackson.databind.JsonMappingException: Numeric value
(100000000000000001) out of range of int (-2147483648 - 2147483647)
at [Source: (String)"{ "v": 100000000000000001 }"; line: 1, column:
26] (through reference chain: C1["v"])
long: 100000000000000001
double: 100000000000000000.000000
String: 100000000000000001
BigInteger: 100000000000000001
BigDecimal: 100000000000000001
Number: 100000000000000001 (java.lang.Long)

In article <v0c6t7$2jtvn$1@dont-email.me>,
Arne Vajhøj <arne@vajhoej.dk> wrote:
>On 4/23/2024 9:03 AM, Dan Cross wrote:
>> In article <v088m8$1juj9$1@dont-email.me>,
>> Simon Clubley <clubley@remove_me.eisner.decus.org-Earth.UFP> wrote:
>>> On 2024-04-22, Dan Cross <cross@spitfire.i.gajendra.net> wrote:
>>>>
>>>> Eh, JSON has its own problems; since the representation of
>>>> numbers is specified to be compatible with floats, it's possible
>>>> to lose data by translating it through JSON (I've seen people
>>>> put e.g. machine addresses in JSON and then be surprised when
>>>> the low bits disappear: floating point representations are not
>>>> exact over the range of 64-bit integers!).
>>>
>>> I would consider that to be strictly a programmer error. That's the
>>> kind of thing that should be stored as a string value unless you are
>>> using a JSON library that preserves integer values unless decimal data
>>> is present in the input data (and hence silently converts it to a float).
>>>
>>> I don't expect people to write their own JSON library (although I hope
>>> they can given how relatively simple JSON is to parse), but I do expect
>>> them to know what values they can use in libraries in general without
>>> experiencing data loss.
>>
>> In modern languages, one can often derive JSON serialization and
>> deserialization methods from the source data type, transparent
>> to the programmer. Those may decide to use the JSON numeric
>> type for numeric data; this has surprised a few people I know
>> (who are extraordinarily competent programmers). Sure, the fix
>> is generally easy (there's often a way to annotate a datum to
>> say "serialize this as a string"), but that doesn't mean that
>> even very senior people don't get caught out at times.
>>
>> But the problem is even more insideous than that; popular tools
>> like `jq` can take properly serialized source data and silently
>> make lossy conversions. So you might have properly written,
>> value preserving libraries at both ends and still suffer loss
>> due to some intermediate tool.
>>
>> JSON is dangerous. Caveat emptor.
>
>This will be a relative long post. Sorry.
>
>The problem at hand has nothing to do with JSON. It is
>a string to numeric and data types problem.

It is a problem with a format that permits silently lossy
conversions.

>[snip]
>But selecting an appropriate data type for a given piece
>of data based on its possible values and usage is
>core responsibility for a developer.

One hopes one also has tools that permit detection of
truncation. You also ignored the point about how third-party
tools for manipulating JSON will silently lose data.

- Dan C.

On 4/24/2024 8:27 PM, Arne Vajhøj wrote:
> Let us see some code.
>
> I picked Groovy as demo language, because I am a J guy
> and Groovy allows to demo a lot.
>
> As a JVM language there are several possible data types to
> pick from:
>   int
>   long
>   double
>   String
>   BigInteger
>   BigDecimal
>
> And obviously int and double has problem with 100000000000000001
> while the rest can store it OK.
>
> To illustrate the option of different JSON libraries I will
> test with both GSON (Google JSON library) and Jackson (probably
> the most widely used JSON library in the Java wold).
>
> Let us first look at the model where the JSON is parsed to
> a tree.

> We see:
> * the expected slightly off value for double
> * the crazy value for int (no exception)
> * other data types are fine
>
> Now mapping/binding to class.

> Very similar behavior except that now I do get the exception for
> int that I so much prefer.

All that is of course if one bother to pick the data type.

Otherwise the result become a bit more implementation
specific.

GSON:

$ type Data3.groovy
import com.google.gson.*

class C1 {
def v = 0
}

class C2 {
def v = 0.0d
}

class C3 {
def v = 0.0
}

class C4 {
def v
}

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

json = '{ "v": 100000000000000001 }'
println(json)
gson = new Gson()
wrap {
C1 v1 = gson.fromJson(json, C1.class)
printf("def = 0: %f (%s)\n", v1.v, v1.v.class.name)
} wrap {
C2 v2 = gson.fromJson(json, C2.class)
printf("def = 0.0d: %f (%s)\n", v2.v, v2.v.class.name)
} wrap {
C3 v3 = gson.fromJson(json, C3.class)
printf("def = 0.0: %f (%s)\n", v3.v, v3.v.class.name)
} wrap {
C4 v4 = gson.fromJson(json, C4.class)
printf("def: %f (%s)\n", v4.v, v4.v.class.name)
} $ groovy Data3.groovy
{ "v": 100000000000000001 }
def = 0: 100000000000000000.000000 (java.lang.Double)
def = 0.0d: 100000000000000000.000000 (java.lang.Double)
def = 0.0: 100000000000000000.000000 (java.lang.Double)
def: 100000000000000000.000000 (java.lang.Double)

Jackson:

$ type Data3X.groovy
import com.fasterxml.jackson.databind.*

class C1 {
def v = 0
}

class C2 {
def v = 0.0d
}

class C3 {
def v = 0.0
}

class C4 {
def v
}

def wrap(block) {
try {
block()
} catch(ex) {
printf("%s: %s\n", ex.class.name, ex.message)
}
}

json = '{ "v": 100000000000000001 }'
println(json)
om = new ObjectMapper()
wrap {
C1 v1 = om.readValue(json, C1.class)
printf("def = 0: %d (%s)\n", v1.v, v1.v.class.name)
} wrap {
C2 v2 = om.readValue(json, C2.class)
printf("def = 0.0d: %d (%s)\n", v2.v, v2.v.class.name)
} wrap {
C3 v3 = om.readValue(json, C3.class)
printf("def = 0.0: %d (%s)\n", v3.v, v3.v.class.name)
} wrap {
C4 v4 = om.readValue(json, C4.class)
printf("def: %d (%s)\n", v4.v, v4.v.class.name)
} $ groovy Data3X.groovy
{ "v": 100000000000000001 }
def = 0: 100000000000000001 (java.lang.Long)
def = 0.0d: 100000000000000001 (java.lang.Long)
def = 0.0: 100000000000000001 (java.lang.Long)
def: 100000000000000001 (java.lang.Long)

GSON picked a "bad" data type (Double) while
Jackson picked a "good" data type (Long).

Groovy is in no way a typical dynamic typed
language, but I still think it sort of show that
JSON parsing may be a case where one really should
prefer static typing.

Arne

On 4/24/2024 8:36 PM, Dan Cross wrote:
> In article <v0c6t7$2jtvn$1@dont-email.me>,
> Arne Vajhøj <arne@vajhoej.dk> wrote:
>> The problem at hand has nothing to do with JSON. It is
>> a string to numeric and data types problem.
>
> It is a problem with a format that permits silently lossy
> conversions.

Trying to stuff 100000000000000001 into a 32 bit integer
or a 64 bit floating point create a problem.

But it does not matter if the format is JSON or XML or YAML
or whatever.

The developer made a design error. And the language/library
did not object.

But none of that is a problem in the format. The JSON has the
correct value.

>> [snip]
>> But selecting an appropriate data type for a given piece
>> of data based on its possible values and usage is
>> core responsibility for a developer.
>
> One hopes one also has tools that permit detection of
> truncation. You also ignored the point about how third-party
> tools for manipulating JSON will silently lose data.

You must have missed this part:

# The fact that it ultimately is the developers responsibility
# to select proper data types does not mean that programming languages
# and JSON libraries can not help catch errors.
# # If it is obvious that an unexpected/useless result is being
# produced then it should be flagged (return error code or throw
# exception depending on technology).
# # Let us go back to the example with 100000000000000001.
# # Trying to stuff that into a 32 bit integer by like parsing
# it as a 64 bit integer and returning the lower 32 bits
# is in my best opinion an error. Nobody wants to get an int
# with 1569325057 from retrieving a 32 bit integer integer
# from "100000000000000001". It should give an error.
# # The case with a 64 bit floating point is more tricky. One
# could argue that 100000000000000001.0 is the expected
# result and that 100000000000000000.0 should be considered
# an error. And it probably would be an error in the majority
# of cases. But there is actually the possibility that
# someone that understand floating point are reading JSON
# and expect what is happening and does not care because
# there are some uncertainty in the underlying data. And
# creating a false error for people that understand FP data
# types to prevent those that do not understand FP data types
# from shooting themself in the foot is not good.

Arne

On 4/24/2024 8:18 PM, Lawrence D'Oliveiro wrote:
> On Wed, 24 Apr 2024 19:13:44 -0400, Arne Vajhøj wrote:
>> On 4/24/2024 6:15 PM, Lawrence D'Oliveiro wrote:
>>> On Wed, 24 Apr 2024 21:52:36 +1200, David Goodwin wrote:
>>>> The difference here is that Windows NT isn't limited to a single
>>>> userspace API/personality, historically it provided three (Win32, OS/2
>>>> and POSIX) in addition to its own Native API.
>>>
>>> That’s the theory. In practice, it doesn’t seem to have worked very
>>> well. The POSIX “personality” for example, was essentially unusable.
>>
>> My impression is that it worked fine.
>
> You got to be kidding <https://www.youtube.com/watch?v=BOeku3hDzrM>.

I did not watch all of it.

But it seems like this person did not really understand the
Posix standard (from that time).

His main argument seems to be that it is not useful.

I believe that.

But that is the limitation of Posix (from that time).

VMS Posix was not considered useful by many either.

Arne

=?UTF-8?Q?Arne_Vajh=C3=B8j?= <arne@vajhoej.dk> wrote:
>But it seems like this person did not really understand the
>Posix standard (from that time).
>
>His main argument seems to be that it is not useful.
>
>I believe that.
>
>But that is the limitation of Posix (from that time).

This is entirely true. The Posix libraries in theory would let you run
unix applications on non-Unix systems but in fact they were never useful
enough to run anything much.

>VMS Posix was not considered useful by many either.

No, it wasn't useful. It met certain requirements for certain government
purchases, which is why it was designed, but it was not actually useful
for anyone porting code.
--scott
--
"C'est un Nagra. C'est suisse, et tres, tres precis."

In article <v0c98k$2kbf5$1@dont-email.me>,
Arne Vajhøj <arne@vajhoej.dk> wrote:
>On 4/24/2024 8:36 PM, Dan Cross wrote:
>> In article <v0c6t7$2jtvn$1@dont-email.me>,
>> Arne Vajhøj <arne@vajhoej.dk> wrote:
>>> The problem at hand has nothing to do with JSON. It is
>>> a string to numeric and data types problem.
>>
>> It is a problem with a format that permits silently lossy
>> conversions.
>
>Trying to stuff 100000000000000001 into a 32 bit integer
>or a 64 bit floating point create a problem.

Obvious.

>The developer made a design error. And the language/library
>did not object.

That's not the problem, though.

>But none of that is a problem in the format. The JSON has the
>correct value.

No, it often doesn't. Because there are numeric constants that
are not representable at all in the most common encoding used
by JSON _tools_.

>>> [snip]
>>> But selecting an appropriate data type for a given piece
>>> of data based on its possible values and usage is
>>> core responsibility for a developer.
>>
>> One hopes one also has tools that permit detection of
>> truncation. You also ignored the point about how third-party
>> tools for manipulating JSON will silently lose data.
>
>You must have missed this part:
>
># The fact that it ultimately is the developers responsibility
># to select proper data types does not mean that programming languages
># and JSON libraries can not help catch errors.

That's obvious. What you don't seem to understand is that there
are very common tools outside of the programmer's control that
are used to manipulate JSON data that perform silently lossy
conversions. That is, those tools not only don't "help catch
errors", they actually introduce them.

- Dan C.

On Sun, 21 Apr 2024 19:18:19 -0400, Arne Vajhøj wrote:

> The 3 most widely formats for config files today are XML, JSON and YAML.

No, those are mainly used for data interchange, not so much for config
files (apart from XML).

> But INI and Java properties may be the next 2 in usage.

“Java properties” isn’t a file format.

There is also TOML.

On 4/25/2024 6:33 PM, Lawrence D'Oliveiro wrote:
> On Sun, 21 Apr 2024 19:18:19 -0400, Arne Vajhøj wrote:
>> The 3 most widely formats for config files today are XML, JSON and YAML.
>
> No, those are mainly used for data interchange, not so much for config
> files (apart from XML).

JSON is used for data exchange, but also for config.

Examples:
* .NET Core/5+ and its app config replacement
* PHP composer and its composer.json

YAML is the preferred config file format in the container/devops world.

>> But INI and Java properties may be the next 2 in usage.
>
> “Java properties” isn’t a file format.

Of course it is.

foobar.ini with:

[a]
a1 = 123
a2 = ABC
[b]
b1 = 456
b2 = DEF

can be done as foobar.properties:

a.a1 = 123
a.a2 = ABC
b.b1 = 456
b.b2 = DEF

> There is also TOML.

Ah. I forgot about that one.

Arne

On 2024-04-25, Lawrence D'Oliveiro <ldo@nz.invalid> wrote:
> On Sun, 21 Apr 2024 19:18:19 -0400, Arne Vajhøj wrote:
>
>> The 3 most widely formats for config files today are XML, JSON and YAML.
>
> No, those are mainly used for data interchange, not so much for config
> files (apart from XML).

JSON and YAML are incredibly common for config files for newer software.
I've not really seen YAML used as a data interchange format, in my
experience it exists almost entirely to be a more human-friendly
alternative to JSON for configuration.

>> But INI and Java properties may be the next 2 in usage.
>
> “Java properties” isn’t a file format.

https://en.wikipedia.org/wiki/.properties

https://docs.oracle.com/en/java/javase/19/docs/api/java.base/java/util/Properties.html#load(java.io.Reader)

The Java properties file format is well-defined and, not surprisingly,
used heavily by Java applications. I've been out of the Java world for a
while, but in the 2000s, properties files were definitely in use for
configuration files for many applications I ran and worked on.

-Matthew

In article <v0c0a6$2ihq5$2@dont-email.me>, ldo@nz.invalid says...
>
> On Wed, 24 Apr 2024 21:52:36 +1200, David Goodwin wrote:
>
> > The difference here is that Windows NT isn't limited to a single
> > userspace API/personality, historically it provided three (Win32, OS/2
> > and POSIX) in addition to its own Native API.
>
> That?s the theory. In practice, it doesn?t seem to have worked very well.
> The POSIX ?personality? for example, was essentially unusable.

There were two different POSIX personalities. There was the original
POSIX Environment Subsystem was included since NT 3.1 for box checking
purposes - it strictly implemented the POSIX specifications *and nothing
more* making it not terribly useful for anything beyond bidding for
government contracts that required "POSIX".

Then there was Interix which was a 3rd-party product that Microsoft
acquired and renamed to Services for Unix (SFU). This was initially sold
as a product and later given away for free. It was last made available
for Windows 8 and Server 2012.

Interix worked quite well, but it wasn't Linux. IIRC getting stuff going
on it was much the same as getting it going on some random proprietary
Unix. But Microsoft did include GCC and a bunch of other stuff, and
there was a community site providing more pre-built binaries.

> When the Windows engineers were working on WSL1, emulating Linux kernel
> APIs on Windows, you would think they would have used this ?personality?
> system. But they did not. I suspect it had already bitrotted into
> nonfunctionality by that point.

Microsoft didn't re-architect Windows NT for the Windows 10 release or
move all of Win32 into the kernel (why would they?). CSRSS is still
there just like its always been providing the Win32 environment that
most windows software runs under.

And its not like WSLv1 is some Cygwin-like thing sitting on top of
Win32. Its even in the name: Windows Subsystem for Linux. WSLv1 is like
Interix but taken several steps further to provide Linux binary
compatibility and better performance.

> In the end, they had to give up, and bring in an honest-to-goodness Linux
> kernel, in WSL2.

Primarily because it gets them them the best compatibility (including
kernel modules) with a wide selection of pre-built software for the
least amount of work. Why go to all the effort reimplementing the whole
Linux syscall interface and keeping it up-to-date when you could just
run the Linux kernel in a VM.

Anyway, I'm not sure what your point is here. Are you trying to argue
that moving code out of the kernel into userspace is a bad idea?

On Fri, 26 Apr 2024 16:36:41 +1200, David Goodwin wrote:

> Why go to all the effort reimplementing the whole
> Linux syscall interface and keeping it up-to-date when you could just
> run the Linux kernel in a VM.

Because that was what the whole “personality” system was supposed to be
for. In practice, it didn’t work.

> Are you trying to argue that moving code out of the kernel into
> userspace is a bad idea?

It would have been great if they could have implemented the Linux API that
way, wouldn’t it? But they couldn’t do it.