"Michael F. Stemper" <michael.stemper@gmail.com> writes:
>Well, I could continue to hard-code the data into one of the test
>programs

One can employ a gradual path from a program with hardcoded
data to an entity sharable by different programs.

When I am hurried to rush to a working program, I often
end up with code that contains configuration data spread
(interspersed) all over the code. For example:

....
open( "C:\\example\\data.txt" )
....

1st step: give a name to all the config data:

....
project_directory = "C:\\example\\"
open( project_directory + "data.txt" )
....

2nd: move all config data to the top of the source code,
directly after all the import statements:

import ...

project_directory = "C:\\example\\"
....

....
open( project_directory + "data.txt" )
....

3rd: move all config data to a separate "config.py" module:

import ...
import config
....

....
open( config.project_directory + "data.txt" )
....

>but that would mean that every time that I wanted to look
>at a different scenario, I'd need to modify a program.

Now you just have to modify "config.py" - clearly separated
from the (rest of the) "program".

>And when I discover anomalous behavior, I'd need to copy the
>hard-coded data into another program.

Now you just have to import "config.py" from the other program.

On 28/09/2021 10.53, Stefan Ram wrote:
> "Michael F. Stemper" <michael.stemper@gmail.com> writes:
>> Well, I could continue to hard-code the data into one of the test
>> programs
>
> One can employ a gradual path from a program with hardcoded
> data to an entity sharable by different programs.
>
> When I am hurried to rush to a working program, I often
> end up with code that contains configuration data spread
> (interspersed) all over the code. For example:

> 1st step: give a name to all the config data:

> 2nd: move all config data to the top of the source code,
> directly after all the import statements:

> 3rd: move all config data to a separate "config.py" module:
>
> import ...
> import config
> ...
>
> ...
> open( config.project_directory + "data.txt" )
> ...
>
>> but that would mean that every time that I wanted to look
>> at a different scenario, I'd need to modify a program.
>
> Now you just have to modify "config.py" - clearly separated
> from the (rest of the) "program".

Well, that doesn't really address what format to store the data
in. I was going to write a module that would read data from an
XML file:

import EDXML
gens = EDXML.GeneratorsFromXML( "gendata1.xml" )
fuels = EDXML.FuelsFromXML( "fueldata3.xml" )

(Of course, I'd really get the file names from command-line arguments.)

Then I read a web page that suggested use of XML was a poor idea,
so I posted here asking for a clarification and alternate suggestions.

One suggestion was that I use YAML, in which case, I'd write:

import EDfromYAML
gens = EDfromYAML( "gendata1.yaml" )
fuels = EDXML.FuelsFromYAML( "fueldata3.yaml" )

>> And when I discover anomalous behavior, I'd need to copy the
>> hard-coded data into another program.
>
> Now you just have to import "config.py" from the other program.

This sounds like a suggestion that I hard-code the data into a
module. I suppose that I could have half-a-dozen modules with
different data sets and ln them as required:

$ rm GenData.py* FuelData.py*
$ ln gendata1.py GenData.py
$ ln fueldata3.py FuelData.py

It seems to me that a more thorough separation of code and data
might be useful.

--
Michael F. Stemper
The name of the story is "A Sound of Thunder".
It was written by Ray Bradbury. You're welcome.

I replied to Michael privately but am intrigued by his words here:

"The thing that creates realistic test cases is my brain."

I consider extensions to my brain to include using a language like Python on
my computer and in particular, to take a model I think of and instantiate
it. Lots of people have shared modules that can be tweaked to do all kinds
of simulations using a skeleton you provide that guides random number usage.
Some will generate lots of those and stare at them and use their brain to
further narrow it down to realistic ones. For example, in designing say a
car with characteristics like miles per gallon should randomly range between
10 and 100 while engine size ranges from this to that and so on, it may turn
out that large engines don't go well with large number for miles per gallon.

I have worked on projects where a set of guides then created hundreds of
thousands of fairly realistic scenarios using every combination of an
assortment of categorical variables and the rest of the program sliced and
diced the results and did all kinds of statistical calculations and then
generated all kinds of graphs. There was no real data but there was a
generator that was based on the kinds of distributions previously published
in the field that helped guide parameters to be somewhat realistic.

In your case, I understand you will decide how to do it and just note you
used language with multiple meanings that misled a few of us into thinking
you either had a python function in mind using one of several ways Python
refers to as generators, such as one that efficiently yields the next prime
number when asked. Clearly your explanation now shows you plan on making a
handful of data sets by hand using an editor like vi. Fair enough. No need
to write complex software if your mind is easily able to just make half a
dozen variations in files. And, frankly, not sure why you need XML or much
of anything. It obviously depends on how much you are working with and how
variable. For simpler things, you can hard-code your data structure directly
into your program, run an analysis, change the variables to your second
simulation and repeat.

I am afraid that I, like a few others here, assumed a more abstract and much
more complex need to be addressed. Yours may be complex in other parts but
may need nothing much for the part we are talking about. It sounds like you
do want something easier to create while editing.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net@python.org> On
Behalf Of Michael F. Stemper
Sent: Tuesday, September 28, 2021 11:38 AM
To: python-list@python.org
Subject: Re: XML Considered Harmful

On 27/09/2021 20.01, Avi Gross wrote:
> Michael,
>
> Given your further explanation, indeed reading varying numbers of
> points in using a CSV is not valid, albeit someone might just make N
> columns (maybe a few more than 7) to handle a hopefully worst case.
> Definitely it makes more sense to read in a list or other data structure.
>
> You keep talking about generators, though. If the generators are
> outside of your program, then yes, you need to read in whatever they
produce.

My original post (which is as the snows of yesteryear) made explicit the
fact that when I refer to a generator, I'm talking about something made from
tons of iron and copper that is oil-filled and rotates at 1800 rpm.
(In most of the world other than North America, they rotate at 1500 rpm.)

Nothing to do with the similarly-named python construct. Sorry for the
ambiguity.

> But if
> your data generator is within your own program,

The data is created in my mind, and approximates typical physical
characteristics of real generators.

> My impression is you may not be using your set of data points for any
> other purposes except when ready to draw a spline.

Nope, the points give a piecewise-linear curve, and values between two
consecutive points are found by linear interpolation. It's industry standard
practice.

> Can I just ask if by a generator, you do NOT mean the more typical use
> of "generator" as used in python

Nope; I mean something that weighs 500 tons and rotates, producing
electrical energy.

> Do you mean something that creates
> realistic test cases to simulate a real-word scenario?

The thing that creates realistic test cases is my brain.

> These often can
> create everything at once and often based on random numbers.

I have written such, but not in the last thirty years. At that time, I
needed to make up data for fifty or one hundred generators, along with tie
lines and loads.

What I'm working on now only needs a handful of generators at a time; just
enough to test my hypothesis. (Theoretically, I could get by with two, but
that offends my engineering sensibilities.)

> create everything at once and often based on random numbers. Again, if
> you have or build such code, it is not clear it needs to be written to
> disk and then read back.

Well, I could continue to hard-code the data into one of the test programs,
but that would mean that every time that I wanted to look at a different
scenario, I'd need to modify a program. And when I discover anomalous
behavior, I'd need to copy the hard-coded data into another program.

Having the data in a separate file means that I can provide a function to
read that file and return a list of generators (or fuels) to a program.
Multiple test cases are then just multiple files, all of which are available
to multiple programs.

> You may of course want to save it, perhaps as a log, to show what
> your program was working on.

That's another benefit of having the data in external files.

--
Michael F. Stemper
A preposition is something you should never end a sentence with.
--
https://mail.python.org/mailman/listinfo/python-list

Well, Michael, if you want to go back to the eighties, and people you worked
with, I did my Thesis with a professor who later had an Erdős number of 1!
Too bad I never got around to publishing something with him or I could have
been a 2!

But that work, being so long ago, was not in Python but mainly in PASCAL.

Ah the good old days.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net@python.org> On
Behalf Of Michael F. Stemper
Sent: Tuesday, September 28, 2021 11:45 AM
To: python-list@python.org
Subject: Re: XML Considered Harmful

On 28/09/2021 02.25, Peter J. Holzer wrote:
> On 2021-09-27 21:01:04 -0400, Avi Gross via Python-list wrote:
>> You keep talking about generators, though. If the generators are
>> outside of your program, then yes, you need to read in whatever they
produce.
>
> As I understood it, the "generators" don't generate the data, they are
> the subject of the data: Devices that generate electricity by burning
> fuel and he's modelling some aspect of their operation. Maybe
> efficiency or power output or something like that (I tried to search
> for "IHR curve", but couldn't find anything).

If you expand "IHR curve" to "incremental heat rate curve", you'll get
better results. When power engineers talk, we say the first, when we publish
papers, we write the second.

If you want to see the bigger picture, search on "Economic Dispatch".
In fact, doing so points me to something written by a guy I worked with back
in the 1980s:
<http://www2.econ.iastate.edu/classes/econ458/tesfatsion/EconomicDispatchInt
roToOptimization.DKirschen2004.LTEdits.pdf>

Slide 3 even shows a piecewise-linear curve.

--
Michael F. Stemper
A preposition is something you should never end a sentence with.
--
https://mail.python.org/mailman/listinfo/python-list

Am Tue, Sep 28, 2021 at 12:53:49PM -0500 schrieb Michael F. Stemper:

> This sounds like a suggestion that I hard-code the data into a
> module. I suppose that I could have half-a-dozen modules with
> different data sets and ln them as required:
>
> $ rm GenData.py* FuelData.py*
> $ ln gendata1.py GenData.py
> $ ln fueldata3.py FuelData.py

vi data.py

generators = {}
generators['name1'] = {'fuel': ..., ...}
generators['name2'] = {...}
...

vi simulation.py

import sys
import data

generator = data.generators[sys.argv[1]]
run_simulation(generator)

or some such ?

Your data "format" is ... Python code.

Karsten
--
GPG 40BE 5B0E C98E 1713 AFA6 5BC0 3BEA AC80 7D4F C89B

On 28/09/2021 13.27, Avi Gross wrote:
> Well, Michael, if you want to go back to the eighties, and people you worked
> with, I did my Thesis with a professor who later had an Erdős number of 1!
> Too bad I never got around to publishing something with him or I could have
> been a 2!

Lucky you. If a paper that a friend of mine is submitting to various
journals gets accepted by one of them, I'll end up with a 4 or 5 through
him. However, as the months pass, it's looking more like mine will end
up NaN.

--
Michael F. Stemper
Isaiah 58:6-7

Not lucky at all, Michael. The problem is he published a number of things
with Paul Erdős a few years after I got my degrees and went to Bell
laboratories. I never met Erdős but he was prolific and had 507 people
publish with him as co-authors. I would have loved to as I also speak
languages he spoke including Hungarian and Math.

Well, time to get back to something remotely about Python. Is there any
concept of a Rossum Number where anyone who worked directly with Guido Van
Rossum is a 1 (or True or truthy) and ...

Hey I just realized my Berners-Lee number might be 1 but it was so long ago
we worked on what Hypertext should look like, ...

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net@python.org> On
Behalf Of Michael F. Stemper
Sent: Tuesday, September 28, 2021 2:41 PM
To: python-list@python.org
Subject: Re: XML Considered Harmful

On 28/09/2021 13.27, Avi Gross wrote:
> Well, Michael, if you want to go back to the eighties, and people you
> worked with, I did my Thesis with a professor who later had an Erdős
number of 1!
> Too bad I never got around to publishing something with him or I could
> have been a 2!

Lucky you. If a paper that a friend of mine is submitting to various
journals gets accepted by one of them, I'll end up with a 4 or 5 through
him. However, as the months pass, it's looking more like mine will end up
NaN.

--
Michael F. Stemper
Isaiah 58:6-7
--
https://mail.python.org/mailman/listinfo/python-list

"Michael F. Stemper" <michael.stemper@gmail.com> writes:
>Well, that doesn't really address what format to store the data
>in.

The format would be Python.

Some people have worked with compiled languages like C.
So they got used to a separation between a "compile time"
and a "run time". Some data should be loaded at "run time"
because one does not want to "recompile" the project,
so one need to find a format for those data.

Python usually is interpreted, and the separation between
a "compile time" and a "run time" is gone. When you change
a python module, there's no need to "recompile" it.

Sometimes, people work with languages like C that have no
rich sublanguage for data structures like list literals and
dictionary literals. So they need to find a different format
for those data.

Python has a rich sublanguage for list literals and
dictionary literal. So the separation between a "procedure
language" and a "data language" is gone. You can write your
structured data directly in Python.

JSON is a kind of a subset of JavaScript for JavaScript
programmers. In Python, we can use JSON too, or we can
use Python itself.

When some external requirement to use a data exchange
notation like JSON should appear, one can still "translate"
such Python modules to JSON. This path is not blocked.

>This sounds like a suggestion that I hard-code the data into a
>module.

Why do we try to avoid "hard-coding"? Part of this rejection
comes from "hard-coding" at "compile time" what cannot be
changed at "run time". With an interpreted language, this
difference is gone.

Part of the rejection comes from a confusing mixture of
concerns (data vs. code) in a single entity, but this is
gone when you clearly split them using modules.

> I suppose that I could have half-a-dozen modules with
>different data sets and ln them as required:
>$ rm GenData.py* FuelData.py*
>$ ln gendata1.py GenData.py
>$ ln fueldata3.py FuelData.py

Maybe "ln" is a bit inspired by compiler-based development
approaches. Let me call the three lines above a "batch file".
In Python you could use a Python "batch file" instead:

batch.py

import application
application.accept_GenData( "gendata1.py" )
application.import_module( "fueldata3", as_="FuelData" )
application.run()

You configure the application similar to how this was done
using "ln". You tell it to use "fueldata3.py" as it "FuelData".
Then you start it. The file "application.py" does not have to
be modified when changing the configuration, as from "fueldata3"
to "fueldata2", for example.

Of course, you could also read all those names (such as
"gendata1" and "fueldata3") from the commandline:

py example.py gendata1 fueldata3

(source code for this is not included in this post).

For completeness, here are the other files:

application.py:

import importlib

def accept_GenData( gendata_argument ):
global gendata
gendata = gendata_argument

def import_module( module_name, *, as_ ):
globals()[ as_ ]= importlib.import_module( module_name )

def run():
print( "This is the application." )
print( gendata )
FuelData.greet()

fueldata3.py

def greet():
print( "This is fueldata3." )

Output:

This is the application.
gendata1.py
This is fueldata3.

>It seems to me that a more thorough separation of code and data
>might be useful.

Such a separation can be useful indeed, but does not require
the use of two different languages.

On Wed, Sep 29, 2021 at 8:00 AM Stefan Ram <ram@zedat.fu-berlin.de> wrote:
> JSON is a kind of a subset of JavaScript for JavaScript
> programmers. In Python, we can use JSON too, or we can
> use Python itself.
>
> When some external requirement to use a data exchange
> notation like JSON should appear, one can still "translate"
> such Python modules to JSON. This path is not blocked.

JSON exists as a transport mechanism because it is restricted and
can't contain malicious code. A Python equivalent would be
ast.literal_eval - a strict subset of the language but restricted for
safety. For trusted code, yes, straight code can be used.

(And ast.literal_eval, unlike JSON, can handle comments.)

ChrisA

On 29/09/21 4:37 am, Michael F. Stemper wrote:
> I'm talking about something made
> from tons of iron and copper that is oil-filled and rotates at 1800 rpm.

To avoid confusion, we should rename them "electricity comprehensions".

--
Greg

On 29/09/2021 06.53, Michael F. Stemper wrote:
> On 28/09/2021 10.53, Stefan Ram wrote:
>> "Michael F. Stemper" <michael.stemper@gmail.com> writes:
>>> Well, I could continue to hard-code the data into one of the test
>>> programs
>>
>>    One can employ a gradual path from a program with hardcoded
>>    data to an entity sharable by different programs.
>>
>>    When I am hurried to rush to a working program, I often
>>    end up with code that contains configuration data spread
>>    (interspersed) all over the code. For example:
>
>>    1st step: give a name to all the config data:
>
>>    2nd: move all config data to the top of the source code,
>>    directly after all the import statements:
>
>>    3rd: move all config data to a separate "config.py" module:
>>
>> import ...
>> import config
>> ...
>>
>> ...
>> open( config.project_directory + "data.txt" )
>> ...
>>
>>> but that would mean that every time that I wanted to look
>>> at a different scenario, I'd need to modify a program.
>>
>>    Now you just have to modify "config.py" - clearly separated
>>    from the (rest of the) "program".
>
> Well, that doesn't really address what format to store the data
> in. I was going to write a module that would read data from an
> XML file:
>
> import EDXML
> gens = EDXML.GeneratorsFromXML( "gendata1.xml" )
> fuels = EDXML.FuelsFromXML( "fueldata3.xml" )
>
> (Of course, I'd really get the file names from command-line arguments.)
>
> Then I read a web page that suggested use of XML was a poor idea,
> so I posted here asking for a clarification and alternate suggestions.
>
> One suggestion was that I use YAML, in which case, I'd write:
>
> import EDfromYAML
> gens = EDfromYAML( "gendata1.yaml" )
> fuels = EDXML.FuelsFromYAML( "fueldata3.yaml" )
>
>>> And when I discover anomalous behavior, I'd need to copy the
>>> hard-coded data into another program.
>>
>>    Now you just have to import "config.py" from the other program.
>
> This sounds like a suggestion that I hard-code the data into a
> module. I suppose that I could have half-a-dozen modules with
> different data sets and ln them as required:
>
> $ rm GenData.py* FuelData.py*
> $ ln gendata1.py GenData.py
> $ ln fueldata3.py FuelData.py
>
> It seems to me that a more thorough separation of code and data
> might be useful.

Dear Michael,

May I suggest that you are right - and that he is right!
(which is a polite way of saying, also, that both are wrong. Oops!)
(with any and all due apologies)

There are likely cross-purposes here.

I am interpreting various clues, from throughout the thread (from when
the snowflakes were still falling!) that you and I were trained
way-back: to first consider the problem, state the requirements
("hypothesis" in Scientific Method), and work our way to a solution
on-paper. Only when we had a complete 'working solution', did we step up
to the machine (quite possibly a Card Punch, cf a 'computer') and
implement.

Also, that we thought in terms of a clear distinction between
"program[me]" and "data" - and the compiler and link[age]-editor
software technology of the time maintained such.

Whereas 'today', many follow the sequence of "Test-Driven Development"
(er, um, often omitting the initial test) of attempting some idea as
code, reviewing the result, and then "re-factoring" (improving), in a
circular progression - until it not only works, but works well.

This requires similar "stepwise decomposition" to what we learned, but
differs when it comes to code-composition. This approach is more likely
to accumulate a solution 'bottom-up' and component-wise, rather than
creating an entire (and close-to-perfect) solution first and as an whole.

Let's consider the Python REPL. Opening a terminal and starting the
Python interpreter, gives us the opportunity to write short "snippets"
of code and see the results immediately. This is VERY handy for ensuring
that an idea is correct, or to learn exactly how a particular construct
works. Thus, we can 'test' before we write any actual code (and can
copy-paste the successful 'prototype' into our IDE/editor!).

We didn't enjoy such luxury back in the good?bad old days. Young people
today - they just don't know how lucky they are!
(cue other 'grumpy old man' mutterings)

Other points to consider: 'terminals' (cf mainframes), interpreted
languages, and 'immediacy'. These have all brought "opportunities" and
thus "change" to the way developers (can) work and think! (which is why
I outlined what I think of as 'our training' and thus 'our thinking
process' when it comes to software design, above)

Another 'tectonic shift' is that in the old days 'computer time' was
hugely expensive and thus had to be optimised. Whereas these days (even
in retirement) programming-time has become the more expensive component
as computers (or compute-time in cloud-speak) have become cheaper - and
thus we reveal one of THE major attractive attributes of the Python
programming language!

Accordingly, (and now any apologies-due may be due to our colleague -
who was amplifying/making a similar comment to my earlier contribution):
if we decompose the wider-problem into (only) the aspects of collecting
the data, we can assume/estimate/document/refer to that, as a Python
function:

def fetch_operating_parameters():
"""Docstring!"""
pass

(yes, under TDD we would first write a test to call the function and
test its results, but for brevity (hah!) I'll ignore that and stick with
the dev.philosophy point)

Decomposing further, we decide there's a need to pull-in characteristics
of generators, fuel, etc. So, then we can similarly expect to need, and
thus declare, a bunch more functions - with the expectation that they
will probably be called from 'fetch_operating_parameters()'. (because
that was our decomposition hierarchy)

Now, let's return to the program[me] cf data contention. This can also
be slapped-together 'now', and refined/improved 'later'. So, our first
'sub' input function could be:

def fetch_generator_parameters() -> tuple[ dict ]:
"""Another docstring."""
skunk_creek_1 = {
"IHRcurve_name" : "normal",
"63" : "8.513",
"105" : "8.907",
etc
}
...
return skunk_creek_1, ...

Accordingly, if we 'rinse-and-repeat' for each type of input parameter
and flesh-out the coding of the overall input-construct
(fetch_operating_parameters() ) we will be able to at least start
meaningful work on the ensuing "process" and "output" decompositions of
the whole.

(indeed, reverting to the Input-Process-Output overview, if you prefer
to stick with the way we were taught, there's no issue with starting at
'the far end' by writing an output routine and feeding it 'expected
results' as arguments (which you have first calculated on-paper) to
ensure it works, and continuing to work 'backwards' through 'Process' to
'Input'. Whatever 'works' for you!)

Note that this is a Python-code solution to the original post about
'getting data in there'. It is undeniably 'quick-and-dirty', but it is
working, and working 'now'! Secondly, because the total-system only
'sees' a function, you may come back 'later' and improve the
code-within, eg by implementing a JSON-file interface, one for XML, one
for YAML, or whatever your heart-desires - and that you can have the
entire system up-and-running before you get to the stage of 'how can I
make this [quick-and-dirty code] better?'.

(with an alternate possible-conclusion)

Here's where "skill" starts to 'count'. If sufficient forethought went
into constructing the (sub-)function's "signature", changing the code
within the function will not result in any 'ripple' of
consequent-changes throughout the entire system! Thus, as long as
'whatever you decide to do' (initially, and during any
improvements/changes) returns a tuple of dict-s (my example only), you
can keep (learning, experimenting, and) improving the function without
other-cost!

(further reading: the Single Responsibility Principle)

So, compared with our mouldy-old (initial) training, today's approach
seems bumbling and to waste time on producing a first-attempt which
(must) then require time to be improved (and watching folk work, I
regularly have to 'bite my tongue' rather than say something that might
generate philosophical conflict). However, when combined with TDD,
whereby each sub-component is known to be working before it is
incorporated into any larger component of the (and eventually the whole)
solution, we actually find a practical and workable, alternate-approach
to the business of coding!

On 28/09/2021 18.21, Greg Ewing wrote:
> On 29/09/21 4:37 am, Michael F. Stemper wrote:
>> I'm talking about something made
>> from tons of iron and copper that is oil-filled and rotates at 1800 rpm.
>
> To avoid confusion, we should rename them "electricity comprehensions".

Hah!

--
Michael F. Stemper
If you take cranberries and stew them like applesauce they taste much
more like prunes than rhubarb does.

I think that to make electricity comprehend, you need a room temperature
superconductor. The Cooper Pairs took a while to comprehend but now ...

I think, seriously, we have established the problems with guessing that
others are using the language in a way we assume.

So how many comprehensions does Python have?

[] - list comprehension
{} - dictionary OR set comprehension
() - generator expression

Tuples are incomprehensible and I wonder if any other comprehensions might
make sense to add, albeit we may need new symbols.

-----Original Message-----
From: Python-list <python-list-bounces+avigross=verizon.net@python.org> On
Behalf Of Michael F. Stemper
Sent: Wednesday, September 29, 2021 9:04 AM
To: python-list@python.org
Subject: Re: XML Considered Harmful

On 28/09/2021 18.21, Greg Ewing wrote:
> On 29/09/21 4:37 am, Michael F. Stemper wrote:
>> I'm talking about something made
>> from tons of iron and copper that is oil-filled and rotates at 1800 rpm.
>
> To avoid confusion, we should rename them "electricity comprehensions".

Hah!

--
Michael F. Stemper
If you take cranberries and stew them like applesauce they taste much more
like prunes than rhubarb does.
--
https://mail.python.org/mailman/listinfo/python-list