Grizz[l]y,

I think the point is not about a sorted list or sorting in general It is
about reasons why maintaining a data structure such as a list in a program
can be useful beyond printing things once. There are many possible examples
such as having a list of lists containing a record where the third item is a
GPA for the student and writing a little list comprehension that selects a
smaller list containing only students who are Magna Cum Laude or Summa Cum
Laude.

studs = [
["Peter", 82, 3.53],
["Paul", 77, 2.83],
["Mary", 103, 3.82]
]
magna = [stud for stud in studs if stud[2] >= 3.5 ]
summa = [stud for stud in studs if stud[2] >= 3.75 ]

print(studs, magna, summa, sep="\n")

OUTPUT:

>>> print(studs, magna, summa, sep="\n")
[['Peter', 82, 3.53], ['Paul', 77, 2.83], ['Mary', 103, 3.82]]
[['Peter', 82, 3.53], ['Mary', 103, 3.82]]
[['Mary', 103, 3.82]]

Of course, for serious work, some might suggest avoiding constructs like a
list of lists and switch to using modules and data structures that are often
more efficient to represent your data such as some form of matrix or
data.frame.

And, yes, you can sort something like the above by name or GPA or number of
credits taken but the point was responding to why bother making a list just
to print it. The answer is that many and even most programs do a bit more
than that and a good choice of data structure facilitates ...

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Grizzy Adams via Python-list
Sent: Thursday, November 16, 2023 8:41 AM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

Thursday, November 16, 2023 at 7:47, Thomas Passin via Python-list wrote:
Re: Newline (NuBe Question) (at least in part)

>I wrote that you don't need the "students" list, which is correct. But
>there could be a use for a list. It would let you change the order in
>which students appear in the printed output. Knowing how to do that is
>a useful skill. But that should be left for a later lesson, not mixed
>in here.

I have a vague memory of seeing sorted list somewhere ;->)
--
https://mail.python.org/mailman/listinfo/python-list

On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
> Grizz[l]y,
>
> I think the point is not about a sorted list or sorting in general It is
> about reasons why maintaining a data structure such as a list in a program
> can be useful beyond printing things once. There are many possible examples
> such as having a list of lists containing a record where the third item is a
> GPA for the student and writing a little list comprehension that selects a
> smaller list containing only students who are Magna Cum Laude or Summa Cum
> Laude.
>
> studs = [
> ["Peter", 82, 3.53],
> ["Paul", 77, 2.83],
> ["Mary", 103, 3.82]
> ]

I've seen Mary, and she didn't look like a "stud" to me.

> Of course, for serious work, some might suggest avoiding constructs like a
> list of lists and switch to using modules and data structures [...]

Those who would recommend that approach do not appear to include Mr.
Rossum, who said:

Avoid overengineering data structures. Tuples are better than
objects (try namedtuple too though). Prefer simple fields over
getter/setter functions... Built-in datatypes are your friends.
Use more numbers, strings, tuples, lists, sets, dicts. Also
check out the collections library, eps. deque.[1]

I was nodding along with the people saying "list of lists" until I
reread this quote. A list of tuples seems most appropriate to me.

[1] <https://gist.github.com/hemanth/3715502>, as quoted by Bill
Lubanovic in _Introducing Python_

--
Michael F. Stemper
This sentence no verb.

On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
<python-list@python.org> wrote:
>
> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
> > Grizz[l]y,
> >
> > I think the point is not about a sorted list or sorting in general It is
> > about reasons why maintaining a data structure such as a list in a program
> > can be useful beyond printing things once. There are many possible examples
> > such as having a list of lists containing a record where the third item is a
> > GPA for the student and writing a little list comprehension that selects a
> > smaller list containing only students who are Magna Cum Laude or Summa Cum
> > Laude.
> >
> > studs = [
> > ["Peter", 82, 3.53],
> > ["Paul", 77, 2.83],
> > ["Mary", 103, 3.82]
> > ]
>
> I've seen Mary, and she didn't look like a "stud" to me.
>

That's what happens when you abbreviate "student" though :) Don't
worry, there's far FAR worse around the place, and juvenile brains
will always find things to snigger at, usually in mathematical
libraries with "cumulative" functions.

ChrisA

Michael F. Stemper via Python-list schreef op 25/11/2023 om 15:32:
> On 24/11/2023 21.45,avi.e.gross@gmail.com wrote:
> > Grizz[l]y,
> >
> > I think the point is not about a sorted list or sorting in general It is
> > about reasons why maintaining a data structure such as a list in a program
> > can be useful beyond printing things once. There are many possible examples
> > such as having a list of lists containing a record where the third item is a
> > GPA for the student and writing a little list comprehension that selects a
> > smaller list containing only students who are Magna Cum Laude or Summa Cum
> > Laude.
> >
> > studs = [
> > ["Peter", 82, 3.53],
> > ["Paul", 77, 2.83],
> > ["Mary", 103, 3.82]
> > ]
>
> > Of course, for serious work, some might suggest avoiding constructs like a
> > list of lists and switch to using modules and data structures [...]
>
> Those who would recommend that approach do not appear to include Mr.
> Rossum, who said:
>
> Avoid overengineering data structures. Tuples are better than
> objects (try namedtuple too though). Prefer simple fields over
> getter/setter functions... Built-in datatypes are your friends.
> Use more numbers, strings, tuples, lists, sets, dicts. Also
> check out the collections library, eps. deque.[1]
>
> I was nodding along with the people saying "list of lists" until I
> reread this quote. A list of tuples seems most appropriate to me.

I prefer namedtuples or dataclasses over tuples. They allow you to refer
to their fields by name instead of index: student.gpa is much clearer
than student[2], and makes it less likely to accidentally refer to the
wrong field.

--
"Man had always assumed that he was more intelligent than dolphins because
he had achieved so much — the wheel, New York, wars and so on — whilst all
the dolphins had ever done was muck about in the water having a good time.
But conversely, the dolphins had always believed that they were far more
intelligent than man — for precisely the same reasons."
-- Douglas Adams

That is an entirely different discussion, Michael.

I do not know what ideas Guido had ages ago and where he might stand now and
I actually seriously disagree with the snippet you quoted below.

Python was started long ago as a way to improve in some ways on what was
there before. Some of the ideas were nice but also for some purposes, way
too slow.

If you regard the original versions of LISP, they too simplicity to an
extreme and pretty much the main or even only data structure was a list.
Functions like CAR and CDR accessed an element but a complex structure
resulted in people creating functions with names like CAAAAAR and CADADADR
to automate climbing a tree of sorts to get to the parts you want. It was a
recipe for complexity and errors.

My point was that although a list can do so many things in principle, it is
not really optimized to do some things that can be way easier using add-ons
or your own data structures like objects and he notes the collection library
and deque as an example that he is not as much of a purist as you may think.

My point was that if you have a fairly detailed and complex application that
will manipulate lots of data, then instead of reading in a CSV with many
columns and rows recorded into a list of lists, it may make sense to import
numpy and pandas that come with all kinds of functionality built in so you
do not need to re-invent everything. Just how easy is it using lists of
lists to rearrange the order of columns of data, or add new columns
containing calculations built from existing columns and so on?

Of course, for many purposes, it is indeed overkill albeit once you learn a
method, ...

I think part of the design of Python, and this is just my guess, included
going away from overly specific things done in earlier compiled languages
and making it more abstract and inclusive. Arrays or vectors or other such
names would normally require everything to be of the same data type and with
a fixed length. The list data structure loosened this up quite a bit and
also allowed lists within lists. That is great and for some purposes, not
very efficient and especially not when your data actually is all of the same
type or of fixed length. You can make matrix-like data structures of any
depth using lists and it may be hard to traverse such as when you want to
multiply two such 2-D matrices. Place the same data (all say floating point
numbers) in a vector-like structure that also has stored info about the
dimensions, and a simple mathematical calculation accesses any item such as
may_tricks[5,42] in the same amount of time as an offset from the top.

I have seen this phenomenon in many languages where a somewhat clean and
sparse design gets added to, often by others, until some core features are
used less often. An example would be R which does have lists nut they are
just one form of vectors which are really more the core data idea. It also
contains data.frames in the core which are implemented as a list of vectors
and more recently a bit more. It was designed to do statistical tasks as one
of the main objectives. Yet the graphics functions have been added to so
there are by now quite a few independent ways to make graphics using
different paradigms. Python also has something like that. And completely new
paradigms such as piping data in a chain were added in packages and it
became so popular that a version has been added to the core language.

Now although the core language includes lots of the functionality you might
see in numpy/pandas and you can do all kinds of things, some others kept
creating new ways to do things including different data structures that
either dealt with weaknesses found or were ore efficient and so on and an
entire growing body of alternate ways to do things with lots more power and
often more speed that I prefer. A collection of lots of these alternative
packages has been assembled and I and others often simply start programs by
including the "tidyverse" and the resulting programs might as well be
written in a different language to anyone who only knows base R.

But I do not see that as a bad thing albeit someone trying to get a part of
a program from an AI-like service may need to specify or they may get code
they cannot trivially read and evaluate but that works fine once they have
loaded the packages.

Guido is like many others who create or invent and do it in the way they are
proud of. Why change it? But the reality is that first attempts are often
done with lots of room for change and improvement. Now if you are teaching a
course on Python basics, it may be a good idea to teach the basics and
require students to only use in their homework what has already been taught.
But if you get a job where the norm is to use modules like numpy, it makes
sense to use the expanded language if it results in faster writing perhaps
of faster code with fewer mistakes.
-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Michael F. Stemper via Python-list
Sent: Saturday, November 25, 2023 9:32 AM
To: python-list@python.org
Subject: Re: RE: Newline (NuBe Question)

On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
> Grizz[l]y,
>
> I think the point is not about a sorted list or sorting in general It is
> about reasons why maintaining a data structure such as a list in a program
> can be useful beyond printing things once. There are many possible
examples
> such as having a list of lists containing a record where the third item is
a > GPA for the student and writing a little list comprehension that selects a
> smaller list containing only students who are Magna Cum Laude or Summa Cum
> Laude.
>
> studs = [
> ["Peter", 82, 3.53],
> ["Paul", 77, 2.83],
> ["Mary", 103, 3.82]
> ]

I've seen Mary, and she didn't look like a "stud" to me.

> Of course, for serious work, some might suggest avoiding constructs like a
> list of lists and switch to using modules and data structures [...]

Those who would recommend that approach do not appear to include Mr.
Rossum, who said:

Avoid overengineering data structures. Tuples are better than
objects (try namedtuple too though). Prefer simple fields over
getter/setter functions... Built-in datatypes are your friends.
Use more numbers, strings, tuples, lists, sets, dicts. Also
check out the collections library, eps. deque.[1]

I was nodding along with the people saying "list of lists" until I
reread this quote. A list of tuples seems most appropriate to me.

[1] <https://gist.github.com/hemanth/3715502>, as quoted by Bill
Lubanovic in _Introducing Python_

--
Michael F. Stemper
This sentence no verb.

--
https://mail.python.org/mailman/listinfo/python-list

Just FYI, I deliberately chose that abbreviation for a sort of irony as for
some people college is about almost anything except learning and some people
think they are studs and just party and ...

And I am very tired of gender discussions. Lots of words now include two or
even more genders. Women are often now "actors", not actresses. I see no
reason women cannot be studs!

But I learn from criticism. If I ever write a program like that and do not
feel like typing, will this do?

dents = [ ...]

Or will that not include students who happen to be edentulous?

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Chris Angelico via Python-list
Sent: Sunday, November 26, 2023 6:49 AM
To: python-list@python.org
Subject: Re: RE: Newline (NuBe Question)

On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
<python-list@python.org> wrote:
>
> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
> > Grizz[l]y,
> >
> > I think the point is not about a sorted list or sorting in general It is
> > about reasons why maintaining a data structure such as a list in a
program
> > can be useful beyond printing things once. There are many possible
examples
> > such as having a list of lists containing a record where the third item
is a
> > GPA for the student and writing a little list comprehension that selects
a > > smaller list containing only students who are Magna Cum Laude or Summa
Cum
> > Laude.
> >
> > studs = [
> > ["Peter", 82, 3.53],
> > ["Paul", 77, 2.83],
> > ["Mary", 103, 3.82]
> > ]
>
> I've seen Mary, and she didn't look like a "stud" to me.
>

That's what happens when you abbreviate "student" though :) Don't
worry, there's far FAR worse around the place, and juvenile brains
will always find things to snigger at, usually in mathematical
libraries with "cumulative" functions.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list

On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
> > Of course, for serious work, some might suggest avoiding constructs like a
> > list of lists and switch to using modules and data structures [...]
>
> Those who would recommend that approach do not appear to include Mr.
> Rossum, who said:
> Avoid overengineering data structures.
^^^^^^^^^^^^^^^

The key point here is *over*engineering. Don't make things more
complicated than they need to be. But also don't make them simpler than
necessary.

> Tuples are better than objects (try namedtuple too though).

If Guido thought that tuples would always be better than objects, then
Python wouldn't have objects. Why would he add such a complicated
feature to the language if he thought it was useless?

The (unspoken?) context here is "if tuples are sufficient, then ..."

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

On 11/27/2023 12:48 AM, Chris Angelico via Python-list wrote:
> On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
> <python-list@python.org> wrote:
>>
>> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
>>> Grizz[l]y,
>>>
>>> I think the point is not about a sorted list or sorting in general It is
>>> about reasons why maintaining a data structure such as a list in a program
>>> can be useful beyond printing things once. There are many possible examples
>>> such as having a list of lists containing a record where the third item is a
>>> GPA for the student and writing a little list comprehension that selects a
>>> smaller list containing only students who are Magna Cum Laude or Summa Cum
>>> Laude.
>>>
>>> studs = [
>>> ["Peter", 82, 3.53],
>>> ["Paul", 77, 2.83],
>>> ["Mary", 103, 3.82]
>>> ]
>>
>> I've seen Mary, and she didn't look like a "stud" to me.
>>
>
> That's what happens when you abbreviate "student" though :) Don't
> worry, there's far FAR worse around the place, and juvenile brains
> will always find things to snigger at, usually in mathematical
> libraries with "cumulative" functions.

The OP used an abbreviation: "studs". Why? Too lazy to type the full
word? Abbreviation has full-meaning in the (narrow) domain? Was wanting
something funny, or to snigger over?

Was the respondent sniggering? Perhaps he, like the OP, was also saving
typing-time by making a joke, hoping that the OP would see the
implicit-error in expecting others to understand that "studs" meant
"students"?

Actually, Peter, Paul, and Mary were a band
(https://www.peterpaulandmary.com/), so "studs" is even less expressive
when the data also tells a story...

Working with "trainees", I avoid the word "student" even though some
might see them as synonyms. In my mind, the abbreviation did not readily
expand to the full word (mea culpa).

Accordingly, would not pass Code Review!
For the want of a few characters...
(https://en.wikipedia.org/wiki/For_Want_of_a_Nail)

--
Regards =dn

On Mon, 27 Nov 2023 at 06:15, <avi.e.gross@gmail.com> wrote:
> But I learn from criticism. If I ever write a program like that and do not
> feel like typing, will this do?
>
> dents = [ ...]
>
> Or will that not include students who happen to be edentulous?
>

If they're learning to drive, this variable name would make complete sense.

ChrisA

On 11/27/2023 1:08 AM, Roel Schroeven via Python-list wrote:
> I prefer namedtuples or dataclasses over tuples. They allow you to refer
> to their fields by name instead of index: student.gpa is much clearer
> than student[2], and makes it less likely to accidentally refer to the
> wrong field.

+1
readability/comprehension!

--
Regards =dn

On 11/27/2023 10:04 AM, Peter J. Holzer via Python-list wrote:
> On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
>> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
>>> Of course, for serious work, some might suggest avoiding constructs like a
>>> list of lists and switch to using modules and data structures [...]
>>
>> Those who would recommend that approach do not appear to include Mr.
>> Rossum, who said:
>> Avoid overengineering data structures.
> ^^^^^^^^^^^^^^^
>
> The key point here is *over*engineering. Don't make things more
> complicated than they need to be. But also don't make them simpler than
> necessary.
>
>> Tuples are better than objects (try namedtuple too though).
>
> If Guido thought that tuples would always be better than objects, then
> Python wouldn't have objects. Why would he add such a complicated
> feature to the language if he thought it was useless?
>
> The (unspoken?) context here is "if tuples are sufficient, then ..."

At recent PUG-meetings I've listened to a colleague asking questions and
conducting research on Python data-structures*, eg lists-of-lists cf
lists-of-tuples, etc, etc. The "etc, etc" goes on for some time!
Respecting the effort, even as it becomes boringly-detailed, am
encouraging him to publish his findings.

* sadly, he is resistant to OOP and included only a cursory look at
custom-objects, and early in the process. His 'new thinking' has been to
look at in-core databases and the speed-ups SQL (or other) might offer...

However, his motivation came from a particular application, and to
create a naming-system so that he could distinguish a list-of-lists
structure from some other tabular abstraction. The latter enables the
code to change data-format to speed the next process, without the coder
losing-track of the data-type/format.

The trouble is, whereas the research reveals which is faster
(in-isolation, and (only) on his 'platform'), my suspicion is that he
loses all gains by reformatting the data between 'the most efficient'
structure for each step. A problem of only looking at the 'micro',
whilst ignoring wider/macro concerns.

Accordingly, as to the word "engineering" (above), a reminder that we
work in two domains: code and data. The short 'toy examples' in training
courses discourage us from a design-stage for the former - until we
enter 'the real world' and meet a problem/solution too large to fit in a
single human-brain. Sadly, too many of us are pre-disposed to be
math/algorithmically-oriented, and thus data-design is rarely-considered
(in the macro!). Yet, here we are...

--
Regards =dn

Isn't it fascinating that a meaningless piece of code used to illustrate
something can be analyzed as if it was full of malicious content?

Yes, my choice of names was as expected. The numbers chosen had no special
meaning other than choosing one number in each of three equivalence classes.

But, if you want me to add subtle meaning for generations to examine as it
it were a literary work, I offer this:

Peter and Paul were studs who got Mary'd.

Can we now go back to our regularly scheduled talking about aspects of a
computer language?

P.S.
And just for history, Paul was really Noel Paul Stookey but Peter, Paul &
Mary sounded more like new testament characters and I think Noel signifies a
birth to Peter and Mary, sort of, which might have fit too unless it was a
computer program where a name with an umlaut was once not common. Another
interpretation is that Noel came from the Latin word for news. Be that as it
may, and I have no interest in this topic, in the future I may use the ever
popular names of Primus, Secundus and Tertius and get blamed for using
Latin.

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of DL Neil via Python-list
Sent: Sunday, November 26, 2023 4:58 PM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

On 11/27/2023 12:48 AM, Chris Angelico via Python-list wrote:
> On Sun, 26 Nov 2023 at 21:08, Michael F. Stemper via Python-list
> <python-list@python.org> wrote:
>>
>> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
>>> Grizz[l]y,
>>>
>>> I think the point is not about a sorted list or sorting in general It is
>>> about reasons why maintaining a data structure such as a list in a
program
>>> can be useful beyond printing things once. There are many possible
examples
>>> such as having a list of lists containing a record where the third item
is a
>>> GPA for the student and writing a little list comprehension that selects
a >>> smaller list containing only students who are Magna Cum Laude or Summa
Cum
>>> Laude.
>>>
>>> studs = [
>>> ["Peter", 82, 3.53],
>>> ["Paul", 77, 2.83],
>>> ["Mary", 103, 3.82]
>>> ]
>>
>> I've seen Mary, and she didn't look like a "stud" to me.
>>
>
> That's what happens when you abbreviate "student" though :) Don't
> worry, there's far FAR worse around the place, and juvenile brains
> will always find things to snigger at, usually in mathematical
> libraries with "cumulative" functions.

The OP used an abbreviation: "studs". Why? Too lazy to type the full
word? Abbreviation has full-meaning in the (narrow) domain? Was wanting
something funny, or to snigger over?

Was the respondent sniggering? Perhaps he, like the OP, was also saving
typing-time by making a joke, hoping that the OP would see the
implicit-error in expecting others to understand that "studs" meant
"students"?

Actually, Peter, Paul, and Mary were a band
(https://www.peterpaulandmary.com/), so "studs" is even less expressive
when the data also tells a story...

Working with "trainees", I avoid the word "student" even though some
might see them as synonyms. In my mind, the abbreviation did not readily
expand to the full word (mea culpa).

Accordingly, would not pass Code Review!
For the want of a few characters...
(https://en.wikipedia.org/wiki/For_Want_of_a_Nail)

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

On Mon, 27 Nov 2023 at 13:52, AVI GROSS via Python-list
<python-list@python.org> wrote:
> Be that as it
> may, and I have no interest in this topic, in the future I may use the ever
> popular names of Primus, Secundus and Tertius and get blamed for using
> Latin.
>

Imperious Prima flashes forth her edict to "begin it". In gentler tone
Secunda hopes there will be nonsense in it. While Tertia interrupts
the tale not more than once a minute.

ChrisA

Dave,

Back on a hopefully more serious note, I want to make a bit of an analogy
with what happens when you save data in a format like a .CSV file.

Often you have a choice of including a header line giving names to the
resulting columns, or not.

If you read in the data to some structure, often to some variation I would
loosely call a data.frame or perhaps something like a matrix, then without
headers you have to specify what you want positionally or create your own
names for columns to use. If names are already there, your program can
manipulate things by using the names and if they are well chosen, with no
studs among them, the resulting code can be quite readable. More
importantly, if the data being read changes and includes additional columns
or in a different order, your original program may run fine as long as the
names of the columns you care about remain the same.

Positional programs can be positioned to fail in quite subtle ways if the
positions no longer apply.

As I see it, many situations where some aspects are variable are not ideal
for naming. A dictionary is an example that is useful when you have no idea
how many items with unknown keys may be present. You can iterate over the
names that are there, or use techniques that detect and deal with keys from
your list that are not present. Not using names/keys here might involve a
longer list with lots of empty slots to designate missing items, This
clearly is not great when the data present is sparse or when the number of
items is not known in advance or cannot be maintained in the right order.

There are many other situations with assorted tradeoffs and to insist on
using lists/tuples exclusively would be silly but at the same time, if you
are using a list to hold the real and imaginary parts of a complex number,
or the X/Y[/Z] coordinates of a point where the order is almost universally
accepted, then maybe it is not worth using a data structure more complex or
derived as the use may be obvious.

I do recall odd methods sometimes used way back when I programmed in C/C++
or similar languages when some method was used to declare small constants
like:

#define FIRSTNAME 1
#define LASTNAME 2

Or concepts like "const GPA = 3"

And so on, so code asking for student_record[LASTNAME] would be a tad more
readable and if the order of entries somehow were different, just redefine
the constant.

In some sense, some of the data structures we are discussing, under the
hood, actually may do something very similar as they remap the name to a
small integer offset. Others may do much more or be slower but often add
value in other ways. A full-blown class may not just encapsulate the names
of components of an object but verify the validity of the contents or do
logging or any number of other things. Using a list or tuple does nothing
else.

So if you need nothing else, they are often suitable and sometimes even
preferable.

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of DL Neil via Python-list
Sent: Sunday, November 26, 2023 5:19 PM
To: python-list@python.org
Subject: Re: Newline (NuBe Question)

On 11/27/2023 10:04 AM, Peter J. Holzer via Python-list wrote:
> On 2023-11-25 08:32:24 -0600, Michael F. Stemper via Python-list wrote:
>> On 24/11/2023 21.45, avi.e.gross@gmail.com wrote:
>>> Of course, for serious work, some might suggest avoiding constructs like
a >>> list of lists and switch to using modules and data structures [...]
>>
>> Those who would recommend that approach do not appear to include Mr.
>> Rossum, who said:
>> Avoid overengineering data structures.
> ^^^^^^^^^^^^^^^
>
> The key point here is *over*engineering. Don't make things more
> complicated than they need to be. But also don't make them simpler than
> necessary.
>
>> Tuples are better than objects (try namedtuple too though).
>
> If Guido thought that tuples would always be better than objects, then
> Python wouldn't have objects. Why would he add such a complicated
> feature to the language if he thought it was useless?
>
> The (unspoken?) context here is "if tuples are sufficient, then ..."

At recent PUG-meetings I've listened to a colleague asking questions and
conducting research on Python data-structures*, eg lists-of-lists cf
lists-of-tuples, etc, etc. The "etc, etc" goes on for some time!
Respecting the effort, even as it becomes boringly-detailed, am
encouraging him to publish his findings.

* sadly, he is resistant to OOP and included only a cursory look at
custom-objects, and early in the process. His 'new thinking' has been to
look at in-core databases and the speed-ups SQL (or other) might offer...

However, his motivation came from a particular application, and to
create a naming-system so that he could distinguish a list-of-lists
structure from some other tabular abstraction. The latter enables the
code to change data-format to speed the next process, without the coder
losing-track of the data-type/format.

The trouble is, whereas the research reveals which is faster
(in-isolation, and (only) on his 'platform'), my suspicion is that he
loses all gains by reformatting the data between 'the most efficient'
structure for each step. A problem of only looking at the 'micro',
whilst ignoring wider/macro concerns.

Accordingly, as to the word "engineering" (above), a reminder that we
work in two domains: code and data. The short 'toy examples' in training
courses discourage us from a design-stage for the former - until we
enter 'the real world' and meet a problem/solution too large to fit in a
single human-brain. Sadly, too many of us are pre-disposed to be
math/algorithmically-oriented, and thus data-design is rarely-considered
(in the macro!). Yet, here we are...

--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list

Avi,

On 11/27/2023 4:15 PM, avi.e.gross@gmail.com wrote:
> Dave,
>
> Back on a hopefully more serious note, I want to make a bit of an analogy
> with what happens when you save data in a format like a .CSV file.
>
> Often you have a choice of including a header line giving names to the
> resulting columns, or not.
>
> If you read in the data to some structure, often to some variation I would
> loosely call a data.frame or perhaps something like a matrix, then without
> headers you have to specify what you want positionally or create your own
> names for columns to use. If names are already there, your program can
> manipulate things by using the names and if they are well chosen, with no
> studs among them, the resulting code can be quite readable. More
> importantly, if the data being read changes and includes additional columns
> or in a different order, your original program may run fine as long as the
> names of the columns you care about remain the same.
>
> Positional programs can be positioned to fail in quite subtle ways if the
> positions no longer apply.

Must admit to avoiding .csv files, if possible, and working directly
with the .xls? original (cf expecting the user to export the .csv - and
NOT change the worksheet thereafter).

However, have recently been using the .csv format (as described) as a
placeholder or introduction to formatting data for an RDBMS.

In a tabular structure, the expectation is that every field (column/row
intersection) will contain a value. In the RDBMS-world, if the value is
not-known then it will be recorded as NULL (equivalent of Python's None).

Accordingly, two points:
1 the special case of missing/unavailable data can be handled with ease,
2 most 'connector' interfaces will give the choice of retrieving data
into a tuple or a dictionary (where the keys are the column-names). The
latter easing data-identification issues (as described) both in terms of
improving over relational-positioning and name-continuity (or column
changes/expansions).

The point about data 'appearing' without headings should be considered
carefully. The phrase "create your own names for columns" only vaguely
accesses the problem. If someone else has created/provided the data,
then we need to know the exact design (schema = rules). What is the
characteristic of each component? Not only column-names, but also what
is the metric (eg the infamous confusion between feet and meters)...

> As I see it, many situations where some aspects are variable are not ideal
> for naming. A dictionary is an example that is useful when you have no idea
> how many items with unknown keys may be present. You can iterate over the
> names that are there, or use techniques that detect and deal with keys from
> your list that are not present. Not using names/keys here might involve a
> longer list with lots of empty slots to designate missing items, This
> clearly is not great when the data present is sparse or when the number of
> items is not known in advance or cannot be maintained in the right order.

Agreed, and this is the draw-back incurred by folk who wish to take
advantage of the schema-less (possibility) NoSQL DBs. The DB enjoys
flexibility, but the downstream-coder has to contort and flex to cope.

In this case, JSON files are an easy place-holder/intro for NoSQL DBs -
in fact, Python dicts and MongoDB go hand-in-glove.

The next issue raised is sparseness. In a table, the assumption is that
all fields, or at least most of them, will be filled with values.
However, a sparse matrix would make such very 'expensive' in terms of
storage-space (efficacy).

Accordingly, there are other ways of doing things. All of these involve
labeling each data-item (thus, the data expressed as a table needs to be
at least 50% empty to justify the structural change).

In this case, one might consider a tree-type of structure - and if we
have to continue the pattern, we might look at a Network Database
methodology (as distinct from a DB on a network!)

> There are many other situations with assorted tradeoffs and to insist on
> using lists/tuples exclusively would be silly but at the same time, if you
> are using a list to hold the real and imaginary parts of a complex number,
> or the X/Y[/Z] coordinates of a point where the order is almost universally
> accepted, then maybe it is not worth using a data structure more complex or
> derived as the use may be obvious.

No argument (in case anyone thought I might...)

See @Peter's earlier advice.

Much of the consideration (apart from mutable/immutable) is likely to be
ease of coding. Getting down 'into the weeds' is probably pointless
unless questions are being asked about (execution-time) performance...

Isn't the word "obvious" where this discussion started? Whereas "studs"
might be an "obvious" abbreviation for "students" to some, it is not to
others (quite aside from the abbreviation being unnecessary in this
day-and-age).

Curiously, whereas I DO happen to think a point as ( x, y, ) or ( x, y,
z, ) and thus quite happily interpret ( 1, 2, 3, ) as a location in 3D
space, I had a trainee bring a 'problem' on this exact assumption:-

He had two positions ( x1, y1, ) and ( x2, y2, ) and was computing the
vector between them ( x2 - x1, y2 - y1 ), accordingly:

def compute_distance( x1, x2, y1, y2, ):
# with return calculated as above

Trouble is, the function-call was:

result = compute_distance( x1, y1, x2, y2, )

In other words, the function's signature was consistent with the
calculation. Whereas, the function-call was consistent with the way the
data had 'arrived'. Oops!

As soon as a (data)class Point( x, y, ) was created, the function's
signature became:

def compute_distance( starting_point:Point, ending_point:Point, ):
# with amended return calculation

and the function-call became congruent, naturally.
(in fact, the function was moved into the dataclass to become a method
which simplified the signature and call(s) )

Thus, what was "obvious" to the same guy's brain when he was writing the
function, and what seemed "obvious" when the function was being used,
were materially (and catastrophically) different!

So, even though we (two) might think in terms of "universally", we
are/were wrong!

Thus, a DESIGNED data-type helps to avoid errors, and even when the
data-usage seems "obvious", offers advantage!

Once again, am tempted to suggest that the saving of:

point = ( 1, 2, )

over:

@dataclass
class Point():
x:float
y:float

is about as easily justified as preferring "studs" over the complete
word "students".

YMMV!
(excepting Code Review expectations)

* will an AI-Assistant code this for us, and thus remove any 'amount of
typing' complaint?

> I do recall odd methods sometimes used way back when I programmed in C/C++
> or similar languages when some method was used to declare small constants
> like:
>
> #define FIRSTNAME 1
> #define LASTNAME 2
>
> Or concepts like "const GPA = 3"
>
> And so on, so code asking for student_record[LASTNAME] would be a tad more
> readable and if the order of entries somehow were different, just redefine
> the constant.

I've been known to do this in Python too! This example is congruent with
what was mentioned (elsewhere/earlier): that LASTNAME is considerably
more meaningful than 2.

Programming principles includes advice that all 'magic constants' should
be hoisted to the top of the code (along with import-statements). Aren't
those positional indices 'magic constants'?

> In some sense, some of the data structures we are discussing, under the
> hood, actually may do something very similar as they remap the name to a
> small integer offset. Others may do much more or be slower but often add
> value in other ways. A full-blown class may not just encapsulate the names
> of components of an object but verify the validity of the contents or do
> logging or any number of other things. Using a list or tuple does nothing
> else.

Not in Python: database keys must be hashable values - for that reason.

Argh! The docs (https://docs.python.org/3/tutorial/datastructures.html)
don't say that - or don't say it any more. Did it change when key-order
became guaranteed, or do I mis-remember?

Those docs say "immutable" - but whilst "hashable" and "immutable" have
related meanings, they are not exactly the same in effect.

Alternately, the wiki (https://wiki.python.org/moin/DictionaryKeys) does
say "hashable"!

> So if you need nothing else, they are often suitable and sometimes even
> preferable.

Yes, (make a conscious choice to) use the best tool for the job - but
don't let bias cloud your judgement, don't take the ideas of the MD's
nephew as 'Gospel', and DO design the way forward...

Dave, I gave an example, again, and make no deep claims so your comments may be valid, without any argument.

I mentioned CSV and a related family such as TSV as they were a common and simple data format that has long been used. There are oodles of others and yes, these days many people can read directly from formats like some from EXCEL. But for data that can be shared to almost anyone using anything, something like Comma Separated Values is often used.

And some programs that generate such data simply keep appending a line at a time to a file and do not have any header line. There are even some programs that may not tolerate a file with a header line, or comments or other optional things, and some where header lines you can create would cause problems such as using an extended character set or escaped characters.

I have worked with these files in many languages and environments and my thought process here focused on recent work in R, albeit much applies everywhere. My point was really not about CSV but the convenience and advantages of data structures you can access by name when you want and sometimes also by position when you want. Too many errors can happen when humans doing programming are not able to concentrate. It is similar to arguments about file names. In the old UNIX days, and the same for other systems like VMS, a filename tended to have a format where relatively few characters were allowed and it might have two parts with the latter being an extension of up to 3 characters, or whatever. So file names like A321G12.dat were common and also next to it similar unpronounceable other file names. It was easy to confuse them and even people who worked with them regularly would forget what it might mean or use the wrong one.

Well, if I load in a CSV in a language like R and there is no header line, as with some other data structures, it may make up a placeholder set of names like V1, V2 and so on. Yes, there are ways to specify the names as they are read in or afterward and they can be changed. But I have seen lots of CSV files offered with way too many columns and no names as well as documentation suggesting what names can be added if you wish.

This may be a bit off topic, but I want to add a bit in this context about additional concepts regarding name. As mentioned, there is a whole set of add-ons people sometimes use and in R, I like the tidyverse family and it allows some fairly sophisticated things to be done using names. There are ways to specify you want a subset of a data.frame (sometimes a version called a tibble) and you can ask for say all columns starting with "xyz" or containing it or ending with it. That can be very helpful if say we wave columns containing the height and weight and other metrics of say people in three clinics and your column names embed the name of the clinic, or other such examples, and you want to select one grouping for processing. You cannot easily do that without external info is it is just positional.

An extension of this is how compactly you can do fairly complex things such as asking to create lots of new columns using calculations. You can specify, as above, which sets of columns to do this too and that you want the results for each XYY in XYZ.mean and XYZ.std and so on. You can skip oodles of carefully crafted and nested loops because of the ability to manipulate using column names at a high and often abstract level.

And, just FYI, many other structures such as lists in R also support names for components. It can be very useful. But the overall paradigm compared to Python has major differences and I see strengths and weaknesses and tradeoffs.

Your dictionary example is one of them as numpy/pandas often make good use of them as part of dealing with similar data.frame type structures that are often simpler or easier to code with.

There is lots of AI discussion these days and some of what you say is applicable in that additional info besides names might be useful in the storage format to make processing it more useful. That is available in formats related to XML where fairly arbitrary markup can be made available.

Have to head out as this is already long enough.

-----Original Message-----
From: 'DL Neil' <PythonList@danceswithmice.info>
Sent: Monday, November 27, 2023 2:49 AM
To: avi.e.gross@gmail.com; python-list@python.org
Subject: Re: Newline (NuBe Question)

Avi,

On 11/27/2023 4:15 PM, avi.e.gross@gmail.com wrote:
> Dave,
>
> Back on a hopefully more serious note, I want to make a bit of an analogy
> with what happens when you save data in a format like a .CSV file.
>
> Often you have a choice of including a header line giving names to the
> resulting columns, or not.
>
> If you read in the data to some structure, often to some variation I would
> loosely call a data.frame or perhaps something like a matrix, then without
> headers you have to specify what you want positionally or create your own
> names for columns to use. If names are already there, your program can
> manipulate things by using the names and if they are well chosen, with no
> studs among them, the resulting code can be quite readable. More
> importantly, if the data being read changes and includes additional columns
> or in a different order, your original program may run fine as long as the
> names of the columns you care about remain the same.
>
> Positional programs can be positioned to fail in quite subtle ways if the
> positions no longer apply.

Must admit to avoiding .csv files, if possible, and working directly
with the .xls? original (cf expecting the user to export the .csv - and
NOT change the worksheet thereafter).

However, have recently been using the .csv format (as described) as a
placeholder or introduction to formatting data for an RDBMS.

In a tabular structure, the expectation is that every field (column/row
intersection) will contain a value. In the RDBMS-world, if the value is
not-known then it will be recorded as NULL (equivalent of Python's None).

Accordingly, two points:
1 the special case of missing/unavailable data can be handled with ease,
2 most 'connector' interfaces will give the choice of retrieving data
into a tuple or a dictionary (where the keys are the column-names). The
latter easing data-identification issues (as described) both in terms of
improving over relational-positioning and name-continuity (or column
changes/expansions).

The point about data 'appearing' without headings should be considered
carefully. The phrase "create your own names for columns" only vaguely
accesses the problem. If someone else has created/provided the data,
then we need to know the exact design (schema = rules). What is the
characteristic of each component? Not only column-names, but also what
is the metric (eg the infamous confusion between feet and meters)...

> As I see it, many situations where some aspects are variable are not ideal
> for naming. A dictionary is an example that is useful when you have no idea
> how many items with unknown keys may be present. You can iterate over the
> names that are there, or use techniques that detect and deal with keys from
> your list that are not present. Not using names/keys here might involve a
> longer list with lots of empty slots to designate missing items, This
> clearly is not great when the data present is sparse or when the number of
> items is not known in advance or cannot be maintained in the right order.

Agreed, and this is the draw-back incurred by folk who wish to take
advantage of the schema-less (possibility) NoSQL DBs. The DB enjoys
flexibility, but the downstream-coder has to contort and flex to cope.

In this case, JSON files are an easy place-holder/intro for NoSQL DBs -
in fact, Python dicts and MongoDB go hand-in-glove.

The next issue raised is sparseness. In a table, the assumption is that
all fields, or at least most of them, will be filled with values.
However, a sparse matrix would make such very 'expensive' in terms of
storage-space (efficacy).

Accordingly, there are other ways of doing things. All of these involve
labeling each data-item (thus, the data expressed as a table needs to be
at least 50% empty to justify the structural change).

In this case, one might consider a tree-type of structure - and if we
have to continue the pattern, we might look at a Network Database
methodology (as distinct from a DB on a network!)

> There are many other situations with assorted tradeoffs and to insist on
> using lists/tuples exclusively would be silly but at the same time, if you
> are using a list to hold the real and imaginary parts of a complex number,
> or the X/Y[/Z] coordinates of a point where the order is almost universally
> accepted, then maybe it is not worth using a data structure more complex or
> derived as the use may be obvious.

No argument (in case anyone thought I might...)

See @Peter's earlier advice.

Much of the consideration (apart from mutable/immutable) is likely to be
ease of coding. Getting down 'into the weeds' is probably pointless
unless questions are being asked about (execution-time) performance...

Isn't the word "obvious" where this discussion started? Whereas "studs"
might be an "obvious" abbreviation for "students" to some, it is not to
others (quite aside from the abbreviation being unnecessary in this
day-and-age).