Is there a neat, pythonic way to store values which are 'sometimes'
changed?

My particular case at the moment is calibration values for ADC inputs
which are set by running a calibration program and used by lots of
programs which display the values or do calculations with them.

From the program readability point of view it would be good to have a
Python module with the values in it but using a Python program to
write/update a Python module sounds a bit odd somehow.

I could simply write the values to a file (or a database) and I
suspect that this may be the best answer but it does make retrieving
the values different from getting all other (nearly) constant values.

Are there any Python modules aimed specifically at this sort of
requirement?

--
Chris Green
·

Chris Green <cl@isbd.net> writes:
>Is there a neat, pythonic way to store values which are 'sometimes'
>changed?

If you're comfortable with SQL, there is sqlite.

I'd write a small wrapper class around it, so that only this
wrapper comes in contact with any SQL. Then you can change
it later to store data in some other way (if necessary).

>I could simply write the values to a file (or a database) and I
>suspect that this may be the best answer but it does make retrieving
>the values different from getting all other (nearly) constant values.

They are not /constant/ values.

When I was young and started to learn BASIC, one of the first
idea for my own programming language was the possibility to
define durations for variables. For example,

perm int i;

. Being declared "perm", your variable "i" now keeps its value
across program instances. So, when you say, "i = 27;", and do
not change it afterwars, and only one instance of your program
is running, next time you start your program, "i" will be 27.

A lazy way to save values would be to use one value per file.
(You do not need a special language like JSON, then.)

# WARNING: DO NOT RUN! (Program will override files named "x" and "y"!)

def load( filename ):
try:
with open( filename )as input_stream:
result = int( next( input_stream ))
except FileNotFoundError:
result = 0
return result

def save( value, filename ):
with open( filename, 'w' )as output_stream:
print( value, file=output_stream )

x = load( "x" )
y = load( "y" )
print( f'{x=}' )
print( f'{y=}' )
x += 1
y -= 1
save( x, "x" )
save( y, "y" )

On 2023-12-05 14:37, Chris Green via Python-list wrote:
> Is there a neat, pythonic way to store values which are 'sometimes'
> changed?
>
> My particular case at the moment is calibration values for ADC inputs
> which are set by running a calibration program and used by lots of
> programs which display the values or do calculations with them.
>
> From the program readability point of view it would be good to have a
> Python module with the values in it but using a Python program to
> write/update a Python module sounds a bit odd somehow.
>
> I could simply write the values to a file (or a database) and I
> suspect that this may be the best answer but it does make retrieving
> the values different from getting all other (nearly) constant values.
>
> Are there any Python modules aimed specifically at this sort of
> requirement?
>
Some kind of key/value store sounds like the correct solution. I
wouldn't go as far a database - that's overkill for a few calibration
values.

I might suggest TOML, except that Python's tomllib (Python 3.11+) is
read-only!

Personally, I'd go for lines of:

key1: value1
key2: value2

Simple to read, simple to write.

Chris Green <cl@isbd.net> writes:
> I could simply write the values to a file (or a database) and I
> suspect that this may be the best answer but it does make retrieving
> the values different from getting all other (nearly) constant values.

I've used configparser for this, though its intention is files that are
manually edited, rather than updated by a program.

You can also read and dump a json file or pickle file. I prefer json to
pickle for most purposes these days.

I don't like YAML and I don't like the proliferation of markup formats
of this type. So while I don't know exactly what TOML is, I figure it
must be bad.

I sometimes use ast.literal_eval though it is Python specific.

Of course there is also sqlite but that is probably overkill.

On 12/5/23 07:37, Chris Green via Python-list wrote:
> Is there a neat, pythonic way to store values which are 'sometimes'
> changed?
>
> My particular case at the moment is calibration values for ADC inputs
> which are set by running a calibration program and used by lots of
> programs which display the values or do calculations with them.
>
> From the program readability point of view it would be good to have a
> Python module with the values in it but using a Python program to
> write/update a Python module sounds a bit odd somehow.
>
> I could simply write the values to a file (or a database) and I
> suspect that this may be the best answer but it does make retrieving
> the values different from getting all other (nearly) constant values.
>
> Are there any Python modules aimed specifically at this sort of
> requirement?

A search term to look for is "data persistence"

there is lots of support at various levels - you can do simpler things
with plain text (or binary), json data, or csv data, or configparser, or
use pickles; if there's not a lot of values a dbapi database may, as
already mentioned, be overkill.

> On 5 Dec 2023, at 14:37, Chris Green via Python-list <python-list@python.org> wrote:
>
> Are there any Python modules aimed specifically at this sort of
> requirement?

I tend to use JSON for this type of thing.
Suggest that you use the options to pretty print the json that is saved so that a human can read it.

For example:
----
import json

config = {
'key2': 42,
'key1': 'value 1',
}

print(json.dumps(config, indent=4, separators=(', ', ': '), sort_keys=True))

----
% python $T/a.py
{ "key1": "value 1",
"key2": 42
}

Barry

On 12/6/23 03:37, Chris Green via Python-list wrote:
> Is there a neat, pythonic way to store values which are 'sometimes'
> changed?
>
> My particular case at the moment is calibration values for ADC inputs
> which are set by running a calibration program and used by lots of
> programs which display the values or do calculations with them.
>
> From the program readability point of view it would be good to have a
> Python module with the values in it but using a Python program to
> write/update a Python module sounds a bit odd somehow.
>
> I could simply write the values to a file (or a database) and I
> suspect that this may be the best answer but it does make retrieving
> the values different from getting all other (nearly) constant values.
>
> Are there any Python modules aimed specifically at this sort of
> requirement?

Another programming-term for these might be "environment variables".
However, be aware that such also has a specific meaning at the Operating
System level.

1 Sysops Environment Variables exist completely outside the code. Python
interrogates the Sysops to fetch/set them. Can be problematic because
only apply on single machine and are not part of change-control, VS, etc.

2 A .con file (in my tradition, likely still .uni type in MSFT) or
similar, which contains the key:value pairs recommended elsewhere. There
are formal .con and .uni (etc) formats. Most of the teams I've
worked-with recently seem to settle on .JSON files which are
very-conveniently structured as (read into/written from) a Python
dictionary, and a single function-call interaction.

Word of warning/voice of [bitter] experience: ALWAYS *trumpet* any
changes in these values AND output them (as you would any "assumptions"
for a calculation) at the top of output reports. The trouble is that
humans assume continuity but such an arrangement is NOT idempotent -
which leads to complaints: "I ran it on Monday and got these results,
but when I ran it again on Tuesday, the results changed"...

Yes there are Python libraries. Choose your method/format first, and
then search - either Duckboards or straight from Pepi.

I have systems which use an DBMS for environment variables, but (a) only
when there's a significant number, and (b) when the application is
already connecting to the DBMS for processing [and maybe (c) because I
know my way around such tools so they're 'easy']. Not recommended!

--
Regards =dn

Apologies: neglected suggested web.refs:

https://datagy.io/python-environment-variables/
https://pypi.org/project/json_environ/

--
Regards =dn

On 12/5/2023 11:50 AM, MRAB via Python-list wrote:
> On 2023-12-05 14:37, Chris Green via Python-list wrote:
>> Is there a neat, pythonic way to store values which are 'sometimes'
>> changed?
>>
>> My particular case at the moment is calibration values for ADC inputs
>> which are set by running a calibration program and used by lots of
>> programs which display the values or do calculations with them.
>>
>>  From the program readability point of view it would be good to have a
>> Python module with the values in it but using a Python program to
>> write/update a Python module sounds a bit odd somehow.
>>
>> I could simply write the values to a file (or a database) and I
>> suspect that this may be the best answer but it does make retrieving
>> the values different from getting all other (nearly) constant values.
>>
>> Are there any Python modules aimed specifically at this sort of
>> requirement?
>>
> Some kind of key/value store sounds like the correct solution. I
> wouldn't go as far a database - that's overkill for a few calibration
> values.
>
> I might suggest TOML, except that Python's tomllib (Python 3.11+) is
> read-only!
>
> Personally, I'd go for lines of:
>
>     key1: value1
>     key2: value2
>
> Simple to read, simple to write.

Just go with an .ini file. Simple, well-supported by the standard
library. And it gives you key/value pairs.

Thomas Passin <list1@tompassin.net> wrote:
> On 12/5/2023 11:50 AM, MRAB via Python-list wrote:
> > On 2023-12-05 14:37, Chris Green via Python-list wrote:
> >> Is there a neat, pythonic way to store values which are 'sometimes'
> >> changed?
> >>
> >> My particular case at the moment is calibration values for ADC inputs
> >> which are set by running a calibration program and used by lots of
> >> programs which display the values or do calculations with them.
> >>
> >>  From the program readability point of view it would be good to have a
> >> Python module with the values in it but using a Python program to
> >> write/update a Python module sounds a bit odd somehow.
> >>
> >> I could simply write the values to a file (or a database) and I
> >> suspect that this may be the best answer but it does make retrieving
> >> the values different from getting all other (nearly) constant values.
> >>
> >> Are there any Python modules aimed specifically at this sort of
> >> requirement?
> >>
> > Some kind of key/value store sounds like the correct solution. I
> > wouldn't go as far a database - that's overkill for a few calibration
> > values.
> >
> > I might suggest TOML, except that Python's tomllib (Python 3.11+) is
> > read-only!
> >
> > Personally, I'd go for lines of:
> >
> >     key1: value1
> >     key2: value2
> >
> > Simple to read, simple to write.
>
> Just go with an .ini file. Simple, well-supported by the standard
> library. And it gives you key/value pairs.
>
My requirement is *slightly* more complex than just key value pairs,
it has one level of hierarchy, e.g.:-

KEY1:
a: v1
c: v3
d: v4
KEY2:
a: v7
b: v5
d: v6

Different numbers of value pairs under each KEY.

--
Chris Green
·

Paul Rubin <no.email@nospam.invalid> wrote:
> Chris Green <cl@isbd.net> writes:
> > I could simply write the values to a file (or a database) and I
> > suspect that this may be the best answer but it does make retrieving
> > the values different from getting all other (nearly) constant values.
>
> I've used configparser for this, though its intention is files that are
> manually edited, rather than updated by a program.
>
> You can also read and dump a json file or pickle file. I prefer json to
> pickle for most purposes these days.
>
> I don't like YAML and I don't like the proliferation of markup formats
> of this type. So while I don't know exactly what TOML is, I figure it
> must be bad.
>
> I sometimes use ast.literal_eval though it is Python specific.
>
That's interesting, I'll add it to my armoury anyway. :-)

> Of course there is also sqlite but that is probably overkill.

It's what my current code uses but does feel a bit OTT and it isn't
particularly convenient to view when debugging.

--
Chris Green
·

Chris Green <cl@isbd.net> writes:
> KEY1:
> a: v1
> c: v3
> d: v4
> KEY2:
> a: v7
> b: v5
> d: v6

That maps nicely to two directories with three files
(under an application-specific configuration directory).

Thank you everyone for all the suggestions, I now have several
possibilities to follow up. :-)

--
Chris Green
·

On 2023-12-06 at 09:32:02 +0000,
Chris Green via Python-list <python-list@python.org> wrote:

> Thomas Passin <list1@tompassin.net> wrote:

[...]

> > Just go with an .ini file. Simple, well-supported by the standard
> > library. And it gives you key/value pairs.
> >
> My requirement is *slightly* more complex than just key value pairs,
> it has one level of hierarchy, e.g.:-
>
> KEY1:
> a: v1
> c: v3
> d: v4
> KEY2:
> a: v7
> b: v5
> d: v6
>
> Different numbers of value pairs under each KEY.

INI files have sections.

See <https://en.wikipedia.org/wiki/INI_file#Sections>.

On 2023-12-06, Stefan Ram wrote:
> Chris Green <cl@isbd.net> writes:
>> KEY1:
>> a: v1
>> c: v3
>> d: v4
>> KEY2:
>> a: v7
>> b: v5
>> d: v6
>
> That maps nicely to two directories with three files
> (under an application-specific configuration directory).

Or an .ini file with two sections (although I don't think you can re-use
key-names in a single ini file)

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

> On 6 Dec 2023, at 09:32, Chris Green via Python-list <python-list@python.org> wrote:
>
> My requirement is *slightly* more complex than just key value pairs,
> it has one level of hierarchy, e.g.:-
>
> KEY1:
> a: v1
> c: v3
> d: v4
> KEY2:
> a: v7
> b: v5
> d: v6
>
> Different numbers of value pairs under each KEY.

JSON will allow you to nest dictionaries.

{
'KEY1': {
'a': v1
'c': v3
'd': v4
}
'KEY2': {
'a': v7
'b': v5
'd': v6
}
}

Personally I would not use .ini style these days as the format does not include type of the data.

Also I would not use the ast.literal_eval as it makes debugging errors in the data harder.

Barry

On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
>
>
>> On 6 Dec 2023, at 09:32, Chris Green via Python-list <python-list@python.org> wrote:
>>
>> My requirement is *slightly* more complex than just key value pairs,
>> it has one level of hierarchy, e.g.:-
>>
>> KEY1:
>> a: v1
>> c: v3
>> d: v4
>> KEY2:
>> a: v7
>> b: v5
>> d: v6
>>
>> Different numbers of value pairs under each KEY.
>
> JSON will allow you to nest dictionaries.
>
> {
> 'KEY1': {
> 'a': v1
> 'c': v3
> 'd': v4
> }
> 'KEY2': {
> 'a': v7
> 'b': v5
> 'd': v6
> }
> }
>
> Personally I would not use .ini style these days as the format does not include type of the data.

Neither does JSON. Besides, JSON is more complicated than necessary
here - in fact, your example isn't even legal JSON since lines are
missing commas.

Fun fact - for at least some, maybe most, JSON files, using eval() on
them is hugely faster than using Python's standard JSON library. I
learned this when I wanted to ingest a large browser bookmarks JSON
file. It wouldn't matter for a much smaller file, of course.

On 2023-12-06 12:23, Thomas Passin via Python-list wrote:
> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
>>
>>
>>> On 6 Dec 2023, at 09:32, Chris Green via Python-list <python-list@python.org> wrote:
>>>
>>> My requirement is *slightly* more complex than just key value pairs,
>>> it has one level of hierarchy, e.g.:-
>>>
>>> KEY1:
>>> a: v1
>>> c: v3
>>> d: v4
>>> KEY2:
>>> a: v7
>>> b: v5
>>> d: v6
>>>
>>> Different numbers of value pairs under each KEY.
>>
>> JSON will allow you to nest dictionaries.
>>
>> {
>> 'KEY1': {
>> 'a': v1
>> 'c': v3
>> 'd': v4
>> }
>> 'KEY2': {
>> 'a': v7
>> 'b': v5
>> 'd': v6
>> }
>> }
>>
>> Personally I would not use .ini style these days as the format does not include type of the data.
>
> Neither does JSON. Besides, JSON is more complicated than necessary
> here - in fact, your example isn't even legal JSON since lines are
> missing commas.
>
> Fun fact - for at least some, maybe most, JSON files, using eval() on
> them is hugely faster than using Python's standard JSON library. I
> learned this when I wanted to ingest a large browser bookmarks JSON
> file. It wouldn't matter for a much smaller file, of course.
>
It would be safer if you used literal_eval.

On 7/12/23 07:12, MRAB via Python-list wrote:
> On 2023-12-06 12:23, Thomas Passin via Python-list wrote:
>> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
>>>
>>>
>>>> On 6 Dec 2023, at 09:32, Chris Green via Python-list
>>>> <python-list@python.org> wrote:
>>>>
>>>> My requirement is *slightly* more complex than just key value pairs,
>>>> it has one level of hierarchy, e.g.:-
>>>>
>>>>     KEY1:
>>>>       a: v1
>>>>       c: v3
>>>>       d: v4
>>>>     KEY2:
>>>>       a: v7
>>>>       b: v5
>>>>       d: v6
>>>>
>>>> Different numbers of value pairs under each KEY.
>>>
>>> JSON will allow you to nest dictionaries.
>>>
>>> {
>>>      'KEY1': {
>>>          'a': v1
>>>          'c': v3
>>>          'd': v4
>>>      }
>>>      'KEY2': {
>>>           'a': v7
>>>           'b': v5
>>>           'd': v6
>>>      }
>>> }
>>>
>>> Personally I would not use .ini style these days as the format does
>>> not include type of the data.
>>
>> Neither does JSON.  Besides, JSON is more complicated than necessary
>> here - in fact, your example isn't even legal JSON since lines are
>> missing commas.
>>
>> Fun fact - for at least some, maybe most, JSON files, using eval() on
>> them is hugely faster than using Python's standard JSON library.  I
>> learned this when I wanted to ingest a large browser bookmarks JSON
>> file. It wouldn't matter for a much smaller file, of course.
>>
> It would be safer if you used literal_eval.

Ah, memories of Python2...

Does this little hack still work?

What about True/False cf true/false?

--
Regards,
=dn

On 2023-12-06 20:11, dn via Python-list wrote:
> On 7/12/23 07:12, MRAB via Python-list wrote:
>> On 2023-12-06 12:23, Thomas Passin via Python-list wrote:
>>> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
>>>>
>>>>
>>>>> On 6 Dec 2023, at 09:32, Chris Green via Python-list
>>>>> <python-list@python.org> wrote:
>>>>>
>>>>> My requirement is *slightly* more complex than just key value pairs,
>>>>> it has one level of hierarchy, e.g.:-
>>>>>
>>>>>     KEY1:
>>>>>       a: v1
>>>>>       c: v3
>>>>>       d: v4
>>>>>     KEY2:
>>>>>       a: v7
>>>>>       b: v5
>>>>>       d: v6
>>>>>
>>>>> Different numbers of value pairs under each KEY.
>>>>
>>>> JSON will allow you to nest dictionaries.
>>>>
>>>> {
>>>>      'KEY1': {
>>>>          'a': v1
>>>>          'c': v3
>>>>          'd': v4
>>>>      }
>>>>      'KEY2': {
>>>>           'a': v7
>>>>           'b': v5
>>>>           'd': v6
>>>>      }
>>>> }
>>>>
>>>> Personally I would not use .ini style these days as the format does
>>>> not include type of the data.
>>>
>>> Neither does JSON.  Besides, JSON is more complicated than necessary
>>> here - in fact, your example isn't even legal JSON since lines are
>>> missing commas.
>>>
>>> Fun fact - for at least some, maybe most, JSON files, using eval() on
>>> them is hugely faster than using Python's standard JSON library.  I
>>> learned this when I wanted to ingest a large browser bookmarks JSON
>>> file. It wouldn't matter for a much smaller file, of course.
>>>
>> It would be safer if you used literal_eval.
>
> Ah, memories of Python2...
>
> Does this little hack still work?
>
> What about True/False cf true/false?
>
Nope, nor None cf null.

If it's numbers, strings, lists and dicts, it works.

On 12/6/2023 1:12 PM, MRAB via Python-list wrote:
> On 2023-12-06 12:23, Thomas Passin via Python-list wrote:
>> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
>>>
>>>
>>>> On 6 Dec 2023, at 09:32, Chris Green via Python-list
>>>> <python-list@python.org> wrote:
>>>>
>>>> My requirement is *slightly* more complex than just key value pairs,
>>>> it has one level of hierarchy, e.g.:-
>>>>
>>>>     KEY1:
>>>>       a: v1
>>>>       c: v3
>>>>       d: v4
>>>>     KEY2:
>>>>       a: v7
>>>>       b: v5
>>>>       d: v6
>>>>
>>>> Different numbers of value pairs under each KEY.
>>>
>>> JSON will allow you to nest dictionaries.
>>>
>>> {
>>>      'KEY1': {
>>>          'a': v1
>>>          'c': v3
>>>          'd': v4
>>>      }
>>>      'KEY2': {
>>>           'a': v7
>>>           'b': v5
>>>           'd': v6
>>>      }
>>> }
>>>
>>> Personally I would not use .ini style these days as the format does
>>> not include type of the data.
>>
>> Neither does JSON.  Besides, JSON is more complicated than necessary
>> here - in fact, your example isn't even legal JSON since lines are
>> missing commas.
>>
>> Fun fact - for at least some, maybe most, JSON files, using eval() on
>> them is hugely faster than using Python's standard JSON library.  I
>> learned this when I wanted to ingest a large browser bookmarks JSON
>> file. It wouldn't matter for a much smaller file, of course.
>>
> It would be safer if you used literal_eval.

He's going to be writing his own calibration data files, though, so it
should be safe for his purposes.

On 2023-12-06 07:23:51 -0500, Thomas Passin via Python-list wrote:
> On 12/6/2023 6:35 AM, Barry Scott via Python-list wrote:
> > Personally I would not use .ini style these days as the format does not include type of the data.
>
> Neither does JSON.

Well, it distinguishes between some primitive types (string, number,
boolean, null) and provides two container types (dict/object,
list/array). As long as those types are sufficient, JSON includes them.
If you need anything else, you're on your own.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

If I enter a one-digit input or a three-digit number, the code works but if I enter a two digit number, the if statement fails and the else condition prevails.

tsReading = input(" Enter the " + Brand + " test strip reading: ")
if tsReading == "": tsReading = "0"
print(tsReading)
if ((tsReading < "400") and (tsReading >= "0")):
tsDose = GetDose(sReading)
print(tsReading + "-" + tsDose)
ValueFailed = False
else:
print("Enter valid sensor test strip Reading.")

I converted the variable to int along with the if statement comparison and it works as expected.
See if it fails for you...

Steve

Steve,

I would say converting to a number, as you eventually did, is the way to go.

When you compare character strings, it will not be in numeric order. Compare
"80" with "400" and since 8 is greater than 4, the comparison is over and
"80" is greater then "40" even though 80 is way less than 400.

The only way to get the kind of comparison you want would be to pad with
zeroes so all your "numbers" are the same length. In that case, "080" would
indeed test as less than "400" but rarely does that seem a good idea. If the
user can enter any text, they might enter ".01" or "hello" or al kinds of
nonsense.

If you converted to numbers and tested whether it failed, ...

-----Original Message-----
From: Python-list <python-list-bounces+avi.e.gross=gmail.com@python.org> On
Behalf Of Steve GS via Python-list
Sent: Saturday, December 9, 2023 9:42 PM
To: python-list@python.org
Subject: A problem with str VS int.

If I enter a one-digit input or a three-digit number, the code works but if
I enter a two digit number, the if statement fails and the else condition
prevails.

tsReading = input(" Enter the " + Brand + " test strip reading: ")
if tsReading == "": tsReading = "0"
print(tsReading)
if ((tsReading < "400") and (tsReading >= "0")):
tsDose = GetDose(sReading)
print(tsReading + "-" + tsDose)
ValueFailed = False
else:
print("Enter valid sensor test strip Reading.")

I converted the variable to int along with the if statement comparison and
it works as expected.
See if it fails for you...

Steve

--
https://mail.python.org/mailman/listinfo/python-list

On 12/9/2023 9:42 PM, Steve GS via Python-list wrote:
> If I enter a one-digit input or a three-digit number, the code works but if I enter a two digit number, the if statement fails and the else condition prevails.
>
> tsReading = input(" Enter the " + Brand + " test strip reading: ")
> if tsReading == "": tsReading = "0"
> print(tsReading)
> if ((tsReading < "400") and (tsReading >= "0")):
> tsDose = GetDose(sReading)
> print(tsReading + "-" + tsDose)
> ValueFailed = False
> else:
> print("Enter valid sensor test strip Reading.")
>
> I converted the variable to int along with the if statement comparison and it works as expected.
> See if it fails for you...

I don't have to try it to know it will fail. You think you are
comparing numbers but you are comparing strings instead, which won't
work as you expect.

You would do better to convert the inputs and limits to numbers, as well
as the GetDose() function. In a more realistic version, you would also
have to make sure the user input is legal, either an int or a float,
whichever you want to work with.

And along those lines (a more realistic version), it would be preferable
to change the limits to be named constants, which will make the code
easier to understand and change when it's revisited later. Something
like this:

UPPER = 400
LOWER = 0
# ...
if LOWER < value < UPPER:
# .....