Hi,

The following is a little bit involved, but I hope can make the problem clear.

Using poetry I have written a dummy application which just uses to typer
to illustrate a possible interface design. The directory structure is a
follows:

$ tree -P *.py
.
|-- dist
|-- stoat
| |-- hpc
| | |-- database.py
| | |-- group.py
| | |-- __init__.py
| | |-- main.py
| | |-- owner.py
| | `-- user.py
| |-- __init__.py
| |-- main.py
| `-- storage
| |-- database.py
| |-- group.py
| |-- __init__.py
| |-- main.py
| |-- owner.py
| |-- share.py
| `-- user.py
`-- tests
|-- __init__.py
`-- test_stoat.py

With in the poetry shell I can run the application successfully:

$ python stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

I then install this in a non-standard path (because the OS Python3 is
3.6.8) and can run the installed version successfully:

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help
Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...

manage HPC users

Options:
--help Show this message and exit.

Commands:
add add a user
remove remove a user

However, poetry creates a script 'stoat' from the entry

[tool.poetry.scripts]
stoat = "stoat.main:main"

in pyproject.toml, which looks like

#!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9
# -*- coding: utf-8 -*-
import re
import sys
from stoat.main import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())

If I run that I get

$ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help
Traceback (most recent call last):
File "/trinity/shared/zedat/bin/stoat", line 5, in <module>
from stoat.main import main
File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in <module>
import hpc.main
ModuleNotFoundError: No module named 'hpc'

Why is the module 'hpc' not found by the poetry script?

Cheers,

Loris

--
This signature is currently under construction.

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:

> Hi,
>
> The following is a little bit involved, but I hope can make the problem clear.
>
> Using poetry I have written a dummy application which just uses to typer
> to illustrate a possible interface design. The directory structure is a
> follows:
>
> $ tree -P *.py
> .
> |-- dist
> |-- stoat
> | |-- hpc
> | | |-- database.py
> | | |-- group.py
> | | |-- __init__.py
> | | |-- main.py
> | | |-- owner.py
> | | `-- user.py
> | |-- __init__.py
> | |-- main.py
> | `-- storage
> | |-- database.py
> | |-- group.py
> | |-- __init__.py
> | |-- main.py
> | |-- owner.py
> | |-- share.py
> | `-- user.py
> `-- tests
> |-- __init__.py
> `-- test_stoat.py
>
> With in the poetry shell I can run the application successfully:
>
> $ python stoat/main.py hpc user --help
> Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...
>
> manage HPC users
>
> Options:
> --help Show this message and exit.
>
> Commands:
> add add a user
> remove remove a user
>
> I then install this in a non-standard path (because the OS Python3 is
> 3.6.8) and can run the installed version successfully:
>
> $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages python /trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py hpc user --help
> Usage: main.py hpc user [OPTIONS] COMMAND [ARGS]...
>
> manage HPC users
>
> Options:
> --help Show this message and exit.
>
> Commands:
> add add a user
> remove remove a user
>
> However, poetry creates a script 'stoat' from the entry
>
> [tool.poetry.scripts]
> stoat = "stoat.main:main"
>
> in pyproject.toml, which looks like
>
> #!/trinity/shared/easybuild/software/Python/3.9.6-GCCcore-11.2.0/bin/python3.9
> # -*- coding: utf-8 -*-
> import re
> import sys
> from stoat.main import main
> if __name__ == '__main__':
> sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
> sys.exit(main())
>
> If I run that I get
>
> $ PYTHONPATH=/trinity/shared/zedat/lib/python3.9/site-packages stoat hpc user --help
> Traceback (most recent call last):
> File "/trinity/shared/zedat/bin/stoat", line 5, in <module>
> from stoat.main import main
> File "/trinity/shared/zedat/lib/python3.9/site-packages/stoat/main.py", line 3, in <module>
> import hpc.main
> ModuleNotFoundError: No module named 'hpc'
>
> Why is the module 'hpc' not found by the poetry script?

Never mind, I worked it out. I had to replace

import hpc.main

with

import stoat.hpc.main

However, this raises the question of why it worked in the first place
in the poetry shell.

Cheers,

Loris

--
This signature is currently under construction.

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>However, this raises the question of why it worked in the first place
>in the poetry shell.

It might have had a different or extended sys.path.

ram@zedat.fu-berlin.de (Stefan Ram) writes:

> "Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>>However, this raises the question of why it worked in the first place
>>in the poetry shell.
>
> It might have had a different or extended sys.path.

In the poetry shell sys.path has this additional path

/home/loris/gitlab/stoat

The module not being found was

/home/gitlab/stoat/stoat/hpc/main.py

But if I run

[~/gitlab/stoat] $ python stoat/main.py hpc user --help

wouldn't

import hpc.main

still fail? Or is it because I am calling

stoat/main.py

and so Python looks for 'hpc' relative to the second 'stoat' directory?

Confused,

loris

--
This signature is currently under construction.

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>Why is the module 'hpc' not found by the poetry script?

I have tried to execute the following sequence of shell
commands to understand your problem. Here they all worked
without error messages. Warning: Some of these commands might
alter your directories or data, so only execute them if you
are aware of their meaning/consequences and agree with them!

mkdir stoat
mkdir stoat/hpc
echo import hpc.main >stoat/main.py
echo >stoat/hpc/main.py
python3 stoat/main.py
cd stoat
python3 main.py

Hi Stefan,

ram@zedat.fu-berlin.de (Stefan Ram) writes:

> "Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>>Why is the module 'hpc' not found by the poetry script?
>
> I have tried to execute the following sequence of shell
> commands to understand your problem. Here they all worked
> without error messages. Warning: Some of these commands might
> alter your directories or data, so only execute them if you
> are aware of their meaning/consequences and agree with them!

I do know what the commands do. However, just to be on the safe side,
but mainly in honour of a classic dad-joke, I changed 'stoat' to
'weasel'.

> mkdir stoat
> mkdir stoat/hpc
> echo import hpc.main >stoat/main.py
> echo >stoat/hpc/main.py
> python3 stoat/main.py
> cd stoat
> python3 main.py

I guess with the production version, the package

stoat

is visible in sys.path and thus the subpackages have to be referred to
via the main package, e.g.

import stoat.hpc.main

However, in the development environment, if I run

python stoat/main.py hpc user --help

then is

stoat/hpc/main.py

being found via

import hpc.main

because Python looks in

stoat

as the parent directory of

stoat/main.py

rather than the current working directory? That doesn't seem likely to
me, but I am already confused.

Cheers,

Loris

--
This signature is currently under construction.

"Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>However, in the development environment, if I run
> python stoat/main.py hpc user --help
>then is
> stoat/hpc/main.py
>being found via
> import hpc.main
>because Python looks in
> stoat
>as the parent directory of
> stoat/main.py
>rather than the current working directory?

When you run "stoat/main.py", the directory of that file
"main.py" is automatically added at the front of sys.path.
(This also would happen after "cd stoat; python main.py".)

Then, when "import hpc.main" is being executed, the system
will search for "hpc/main.py" in every entry of sys.path
and will use the first entry wherein it finds "hpc/main.py".
So it will use the directory of the file "stoat/main.py",
i.e., the directory "stoat". It finds "stoat/hpc/main.py".

You can call "python" with "-v", and it will show some lines
with information about the imports executed, including the
directories used (albeit hidden in a lot of other lines).

ram@zedat.fu-berlin.de (Stefan Ram) writes:

> "Loris Bennett" <loris.bennett@fu-berlin.de> writes:
>>However, in the development environment, if I run
>> python stoat/main.py hpc user --help
>>then is
>> stoat/hpc/main.py
>>being found via
>> import hpc.main
>>because Python looks in
>> stoat
>>as the parent directory of
>> stoat/main.py
>>rather than the current working directory?
>
> When you run "stoat/main.py", the directory of that file
> "main.py" is automatically added at the front of sys.path.
> (This also would happen after "cd stoat; python main.py".)

OK, that explains it. I initially found that a bit odd, but thinking
about it I see that "the directory containing the file being run" is a
more sensible reference point than the current working directory, which
is totally arbitrary.

> Then, when "import hpc.main" is being executed, the system
> will search for "hpc/main.py" in every entry of sys.path
> and will use the first entry wherein it finds "hpc/main.py".
> So it will use the directory of the file "stoat/main.py",
> i.e., the directory "stoat". It finds "stoat/hpc/main.py".
>
> You can call "python" with "-v", and it will show some lines
> with information about the imports executed, including the
> directories used (albeit hidden in a lot of other lines).

That's useful, although in the production case I would have to tweak the
script generated by poetry.

Cheers,

Loris

--
This signature is currently under construction.