It can be very useful to pipe commands into a shell such as with

ls sample* | sed 's/^/rm -v /' | sh

That works, finding all the files which match the pattern and converting
them into commands which it then executes. (In reality I have a piece of
code which generates the file names but ls is a simpler example which
shows the germane part of the mechanism.)

However, if I add the -i option to rm as in

ls sample* | sed 's/^/rm -iv /' | sh

then it doesn't work, printing all the questions on one line and not
waiting for a response:

rm: remove regular empty file 'sample3'? rm: remove regular empty file
'sample7'? $

I presumed that's because sh has only stdin as its input stream.
However, consider this:

sh <(ls sample* | sed 's/^/rm -iv /')

That does work:

$ sh <(ls sample* | sed 's/^/rm -iv /')
rm: remove regular file 'sample3'? n
rm: remove regular file 'sample7'? n
$

Anyone know why the latter example is different? And is there a way to
change the command to put sh at the end rather than the beginning?

--
James Harris

* James Harris <james.harris.1@gmail.com>
| However, if I add the -i option to rm as in
>
| ls sample* | sed 's/^/rm -iv /' | sh
>
| then it doesn't work, printing all the questions on one line and not
| waiting for a response:
>
| rm: remove regular empty file 'sample3'? rm: remove regular empty file
| 'sample7'? $
>
| I presumed that's because sh has only stdin as its input
| stream.

Correct.

| However, consider this:
>
| sh <(ls sample* | sed 's/^/rm -iv /')
>
| That does work:
>
| $ sh <(ls sample* | sed 's/^/rm -iv /')
| rm: remove regular file 'sample3'? n
| rm: remove regular file 'sample7'? n
| $
>
| Anyone know why the latter example is different? And is there a way to
| change the command to put sh at the end rather than the beginning?

Because <(cmd) produces a (temporary) file name which is added to the
cmd as first argument (cf. "man bash"). Therefore the command executed
is actually

sh filename

where stdin is still available.

Process Substitution
Process substitution allows a process's input or output to be referred
to using a filename. It takes the form of <(list) or >(list). The
process list is run asynchronously, and its input or output appears as
a filename. This filename is passed as an argument to the current com-
mand as the result of the expansion. [...] If the <(list) form
is used, the file passed as an argument should be read to obtain the
output of list.

cf.
$ echo <(ls)
/dev/fd/63

HTH
R'

On 2024-02-26 3:50 a.m., James Harris wrote:
> It can be very useful to pipe commands into a shell such as with
>
>   ls sample* | sed 's/^/rm -v /' | sh

Can I ask why you pipe to sed and a shell instead of simply using xargs:

ls sample* | xargs rm -vi

That seems to be much a simpler solution.

John-Paul Stewart <jpstewart@personalprojects.net> wrote:
> On 2024-02-26 3:50 a.m., James Harris wrote:
>> It can be very useful to pipe commands into a shell such as with
>>
>>   ls sample* | sed 's/^/rm -v /' | sh
>
> Can I ask why you pipe to sed and a shell instead of simply using xargs:
>
> ls sample* | xargs rm -vi
>
> That seems to be much a simpler solution.

Even better, this version which won't choke on spaces or other odd
characters in filenames:

find -name sample\* -maxdepth 1 -print0 | xargs -0 rm -vi

Rich <rich@example.invalid> writes:
> John-Paul Stewart <jpstewart@personalprojects.net> wrote:
>> On 2024-02-26 3:50 a.m., James Harris wrote:
>>> It can be very useful to pipe commands into a shell such as with
>>>
>>>   ls sample* | sed 's/^/rm -v /' | sh
>>
>> Can I ask why you pipe to sed and a shell instead of simply using xargs:
>>
>> ls sample* | xargs rm -vi
>>
>> That seems to be much a simpler solution.
>
> Even better, this version which won't choke on spaces or other odd
> characters in filenames:
>
> find -name sample\* -maxdepth 1 -print0 | xargs -0 rm -vi

That handles directories better too. But it doesn’t work, due to the
stdin issue that the OP was originally asking about.

The working answer is much simpler:

rm -vi sample*

The ‘ls’ versions will also behave quite badly in the face of
directories matching the pattern.

--
https://www.greenend.org.uk/rjk/

On 2024-02-27 3:46 a.m., Richard Kettlewell wrote:
> Rich <rich@example.invalid> writes:
>> John-Paul Stewart <jpstewart@personalprojects.net> wrote:
>>> On 2024-02-26 3:50 a.m., James Harris wrote:
>>>> It can be very useful to pipe commands into a shell such as with
>>>>
>>>>   ls sample* | sed 's/^/rm -v /' | sh
>>>
>>> Can I ask why you pipe to sed and a shell instead of simply using xargs:
>>>
>>> ls sample* | xargs rm -vi
>>>
>>> That seems to be much a simpler solution.
>>
>> Even better, this version which won't choke on spaces or other odd
>> characters in filenames:
>>
>> find -name sample\* -maxdepth 1 -print0 | xargs -0 rm -vi
>
> That handles directories better too. But it doesn’t work, due to the
> stdin issue that the OP was originally asking about.
>
> The working answer is much simpler:
>
> rm -vi sample*
>
> The ‘ls’ versions will also behave quite badly in the face of
> directories matching the pattern.

Well, the OP did say that 'ls' was just an example. In reality it is
some other code that generates the list of names.

On 26/02/2024 13:29, Ralf Fassel wrote:
> * James Harris <james.harris.1@gmail.com>

....

> | Anyone know why the latter example is different? And is there a way to
> | change the command to put sh at the end rather than the beginning?
>
> Because <(cmd) produces a (temporary) file name which is added to the
> cmd as first argument (cf. "man bash"). Therefore the command executed
> is actually
>
> sh filename
>
> where stdin is still available.
>
> Process Substitution
> Process substitution allows a process's input or output to be referred
> to using a filename. It takes the form of <(list) or >(list). The
> process list is run asynchronously, and its input or output appears as
> a filename. This filename is passed as an argument to the current com-
> mand as the result of the expansion. [...] If the <(list) form
> is used, the file passed as an argument should be read to obtain the
> output of list.

Thanks. That's very clear.

--
James Harris

On 28/02/2024 01:07, John-Paul Stewart wrote:
> On 2024-02-27 3:46 a.m., Richard Kettlewell wrote:
>> Rich <rich@example.invalid> writes:
>>> John-Paul Stewart <jpstewart@personalprojects.net> wrote:
>>>> On 2024-02-26 3:50 a.m., James Harris wrote:
>>>>> It can be very useful to pipe commands into a shell such as with
>>>>>
>>>>>   ls sample* | sed 's/^/rm -v /' | sh
>>>>
>>>> Can I ask why you pipe to sed and a shell instead of simply using xargs:
>>>>
>>>> ls sample* | xargs rm -vi
>>>>
>>>> That seems to be much a simpler solution.
>>>
>>> Even better, this version which won't choke on spaces or other odd
>>> characters in filenames:
>>>
>>> find -name sample\* -maxdepth 1 -print0 | xargs -0 rm -vi
>>
>> That handles directories better too. But it doesn’t work, due to the
>> stdin issue that the OP was originally asking about.
>>
>> The working answer is much simpler:
>>
>> rm -vi sample*
>>
>> The ‘ls’ versions will also behave quite badly in the face of
>> directories matching the pattern.
>
> Well, the OP did say that 'ls' was just an example. In reality it is
> some other code that generates the list of names.

That's right. What this is for is code to list files in a folder which
are duplicates of those in another folder (same name, same relative
place in the folder hierarchy, etc). For example, say there are two
folders, c and d, and one is potentially a copy of the other. The command

$ ./lsdup.py -r c d

lists files in d (and subdirectories due to the -r recurse option) which
are duplicates of those in c.

In answer to your other point about using xargs, I would use it if it
would do what's required, and do so consistently, but I am not sure
whether I can trust it or not. Here's a full example.

$ ./lsdup.py c d -r | sed 's/^/rm -v /'

which generates commands such as

rm -v 'd/contentsame.txt'
rm -v 'd/hardlink.txt'
rm -v 'd/subdir/filesame.txt'

Once the commands have been checked, if required, I pipe them into sh to
actually delete the files. (A separate command, lsempty.py, is is used
to delete the resultant empty folders.)

In answer to the point about filenames with spaces and odd characters, I
currently output names in single quotes, as above.

--
James Harris

On Thu, 7 Mar 2024 16:24:26 +0000, James Harris wrote:

> In answer to your other point about using xargs, I would use it if it
> would do what's required, and do so consistently, but I am not sure
> whether I can trust it or not.

You mean, accidentally operating on the wrong files?

That’s quite easy to test: instead of

xargs «cmd...»

do

xargs echo «cmd...»

and check the output to see that it shows what you expect. Once you are
satisfied, you can run it again without the “echo”.

James Harris <james.harris.1@gmail.com> wrote:
> That's right. What this is for is code to list files in a folder
> which are duplicates of those in another folder (same name, same
> relative place in the folder hierarchy, etc). For example, say there
> are two folders, c and d, and one is potentially a copy of the other.
> The command
>
> $ ./lsdup.py -r c d
>
> lists files in d (and subdirectories due to the -r recurse option) which
> are duplicates of those in c.
>
> In answer to your other point about using xargs, I would use it if it
> would do what's required, and do so consistently,

It will, provided you output file names with ASCII null terminators
instead of ASCII newlines, and use the -0 option to tell xargs the
filenames are null separated.

> but I am not sure whether I can trust it or not.

You can. Provided you feed it ASCII null terminated filenames, it will
work properly, no matter what other weird characters might be in the
filenames.

> Here's a full example.
>
> $ ./lsdup.py c d -r | sed 's/^/rm -v /'
>
> which generates commands such as
>
> rm -v 'd/contentsame.txt'
> rm -v 'd/hardlink.txt'
> rm -v 'd/subdir/filesame.txt'

Provided you modified lsdup.py to output null terminated filenames, you
could do:

$ ./lsdup.py c d -r | xargs -0 rm -v

And the chosen files would be removed, no matter what odd characters or
spaces they might contain. Note that if you want to 'inspect' before
you delete, then you might want to add a "-0" option to lsdup.py to
toggle between ASCII newline and ASCII null terminators.

> Once the commands have been checked, if required, I pipe them into sh to
> actually delete the files.

xargs, with -0, into "rm" or "rm -v" will be much more efficient (i.e.,
faster) because xargs will call rm with the maximum number of files it
can pass per invocation, avoiding calling rm once per file, and
avoiding shell parsing to then fork and call rm, once per each file.

> (A separate command, lsempty.py, is is used to delete the resultant
> empty folders.)

A separate python command is completely unnecessary. Removing empty
leaf directories is already easy using the tools provided by the
system.:

$ find c d -type d -empty -print0 | xargs -0 rmdir

If you also want to remove parent empty directories should all their
children go away, change to:

$ find c d -type d -empty -print0 | xargs -0 rmdir -p

> In answer to the point about filenames with spaces and odd characters, I
> currently output names in single quotes, as above. This will
> complain until the parent is empty, but the complaints can be
> ignored.

Single quote is also a possible filename character, so if, by chance,
you end up with a file containing one ' somewhere your wrapping in
single quotes will result in a fail at that point.

James Harris <james.harri...@gmail.com> [JH]:
JH> What this is for is code to list files in a folder
JH> which are duplicates of those in another folder
JH> $ ./lsdup.py -r c d

No need to reinvent the wheel:

$ fdupes --recurse c d

fdupes(1) will find duplicate files even with different filenames.

Adding the "--delete" option will prompt user for files to preserve,
deleting all others.

Fritz Wuehler <fritz@spamexpire-202403.rodent.frell.theremailer.net> wrote at 04:56 this Friday (GMT):
> James Harris <james.harri...@gmail.com> [JH]:
> JH> What this is for is code to list files in a folder
> JH> which are duplicates of those in another folder
> JH> $ ./lsdup.py -r c d
>
> No need to reinvent the wheel:
>
> $ fdupes --recurse c d
>
> fdupes(1) will find duplicate files even with different filenames.
>
> Adding the "--delete" option will prompt user for files to preserve,
> deleting all others.

I never knew about this program, thanks!
--
user <candycane> is generated from /dev/urandom

On 08/03/2024 04:56, Fritz Wuehler wrote:
> James Harris <james.harri...@gmail.com> [JH]:
> JH> What this is for is code to list files in a folder
> JH> which are duplicates of those in another folder
> JH> $ ./lsdup.py -r c d
>
> No need to reinvent the wheel:
>
> $ fdupes --recurse c d
>
> fdupes(1) will find duplicate files even with different filenames.

Different use case. fdupes reports duplicates anywhere, even in the same
'master' folder. By contrast,

$ ./lsdup.py -h
Usage:

./lsdup.py [opt...] master candidate...

Lists files etc in the candidate folders which are duplicates of
those in the master folder.

By default, this program will report those files which match on all of
eight attributes: name, size, UID, GID, disposition, permissions,
modification time, and content but can be told to ignore certain
mismatches (and thus list those files as duplicates).

For example, you may wish to identify a file as a duplicate even if
it has a different UID and GID as long as all other attributes match.

A file's name and its position in the folder hierarchy must always
match, and both must be regular files (not named pipes, for example).

etc.

>
> Adding the "--delete" option will prompt user for files to preserve,
> deleting all others.

I have tried it but it required too much manual intervention.

--
James Harris

On 08/03/2024 03:39, Rich wrote:
> James Harris <james.harris.1@gmail.com> wrote:
>> That's right. What this is for is code to list files in a folder
>> which are duplicates of those in another folder (same name, same
>> relative place in the folder hierarchy, etc). For example, say there
>> are two folders, c and d, and one is potentially a copy of the other.
>> The command
>>
>> $ ./lsdup.py -r c d
>>
>> lists files in d (and subdirectories due to the -r recurse option) which
>> are duplicates of those in c.
>>
>> In answer to your other point about using xargs, I would use it if it
>> would do what's required, and do so consistently,
>
> It will, provided you output file names with ASCII null terminators
> instead of ASCII newlines, and use the -0 option to tell xargs the
> filenames are null separated.
>
>> but I am not sure whether I can trust it or not.
>
> You can. Provided you feed it ASCII null terminated filenames, it will
> work properly, no matter what other weird characters might be in the
> filenames.

You are right: unusual characters in file names is the kind of issue I
am wary about with xargs. Running a command to do automatic deletion
requires a lot of trust in the command's operation.

I /have/ been thinking about adding a -0 option but it has issues such as:

(1) With -0 one cannot easily postprocess the list of file names, if
required.

(2) With xargs interactivity (e.g. the -i in rm -i) is lost.

Yes, there are ways round such issues but they may require altering a
command line /after/ it has been found to be correct. That was the issue
which led to this thread. If I I generate a list of delete commands with
a series of commands such as

A | B

then once I am happy with them I would prefer simply to append | sh as in

A | B | sh

rather than changing the form to

sh <(A | B)

This is not so much about convenience as about making sure less can go
wrong - always a good idea when deleting files by means of a command.

IMO better than a -print- or -0 option would be a function which would
render file names exactly as the current shell would - same escape
sequences, etc.

....

>> (A separate command, lsempty.py, is is used to delete the resultant
>> empty folders.)
>
> A separate python command is completely unnecessary. Removing empty
> leaf directories is already easy using the tools provided by the
> system.:
>
> $ find c d -type d -empty -print0 | xargs -0 rmdir
>
> If you also want to remove parent empty directories should all their
> children go away, change to:
>
> $ find c d -type d -empty -print0 | xargs -0 rmdir -p

Thanks. That looks as though it would work, albeit that it would print a
few spurious error messages which add to the work of the person running
the code. For example,

$ find f -type d -empty -print0 | xargs -0 rmdir -pv
rmdir: removing directory, 'f/1/2/3'
rmdir: removing directory, 'f/1/2'
rmdir: removing directory, 'f/1'
rmdir: failed to remove directory 'f/1': Directory not empty
rmdir: removing directory, 'f/b/c/d'
rmdir: removing directory, 'f/b/c'
rmdir: removing directory, 'f/b'
rmdir: failed to remove directory 'f/b': Directory not empty
rmdir: removing directory, 'f/b/g/h'
rmdir: removing directory, 'f/b/g'
rmdir: removing directory, 'f/b'
rmdir: failed to remove directory 'f/b': Directory not empty
$

Note, in particular, the repetitions of f/b.

By contrast, my version of the above is clearer:

$ ./lsempty.py f | sed 's/^/rmdir -v /' | sh
rmdir: removing directory, 'f/1/2/3'
rmdir: removing directory, 'f/1/2'
rmdir: removing directory, 'f/b/c/d'
rmdir: removing directory, 'f/b/c'
rmdir: removing directory, 'f/b/g/h'
rmdir: removing directory, 'f/b/g'
$

>
>> In answer to the point about filenames with spaces and odd characters, I
>> currently output names in single quotes, as above. This will
>> complain until the parent is empty, but the complaints can be
>> ignored.
>
> Single quote is also a possible filename character, so if, by chance,
> you end up with a file containing one ' somewhere your wrapping in
> single quotes will result in a fail at that point.

I deal with that (at present) by juxtaposing single-quoted strings. For
example, if there is a file called

won't scan.txt

with an apostrophe and a space then I get the following results. First,
the command:

$ ./lsdup.py c d -r | grep won | sed 's/^/rm -v /'
rm -v 'd/won'\''t scan.txt'

Then, piping that command to the shell works properly in:

$ ./lsdup.py c d -r | grep won | sed 's/^/rm -v /' | sh
removed "d/won't scan.txt"

IOW the file name becomes the concatenation of

'won'
\'
't scan.txt'

That said, I am not sure that this will work in all cases and expect
that a function to render a file name to match the current shell would
be more dependable. Pity that shells don't provide such a function ...
AFAIK.

--
James Harris

James Harris <james.harris.1@gmail.com> wrote:
> On 08/03/2024 03:39, Rich wrote:
>> James Harris <james.harris.1@gmail.com> wrote:
>>> That's right. What this is for is code to list files in a folder
>>> which are duplicates of those in another folder (same name, same
>>> relative place in the folder hierarchy, etc). For example, say there
>>> are two folders, c and d, and one is potentially a copy of the other.
>>> The command
>>>
>>> $ ./lsdup.py -r c d
>>>
>>> lists files in d (and subdirectories due to the -r recurse option) which
>>> are duplicates of those in c.
>>>
>>> In answer to your other point about using xargs, I would use it if it
>>> would do what's required, and do so consistently,
>>
>> It will, provided you output file names with ASCII null terminators
>> instead of ASCII newlines, and use the -0 option to tell xargs the
>> filenames are null separated.
>>
>>> but I am not sure whether I can trust it or not.
>>
>> You can. Provided you feed it ASCII null terminated filenames, it will
>> work properly, no matter what other weird characters might be in the
>> filenames.
>
> You are right: unusual characters in file names is the kind of issue I
> am wary about with xargs.

And this is exactly why xargs was modified to consume null terminated
names, so that no otherwise legal filename character would create a
problem. And ASCII newline is also a legal filename character.

> Running a command to do automatic deletion requires a lot of trust in
> the command's operation.

Agreed.

> I /have/ been thinking about adding a -0 option but it has issues such as:
>
> (1) With -0 one cannot easily postprocess the list of file names, if
> required.

That depends upon what you mean by 'postprocessing'.

GNU sort has -z, --zero-terminated options to inform it to process
"lines" with ASCII null terminators.

GNU grep has -z, --null-data options to also inform it to process
"lines" that are ASCII null terminated.

If you want to "preview" in something like less, you can add
"tr '\0' '\n'" to the pipeline before less to conver the nulls into
newlines so that any filename other than one with an actual newline in
it would be one line in less.

> (2) With xargs interactivity (e.g. the -i in rm -i) is lost.

This is true, so if you value using -i to acknowledge each file
individually then yes, using xargs to launch rm is not in the cards.

Although a small workaround would be to use xargs (GNU xargs at least)
option -p, --interactive combined with -n 1 to run one rm per filename
and have xargs prompt (instead of rm) before each. This loses the
efficiency gains of one rm and a maximum number of filenames, but it
does allow 'interactivity' again.

> Yes, there are ways round such issues but they may require altering a
> command line /after/ it has been found to be correct. That was the issue
> which led to this thread. If I I generate a list of delete commands with
> a series of commands such as
>
> A | B
>
> then once I am happy with them I would prefer simply to append | sh as in
>
> A | B | sh
>
> rather than changing the form to
>
> sh <(A | B)

With xargs and nulls, the change becomes:

A | B | tr '\0' '\n' | less
to
A | B | xargs -0 rm
or
A | B | xargs -0 -n 1 -p rm (for interactive deletes)

> This is not so much about convenience as about making sure less can go
> wrong - always a good idea when deleting files by means of a command.

Yes, if you have no backup with which to recover then agreed, verifying
the deletion before deleting is important.

>>> (A separate command, lsempty.py, is is used to delete the resultant
>>> empty folders.)
>>
>> A separate python command is completely unnecessary. Removing empty
>> leaf directories is already easy using the tools provided by the
>> system.:
>>
>> $ find c d -type d -empty -print0 | xargs -0 rmdir
>>
>> If you also want to remove parent empty directories should all their
>> children go away, change to:
>>
>> $ find c d -type d -empty -print0 | xargs -0 rmdir -p
>
> Thanks. That looks as though it would work, albeit that it would print a
> few spurious error messages which add to the work of the person running
> the code. For example,

If you dislike the spurious erros, then add the --ignore-fail-on-non-empty
option, which suppresses the errors for 'not empty' as rmdir walks back
up the tree.

>>> In answer to the point about filenames with spaces and odd
>>> characters, I currently output names in single quotes, as above.
>>> This will complain until the parent is empty, but the complaints
>>> can be ignored.
>>
>> Single quote is also a possible filename character, so if, by
>> chance, you end up with a file containing one ' somewhere your
>> wrapping in single quotes will result in a fail at that point.
>
> I deal with that (at present) by juxtaposing single-quoted strings.
> For example, if there is a file called
>
> won't scan.txt
>
> with an apostrophe and a space then I get the following results. First,
> the command:
>
> $ ./lsdup.py c d -r | grep won | sed 's/^/rm -v /'
> rm -v 'd/won'\''t scan.txt'

Ok, you had not yet revealed you were doing so, and we can't read your
mind remotely over Usenet to know you are handling single quotes
properly (at least for one single quote in a filename).

> That said, I am not sure that this will work in all cases and expect

Provided you replace every input instance of ' with '\'' it would work
(at least for ').

The minefield aspect into which you are traversing is that each shell
has a different set of meta-characters that are 'special' to it and so
the proper "escaping" is dependent upon /which/ shell the output is
being piped into.

And by piping into the shell, if your escaping is not 100% perfect, you
open yourself up to someone crafting a filename that results in your
shell doing one of:

"rm -fr /"

or

"rm -fr ~"

depending upon whether you run the deletion as root or a non-root user.