This might be a Linux question but I'm not sure.
When I run a Fortran program that prints a lot of stuff
as it runs, directly, it does that. But when I run it
offline, with the output going into a file, it only prints
right near the end of execution. So that file is empty for
the whole time.

I run it with a script with this order:

../<script> >& <output file name> &

Is there a way to run it so that it prints as it runs?
--
Dieter Britz

Dieter Britz <dieterhansbritz@gmail.com> schrieb:
> This might be a Linux question but I'm not sure.
> When I run a Fortran program that prints a lot of stuff
> as it runs, directly, it does that. But when I run it
> offline, with the output going into a file, it only prints
> right near the end of execution. So that file is empty for
> the whole time.

Often, terminal output is unbuffered, file output is.

Your compiler probably supports the FLUSH statement, as in

program main
use iso_fortran_env, only : output_unit
implicit none
write (*,'(A)') "Hello"
flush output_unit
! Do something that takes a long time
write (*,'(A)') "Goodbye"
end program main

Try that.

On Thursday, June 3, 2021 at 3:24:03 PM UTC+2, Thomas Koenig wrote:
> Dieter Britz schrieb:
> > This might be a Linux question but I'm not sure.
> > When I run a Fortran program that prints a lot of stuff
> > as it runs, directly, it does that. But when I run it
> > offline, with the output going into a file, it only prints
> > right near the end of execution. So that file is empty for
> > the whole time.
> Often, terminal output is unbuffered, file output is.
>
> Your compiler probably supports the FLUSH statement, as in
>
> program main
> use iso_fortran_env, only : output_unit
> implicit none
> write (*,'(A)') "Hello"
> flush output_unit
> ! Do something that takes a long time
> write (*,'(A)') "Goodbye"
> end program main
>
> Try that.
You may also get the effect you need using the tee program:

myprog | tee report.out

as tee reads standard input and writes it to standard output or a file. It may set things as if it was a terminal (one of those magic things that can happen). Note that I have not tried this myself, just a half-informed guess.

Regards,

Arjen

On Thursday, June 3, 2021 at 7:47:55 AM UTC-7, arjen.m...@gmail.com wrote:
> On Thursday, June 3, 2021 at 3:24:03 PM UTC+2, Thomas Koenig wrote:

(snip)

> > Often, terminal output is unbuffered, file output is.

> > Your compiler probably supports the FLUSH statement, as in

(snip)

> You may also get the effect you need using the tee program:

> myprog | tee report.out

I used to use tee a lot, less often now.
But I do remember that the output to it is buffered, so it doesn't help.
It might be that you can turn off buffering in the program that
writes to the pipe.

Now I more often use tail -f to follow the output written to the file, but again
(as the OP notes) it is buffered, so you won't see it for a while.

On Friday, June 4, 2021 at 3:43:50 AM UTC+2, gah4 wrote:
> On Thursday, June 3, 2021 at 7:47:55 AM UTC-7, arjen wrote:
> > On Thursday, June 3, 2021 at 3:24:03 PM UTC+2, Thomas Koenig wrote:
> (snip)
> > > Often, terminal output is unbuffered, file output is.
>
> > > Your compiler probably supports the FLUSH statement, as in
> (snip)
> > You may also get the effect you need using the tee program:
>
> > myprog | tee report.out
> I used to use tee a lot, less often now.
> But I do remember that the output to it is buffered, so it doesn't help.
> It might be that you can turn off buffering in the program that
> writes to the pipe.
>
> Now I more often use tail -f to follow the output written to the file, but again
> (as the OP notes) it is buffered, so you won't see it for a while.
Hm, so it is not as smart/magic as I thought. Well, in that case the simplest solution would indeed be to use flush explicitly.

Should that be impractical or impossible, I can think of one other possibility: use the "expect" tool to fool the program into thinking it is outputting to a terminal.

Regards,

Arjen

Arjen Markus <arjen.markus895@gmail.com> schrieb:
> On Friday, June 4, 2021 at 3:43:50 AM UTC+2, gah4 wrote:
>> On Thursday, June 3, 2021 at 7:47:55 AM UTC-7, arjen wrote:
>> > On Thursday, June 3, 2021 at 3:24:03 PM UTC+2, Thomas Koenig wrote:
>> (snip)
>> > > Often, terminal output is unbuffered, file output is.
>>
>> > > Your compiler probably supports the FLUSH statement, as in
>> (snip)
>> > You may also get the effect you need using the tee program:
>>
>> > myprog | tee report.out
>> I used to use tee a lot, less often now.
>> But I do remember that the output to it is buffered, so it doesn't help.
>> It might be that you can turn off buffering in the program that
>> writes to the pipe.
>>
>> Now I more often use tail -f to follow the output written to the file, but again
>> (as the OP notes) it is buffered, so you won't see it for a while.
> Hm, so it is not as smart/magic as I thought. Well, in that case the simplest solution would indeed be to use flush explicitly.
>
> Should that be impractical or impossible, I can think of one
> other possibility: use the "expect" tool to fool the program into
> thinking it is outputting to a terminal.

That may be a bit of overkill.

There is another method: Look into the compiler documentation
and see if there is anything there.

For example, gfortran supports the GFORTRAN_UNBUFFERED_PRECONNECTED
environment variable; if that is set to "y", then standard output
should also be unbuffered. I'm fairly certain other compilers also
support this kind of thing.

Arjen Markus <arjen.markus895@gmail.com> wrote:
> On Friday, June 4, 2021 at 3:43:50 AM UTC+2, gah4 wrote:
>> On Thursday, June 3, 2021 at 7:47:55 AM UTC-7, arjen wrote:
>> > On Thursday, June 3, 2021 at 3:24:03 PM UTC+2, Thomas Koenig wrote:
>> (snip)
>> > > Often, terminal output is unbuffered, file output is.
>>
>> > > Your compiler probably supports the FLUSH statement, as in
>> (snip)
>> > You may also get the effect you need using the tee program:
>>
>> > myprog | tee report.out
>> I used to use tee a lot, less often now.
>> But I do remember that the output to it is buffered, so it doesn't help.
>> It might be that you can turn off buffering in the program that
>> writes to the pipe.
>>
>> Now I more often use tail -f to follow the output written to the file, but again
>> (as the OP notes) it is buffered, so you won't see it for a while.
> Hm, so it is not as smart/magic as I thought. Well, in that case the simplest solution would indeed be to use flush explicitly.
>
> Should that be impractical or impossible, I can think of one other possibility: use the "expect" tool to fool the program into thinking it is outputting to a terminal.

If the flush statement (Fortran 2003 and later) is unavailable and the
various flush extensions that existed before that are also
unavailable, one can often achieve the same effect by closing and
reopening the file, e.g.,

...
write(7,*)lots_of_stuff
close(7)
open(unit=7,file="whatever.txt",position="append")
write(7,*)lots_more_stuff
...

On Thu, 03 Jun 2021 13:24:01 +0000, Thomas Koenig wrote:

> Dieter Britz <dieterhansbritz@gmail.com> schrieb:
>> This might be a Linux question but I'm not sure.
>> When I run a Fortran program that prints a lot of stuff as it runs,
>> directly, it does that. But when I run it offline, with the output
>> going into a file, it only prints right near the end of execution. So
>> that file is empty for the whole time.
>
> Often, terminal output is unbuffered, file output is.
>
> Your compiler probably supports the FLUSH statement, as in
>
> program main
> use iso_fortran_env, only : output_unit implicit none write (*,'(A)')
> "Hello"
> flush output_unit ! Do something that takes a long time write
> (*,'(A)') "Goodbye"
> end program main
>
> Try that.

I just did. But I'm using print instead of write, and flush seems
to want the LUN. What is the LUN for print? I tried 5 but that didn't
work.

--
Dieter Britz

On Sat, 05 Jun 2021 15:34:04 +0000, Dieter Britz wrote:

> On Thu, 03 Jun 2021 13:24:01 +0000, Thomas Koenig wrote:
>
>> Dieter Britz <dieterhansbritz@gmail.com> schrieb:
>>> This might be a Linux question but I'm not sure.
>>> When I run a Fortran program that prints a lot of stuff as it runs,
>>> directly, it does that. But when I run it offline, with the output
>>> going into a file, it only prints right near the end of execution. So
>>> that file is empty for the whole time.
>>
>> Often, terminal output is unbuffered, file output is.
>>
>> Your compiler probably supports the FLUSH statement, as in
>>
>> program main
>> use iso_fortran_env, only : output_unit implicit none write (*,'(A)')
>> "Hello"
>> flush output_unit ! Do something that takes a long time write
>> (*,'(A)') "Goodbye"
>> end program main
>>
>> Try that.
>
> I just did. But I'm using print instead of write, and flush seems to
> want the LUN. What is the LUN for print? I tried 5 but that didn't work.

Sorry, my silly error, it should have been 6, and that works. Thanks
for the tip. I'm running programs that run for days on end and I would
like to see how much they do along the way, and this will let me do it.

--
Dieter Britz

On 6/5/21 10:40 AM, Dieter Britz wrote:
> On Sat, 05 Jun 2021 15:34:04 +0000, Dieter Britz wrote:
>
>> On Thu, 03 Jun 2021 13:24:01 +0000, Thomas Koenig wrote:
>>
>>> Dieter Britz <dieterhansbritz@gmail.com> schrieb:
>>>> This might be a Linux question but I'm not sure.
>>>> When I run a Fortran program that prints a lot of stuff as it runs,
>>>> directly, it does that. But when I run it offline, with the output
>>>> going into a file, it only prints right near the end of execution. So
>>>> that file is empty for the whole time.
>>>
>>> Often, terminal output is unbuffered, file output is.
>>>
>>> Your compiler probably supports the FLUSH statement, as in
>>>
>>> program main
>>> use iso_fortran_env, only : output_unit implicit none write (*,'(A)')
>>> "Hello"
>>> flush output_unit ! Do something that takes a long time write
>>> (*,'(A)') "Goodbye"
>>> end program main
>>>
>>> Try that.
>>
>> I just did. But I'm using print instead of write, and flush seems to
>> want the LUN. What is the LUN for print? I tried 5 but that didn't work.
>
> Sorry, my silly error, it should have been 6, and that works. Thanks
> for the tip. I'm running programs that run for days on end and I would
> like to see how much they do along the way, and this will let me do it.

Are you using a hardwired 6, or are you using output_unit as shown
above? I don't use "print" statements, but I think print and write will
both use the same output buffers and the output from each is properly
intermingled.

I use lots of old 40+ year old code still with hardwired 6 output units,
but I've been using iso_fortran_env for several years now with no
portability problems. That is what I would now recommend.

$.02 -Ron Shepard

extensions on OPEN, compiler switches and environment variables, FLUSH in particular are good solutions, but note there is a command on most Linux platforms that does what it was hoped tee(1) did -- unbuffer(1). If the program happens to be something you cannot change the code for or is not Fortran you might find it handy. It is also part of the expect(1) package so depending on your system you might need to install it or ask for expect(1) to be installed.

On Sunday, June 6, 2021 at 10:33:57 AM UTC-4, John wrote:
> extensions on OPEN, compiler switches and environment variables, FLUSH in particular are good solutions, but note there is a command on most Linux platforms that does what it was hoped tee(1) did -- unbuffer(1). If the program happens to be something you cannot change the code for or is not Fortran you might find it handy. It is also part of the expect(1) package so depending on your system you might need to install it or ask for expect(1) to be installed.

There is another command called stdbuf(1) that is similiar that also might be on your machine, although I prefer unbuffer.

On Sunday, June 6, 2021 at 1:40:05 AM UTC+10, Dieter Britz wrote:
> On Sat, 05 Jun 2021 15:34:04 +0000, Dieter Britz wrote:
>
> > On Thu, 03 Jun 2021 13:24:01 +0000, Thomas Koenig wrote:
> >
> >> Dieter Britz <dieterh...@gmail.com> schrieb:
> >>> This might be a Linux question but I'm not sure.
> >>> When I run a Fortran program that prints a lot of stuff as it runs,
> >>> directly, it does that. But when I run it offline, with the output
> >>> going into a file, it only prints right near the end of execution. So
> >>> that file is empty for the whole time.
> >>
> >> Often, terminal output is unbuffered, file output is.
> >>
> >> Your compiler probably supports the FLUSH statement, as in
> >>
> >> program main
> >> use iso_fortran_env, only : output_unit implicit none write (*,'(A)')
> >> "Hello"
> >> flush output_unit ! Do something that takes a long time write
> >> (*,'(A)') "Goodbye"
> >> end program main
> >>
> >> Try that.
> >
> > I just did. But I'm using print instead of write, and flush seems to
> > want the LUN. What is the LUN for print? I tried 5 but that didn't work.
> Sorry, my silly error, it should have been 6, and that works. Thanks
> for the tip. I'm running programs that run for days on end and I would
> like to see how much they do along the way, and this will let me do it.
>
> --
> Dieter Britz
My solution to this problem is to have a result update routine which packages the output in:

!$OMP CRITICAL
open (unit=111, file='run_hist.log', position='append')
! call write update routines
close (unit=111)
!$OMP END CRITICAL

I use this approach for analysis packets of about 30 minutes in a multi-threaded and repeated analysis.
It addresses my requirement for collecting results and reviewing a set of runs over days

On 6/10/21 4:36 AM, JCampbell wrote:
> On Sunday, June 6, 2021 at 1:40:05 AM UTC+10, Dieter Britz wrote:
>> On Sat, 05 Jun 2021 15:34:04 +0000, Dieter Britz wrote:
>>
>>> On Thu, 03 Jun 2021 13:24:01 +0000, Thomas Koenig wrote:
>>>
>>>> Dieter Britz <dieterh...@gmail.com> schrieb:
>>>>> This might be a Linux question but I'm not sure.
>>>>> When I run a Fortran program that prints a lot of stuff as it runs,
>>>>> directly, it does that. But when I run it offline, with the output
>>>>> going into a file, it only prints right near the end of execution. So
>>>>> that file is empty for the whole time.
>>>>
>>>> Often, terminal output is unbuffered, file output is.
>>>>
>>>> Your compiler probably supports the FLUSH statement, as in
>>>>
>>>> program main
>>>> use iso_fortran_env, only : output_unit implicit none write (*,'(A)')
>>>> "Hello"
>>>> flush output_unit ! Do something that takes a long time write
>>>> (*,'(A)') "Goodbye"
>>>> end program main
>>>>
>>>> Try that.
>>>
>>> I just did. But I'm using print instead of write, and flush seems to
>>> want the LUN. What is the LUN for print? I tried 5 but that didn't work.
>> Sorry, my silly error, it should have been 6, and that works. Thanks
>> for the tip. I'm running programs that run for days on end and I would
>> like to see how much they do along the way, and this will let me do it.
>>
>> --
>> Dieter Britz
> My solution to this problem is to have a result update routine which packages the output in:
>
> !$OMP CRITICAL
> open (unit=111, file='run_hist.log', position='append')
> ! call write update routines
> close (unit=111)
> !$OMP END CRITICAL
>
> I use this approach for analysis packets of about 30 minutes in a multi-threaded and repeated analysis.
> It addresses my requirement for collecting results and reviewing a set of runs over days

This uses unit=111, so it does not work with print, right? What is the
difference in this and the simple "flush output_unit" statement
suggested previously? Does this work correctly but "flush" did not? The
reason I'm asking is that simply closing a file is no guarantee that the
output buffer will be immediately flushed. The program could wait until
final termination before the output buffers within the i/o library are
actually flushed into the external file system. There is no strict
guarantee that "flush" will do that either, but at least that is the
stated intention of the operation, unlike the situation with "close",
whose real purpose is to allow the lun to be reused within the i/o library.

$.02 -Ron Shepard

Unbuffered I/O can be very inefficient so I prefer not to use unconditional coding to force a flush, but prefer using commands like unbuffer(1) or environment variables when the extension is available, the drawback being that you cannot do it retroactively if you decide you want to peek at the output of a long running job long after you started it; but I do always do a flush after an error message as I have seen messages get lost if an abnormal termination follows but I don't think it even occurred to me to make it a critical region. I though a CLOSE did say it did an implicit flush but (without trying it) I would expect a CLOSE() /OPEN() to be (at least potentially) a very expensive method performance-wise so I would hesitate to do that except for critical rarely-encountered messages. I thought a CLOSE() did guarantee the file would be flushed but so far I do not see that in the standard, so I am starting to doubt that after all.

PS: I have noticed on certain platforms that some commands seem to cause files to flush, like strace(1) and suspending and resuming a process but it has varied wildly depending on the OS but thought it is worth mentioning in case someone knows of a system routine that causes a process to flush everything to disk that I have missed. If there is one and no command to go with it it would make a nice utility to make a program to do that, even if just on Linux or Unix.

On 6/10/2021 8:19 PM, John wrote:
> Unbuffered I/O can be very inefficient so I prefer not to use unconditional coding to force a flush, but prefer using commands like unbuffer(1) or environment variables when the extension is available, the drawback being that you cannot do it retroactively if you decide you want to peek at the output of a long running job long after you started it; but I do always do a flush after an error message as I have seen messages get lost if an abnormal termination follows but I don't think it even occurred to me to make it a critical region. I though a CLOSE did say it did an implicit flush but (without trying it) I would expect a CLOSE() /OPEN() to be (at least potentially) a very expensive method performance-wise so I would hesitate to do that except for critical rarely-encountered messages. I thought a CLOSE() did guarantee the file would be flushed but so far I do not see that in the standard, so I am starting to doubt that after all.
>
You could just read for a key press (peak, not a wait) and only flush
when the process is started (i.e. read for either key not pressed or
another key to stop)...

On Saturday, June 12, 2021 at 11:52:06 AM UTC+10, Gary Scott wrote:
> On 6/10/2021 8:19 PM, John wrote:
> > Unbuffered I/O can be very inefficient so I prefer not to use unconditional coding to force a flush, but prefer using commands like unbuffer(1) or environment variables when the extension is available, the drawback being that you cannot do it retroactively if you decide you want to peek at the output of a long running job long after you started it; but I do always do a flush after an error message as I have seen messages get lost if an abnormal termination follows but I don't think it even occurred to me to make it a critical region. I though a CLOSE did say it did an implicit flush but (without trying it) I would expect a CLOSE() /OPEN() to be (at least potentially) a very expensive method performance-wise so I would hesitate to do that except for critical rarely-encountered messages. I thought a CLOSE() did guarantee the file would be flushed but so far I do not see that in the standard, so I am starting to doubt that after all.
> >
> You could just read for a key press (peak, not a wait) and only flush
> when the process is started (i.e. read for either key not pressed or
> another key to stop)..

While the use of CLOSE/OPEN might be considered expensive, if it is used once every 30 minutes (infrequently) this is not an issue. (There was an ifort/windows problem with repeated opening of the same file, but at the frequency I am using this approach, this problem has not been identified)
I used CRITICAL, to indicate I was using OpenMP and so sharing the unit 111 between all threads in a critical region appears to work effectively, overcoming any possible file buffer conflicts.
I also prefer a file output, rather than write (*.., managed via critical as it generates packets of reports that can be better understood and used as a run log. Write (* typically generates a mess and can easily overflow the screen buffer.
The way different O/S and compilers implement flush (I use gFortran on Windows), I would expect others to have different problems, but I suggested my approach as it has worked effectively for long mult-threaded runs.
If not addressed, my expoerience has been that gFortran does not flush text files which makes monitoring of long runs more difficult.

On 6/3/2021 7:19 AM, Dieter Britz wrote:
> This might be a Linux question but I'm not sure.
> When I run a Fortran program that prints a lot of stuff
> as it runs, directly, it does that. But when I run it
> offline, with the output going into a file, it only prints
> right near the end of execution. So that file is empty for
> the whole time.
>
> I run it with a script with this order:
>
> ./<script> >& <output file name> &
>
> Is there a way to run it so that it prints as it runs?

The 'script' command might work for you.

I compiled this program -- tsl.f90 (where "tsl" is short for "test
sleep" or something) -- as tsl:

===
program tsl
implicit none

integer n

do n = 1, 1000
    print *, n
    call sleep(5)
end do

end program
===

I ran it like this:

script -c ./tsl -f -O tsl.script

and it saved constantly flushed output in tsl.script.

Running it like this:

script -c ./tsl -f -O tsl.script > /dev/null

saved regularly flushed output in tsl.script but didn't send it to my
terminal window.

The saved script file has a header line "Script started on ..." and a
carriage return character after every output line.

I'm running Fedora 34. Your mileage may vary.

Louis