On Saturday, July 16, 2022 at 12:23:24 PM UTC-7, Louis Krupp wrote:

(snip)

> If each READ statement is definitely reading one and only one line, then
> reading each line into a character variable, deleting each instance of
> the word 'null', and then reading data from the character variable seems
> like the easiest solution by far.

I would find it easy enough to preprocess the file, such as:

sed "s/null//" < infile > outfile

Or maybe the generating program has an option to change this.

But yes, as a Fortran solution, reading line by line and processing the
line is often fine, too.

Many years ago (Fortran 77 days), I was reading a file something like:

READ(5,*, END=999) (X(I), Y(I), I=1,999)

and expecting the data to be read, and I to be 1 more than the pairs read.
That was before I saw a copy of the standard, and only had the DEC manuals.

I even had the DEC bug report form, before learning that the standard,
and especially the DEC compilers, didn't require that.

In the case of either EOF or I/O error, what is actually stored, or the value
of implied-DO variables, is undefined.

For my problem, a normal (not implied) DO fixed it, but others are
not so easy to fix.

On 7/16/22 1:04 PM, Robin Vowels wrote:
> Characters are characters, and cannot be read into an integer or real variable.

A machine-specific exception to this was list-directed i/o in VAX
fortran. It would read a T into an integer and translate it to -1, and
it would read a F into an integer and translate it to 0. These were the
internal bit representations for .true. and .false., so it was really
translating between logicals and integers, rather than between
characters and integers.

What do most compilers do when reading the characters NaN or Inf into a
real variable? Do they generate an error, or do they insert the IEEE bit
patterns?

$.02 -Ron Shepard

>>> On a completely separate matter, I have a different program that
>>> didn't behave as expected, and that misbehavior was totally repeatable.
>>> In an attempt to debug the program, I added a WRITE statement to check
>>> on the value of a variable during execution.  However, once the WRITE
>>> statement was added, the program started behaving properly, repeatably.
>>> Comment out the added WRITE statement, and the program once again
>>> misbehaves, repeatedly.  Re-enable the WRITE statement, and everything
>>> is once again hunky-dory.  Damned frustrating.  It's too easy to blame
>>> the optimizer.  Anybody have any generic advice on what to look for in
>>> such a situation?

>> Remove the write statement and compile with -Wall -fcheck=all.  This
>> might find where you are stomping on memory.

> I would add -Werror. If you always compile without warnings, your life will be simpler.

Oh, that's a deadly option for me. I use A LOT of implicit type conversion,
such as doing a calculation internally using double precision, but saving
the result in single precision, such as

snglz = dblex * dbley

-Werror is happy if an explicit REAL() is used. For the single precision
version of the real component of a double complex number, one needs to use
REAL(REAL(dblecmplx)) to make -Werror happy.

> If the presumably unlikely event that you're not already using IMPLICIT NONE in every program unit, I would do that, too.

I'm pretty religious about using IMPLICIT NONE in production code.

> Among the other things mentioned, a WRITE statement might mask problems caused by a variable that's not assigned a value before it's
> used.

Intriguing. How would that work? I've been trying to visualize how the
addition of a WRITE statement could change the computation.
>
> If all else fails, the debugger is your friend. If you haven't experienced the thrill of stepping through assembler instructions
> finding where your wayward variable is stored and examining its value and figuring out how it got to be what it is, this could be
> your chance.

I haven't used a real GUI debugger since the days of WATCOM on OS/2. If
only gfortran had something like that on Windows.

>>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
>>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>>>>>> of IOSTAT to 5001. I discovered this situation after having written a
>>>>>> program expecting subsequent READs to all return a negative number, and
>>>>>> of course the program didn't behave as expected until I investigated the
>>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
>>>>>> set to something other than -1 on any READ attempt after the end-of-file
>>>>>> condition has been triggered. Isn't the file pointer still just beyond
>>>>>> the last record in the file, even after the first failed READ?

>>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
>>>>> You are not expected to continue reading the file.
>>>> I actually do test for IOSTAT and deal with the end-of-file condition by
>>>> exiting the DO loop with the READ statement. But once the DO loop has
>>>> been exited, the program is in an outer DO loop, which can also read the
>>>> file.

>>> You need to fix the logic.

>> Apparently I haven't been clear.
>> The program has already been fixed.

> But it hasn't been fixed.

Incorrect; reread my previous statement.

> You admitted it.

Incorrect.

> You said that after detecting end of file using IOSTAT, you exited the loop,
> and the outer loop then proceeded to execute READs on the same file.

That was before the fix. Now it is using a named DO loop, and when
the data-reading DO loop encounters the end-of-file condition, it
exits the outer DO loop, so the outer DO loop's READ statement is
not encountered. That made the value of IOSTAT in the outer DO loop
irrelevant. However, that I succeeded in making my program work
properly did not change my curiosity as to why the value of IOSTAT
changed upon the second end-of-file READ.

>> It was fixed before I even posted on this forum. The question is not
>> how to fix the problem. The question is why the value of IOSTAT was
>> changed.

> That has already been explained to you.

Where? The closest anyone has come is Steve Lionel, who said he could
make an argument for the second instance not being -1, but he didn't
present that argument. That ifort returns -1 upon a second READ
demonstrates that the situation is "processor dependent".

>> There must be some rationale.

>>>> It also tests IOSTAT to deal with the end-of-file condition,

>>> The program has already hit the end of file.
>>> Now you are trying to read read the same file at the same place.

>> That's just it: same file, same place, same action, yet a different
>> IOSTAT result. Not what I had expected.

> Again, that has been explained to you.

Again, where?

>>>> but
>>>> it tested for a negative number, which IOSTAT was no longer set to.

>>> You need to avoid reading after end of file has been detected.

>> Why? IOSTAT was added so that a program could handle error conditions.

> Provided that you deal with the condition and do not do silly things like reading
> the same file again, after detecting end-of file.

Hardly silly, given that reading the same file again just triggers yet another
error condition, and IOSTAT was designed to allow programs to deal with the
error condition.

>> My program was designed to deal with the error condition. Had gfortran
>> consistently returned a negative IOSTAT for every attempt to READ past
>> the end-of-file, my program would have worked exactly as intended.

> Listen up. Your program is in error. Fix it.

Listen up: you are incorrect. My program *WAS* incompatible with gfortran's
handling of a second end-of-file READ. However, my program *WAS* compatible
with ifort's handling of a second end-of-file READ. My program *IS* no longer
incompatible with either compiler, and that was the case before I even
posted on this forum. Get your tenses straight.

>> Had I used ifort, my program would have worked exactly as intended.

>>>> And if you must know why the nested DO loops, one loop dealt with data
>>>> while the other loop dealt with metadata. It was easily fixable. My
>>>> question here was not about how to fix the problem, but rather to find
>>>> out the rationale for changing the value of IOSTAT in such a situation.
>>>> My expectation was that IOSTAT would remain negative. My expectation
>>>> was wrong, hence the curiosity.

>>>>>> On a completely separate matter, I have a different program that
>>>>>> didn't behave as expected, and that misbehavior was totally repeatable.
>>>>>> In an attempt to debug the program, I added a WRITE statement to check
>>>>>> on the value of a variable during execution. However, once the WRITE
>>>>>> statement was added, the program started behaving properly, repeatably.

>>>>> That suggests that there is a bug in your program.

>>>> Brilliant.

>>>>> Possibly something has
>>>>> been overwritten.

>>>> I already thought about that, hence the WRITE statement to examine
>>>> the values of the array indices.

>>> In every program unit, subroutine and function in the entire program?

>> I usually sprinkle WRITE statements around to isolate where the crash
>> is happening. As I narrow things down, unnecessary WRITE statements are
>> either removed or commented out. In this particular case, I got it down
>> to a single WRITE statement that could trigger proper behavior.

> You program has errors.

Perhaps. I doubt anyone can guarantee that the optimizer has zero bugs.
It wouldn't be the first time I've encountered a bug in an optimizer.
Clearly there is a bug somewhere, but you can't guarantee that it is in
my code.

> Find them with the help of debug options
> such as subscript bound errors, substring errors, and the like.

The WRITE statements showed that my subscripts are within bounds.
Substrings are not involved.

>>>> The values were within range and
>>>> the program worked properly with the WRITE statement enabled. Comment
>>>> out the WRITE statement, and the misbehavior returned. Hence the
>>>> frustration.

>>>>> Turn on subscript bounds checking (if available).
>>>>> And while you're at it, turn on all checks.

>>>> Easier said than done, as the program was built from dozens of separately
>>>> written and compiled subprograms. It has not always been obvious to me
>>>> whether a compilation flag needs to be applied to every single subprogram
>>>> in order to accomplish a goal.

>>> Obviously, program checking needs to be specified for every separate compilation,
>>> as the apparent overwriting could be in any one or more subroutine and function.

>> So, you're saying that if my program has subprograms A, B, C, D, and E,
>> and the crash occurs while it's executing subprogram C, the apparent
>> overwriting could be in A, B, D, or E?

>>>>>> Comment out the added WRITE statement, and the program once again
>>>>>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
>>>>>> is once again hunky-dory. Damned frustrating. It's too easy to blame
>>>>>> the optimizer. Anybody have any generic advice on what to look for in
>>>>>> such a situation?

>> So, you're saying that if my program has subprograms A, B, C, D, and E,
>> and the crash occurs while it's executing subprogram C, the apparent
>> overwriting could be in A, B, D, or E?

> The error could be in any of A, B, C, D, E, and/or the main program.

Can you provide an example?

On Sunday, July 17, 2022 at 6:13:50 AM UTC-4, Dave Tholen wrote:

> ..
> I haven't used a real GUI debugger since the days of WATCOM on OS/2. If
> only gfortran had something like that on Windows.

@Dave Tholen,

Re: "real GUI debugger" on Windows, you have a couple of options;

1. In the FOSS space, take a look at CodeBlocks::Fortran: there are also brief instructions, videos (e.g., YouTube) online as to how to use the graphical debugging capabilities with gfortran.
https://cbfortran.sourceforge.io/

2. In the complimentary software space, you may want to consider "Community" edition of Microsoft Visual Studio that you may be able to use without having to pay for a license. You can then look at Intel oneAPI HPC toolkit which now offers two Fortran compilers, IFORT and Intel's new LLVM based IFX, for free
https://visualstudio.microsoft.com/vs/community/
https://www.intel.com/content/www/us/en/developer/tools/oneapi/hpc-toolkit.html#gs.6kfdva

With Visual Studio IDE + Intel Fortran compilers, you can gain a real GUI debugging experience on Windows. Though there are some exasperating gaps and issues with this integration, they only come into play with a lot of modern Fortran features or use in a heavy-duty usage in a commercial big software type of setting where personal productivity demands are high.

On Sunday, July 17, 2022 at 10:02:53 AM UTC-4, Dave Tholen wrote:

> ..
> Can you provide an example?

@Dave Tholen,

If your interest is in gaining some insight and answers as opposed to an open-ended discussion, you may also want to post at the Fortran Discourse site:
https://fortran-lang.discourse.group/

And consider posting there first *your own small example* that illustrates what you are doing in your code.

On 7/15/22 6:47 AM, Dave Tholen wrote:
> If you read (sequentially) past the end-of-file, gfortran sets the value
> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> of IOSTAT to 5001. I discovered this situation after having written a
> program expecting subsequent READs to all return a negative number, and
> of course the program didn't behave as expected until I investigated the
> problem. So I'm curious as to why the choice was made to have IOSTAT
> set to something other than -1 on any READ attempt after the end-of-file
> condition has been triggered. Isn't the file pointer still just beyond
> the last record in the file, even after the first failed READ?

As others have explained, this dates back to when files were mostly
stored and accessed on magnetic tape. There were separate end-of-record
EOR and end-of-file EOF markers that were written to the tape, and you
could have separate files stored on a single tape, one after another.
The logical way to think of this is that each write to the tape would
write the data, followed by the EOR mark, followed by the EOF mark, and
then the tape was positioned to right before the EOF mark. If subsequent
records were written, they would overwrite the old EOF mark. If the tape
was rewound or unmounted or backspaced, or anything else, then that old
EOF mark would be there for the next time the tape was mounted or
positioned at that point.

After the last record in a tape file was read, a subsequent read would
detect the EOF mark on the tape. That information was returned through
the end= branch in the read statement or through the IOSTAT variable.

Once that happened, there were several things that could occur. One of
them was that you could read the next file. That next file might have
different record lengths, or it could have different
formatted/unformatted status, and so on, so the program needed some
ability to change those characteristics. This was all done in
nonportable, machine-specific ways before f77, and then f77 introduced
OPEN, CLOSE, and INQUIRE to make it more portable. But it was sometimes
still difficult to do things like this in a portable way.

Another thing that you could do was to BACKSPACE on the unit to move the
tape back to right before the EOF mark. From there, you could append
more records to that file. Of course not all files were stored on
magnetic tapes. There were other devices including paper tape, cards,
drums, disks, and now in modern times various kinds of solid state
devices and network file systems and so on. But the language allowed
these newer devices to look like they were magnetic tapes, so all this
stuff about records and EOR and EOF marks was still retained.

However, this one thing about EOF marks was not always done
consistently. In f77, the only way to append to a file was to read
records until the end= or IOSTAT value indicated that the file was
beyond the virtual EOF mark. From there, you were supposed to be able to
BACKSPACE the unit and then append new records. But many machines did
not do this correctly. There was often no actual EOF record written to
the file, and instead the end= condition was triggered when the file
pointer was at the end of the filesystem metadata for that file. On
these machines, no BACKSPACE was required, you could just start writing
the new records at that point. And even worse, if you did BACKSPACE, in
what should have been the correct instruction sequence, it would
backspace over the last record in the file. Then when you wrote the new
records, you would lose the information in that last record and the
record counts thereafter would all be off by one. So for the 1980s and
even 1990s, a cautious programmer had to test for this condition and
only do the BACKSPACE conditionally. I have not tested this recently,
but I expect it still happens when one tries to append to a file that
has been positioned with end=.

F90 allowed a file to be opened and positioned appropriately to append
new records. So the conditional BACKSPACE code could be replaced with
this new portable functionality.

So for your question, it is because EOF is not treated as an error
condition. It is intended to be a normal situation that a programmer can
encounter, detect, and continue processing accordingly in a portable
way. However, reading or writing past the EOF might or might not be
allowed on a given device, or for a file on some device. If it is not
allowed, then it is an error condition, an exceptional situation that
should not be normally encountered.

> On a completely separate matter, I have a different program that
> didn't behave as expected, and that misbehavior was totally repeatable.
> In an attempt to debug the program, I added a WRITE statement to check
> on the value of a variable during execution. However, once the WRITE
> statement was added, the program started behaving properly, repeatably.
> Comment out the added WRITE statement, and the program once again
> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
> is once again hunky-dory. Damned frustrating. It's too easy to blame
> the optimizer. Anybody have any generic advice on what to look for in
> such a situation?

These are called 'Heisenbugs'. The program is corrupting either the data
or the stored instruction code somehow. It could be caused by
out-of-range array indexing, or by some illegal argument mismatch (e.g.
a scalar actual argument associated with a dummy array argument), or
referencing an undefined or stale pointer, or maybe by switching the
order of some arguments that otherwise pass the type-kind-rank (TKR)
detection that the compiler can detect (an example might be switching
row and column indices of an array). The difficult thing about locating
these types of errors is that the backtrace does not always occur right
when the error occurs. The corruption can occur at one point, then then
execution can continue for some time before that data is accessed or
those instructions are executed.

These bugs were quite common before f90. F90 introduced several new
features that help to detect or eliminate these errors. These include
TKR checking for subroutines that have explicit interfaces, assumed
shape array declarations, the IMPLICIT NONE declaration (which helps
eliminate local variables that arise because of typos and misspellings),
and free-form source (which eliminates errors due to running past column
72 in the older fixed-form source). If you are writing new code, then
try to use these features. If you are working with legacy code, then you
might try moving some of the newer features into the code in order to
get some help from the compiler. Of course, modern fortran also has new
ways to introduce Heisenbugs, such as pointers which can become stale
and reference data that no longer exists in memory.

$.02 -Ron Shepard

On Monday, July 18, 2022 at 12:01:30 AM UTC+10, Dave Tholen wrote:
> >>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
> >>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> >>>>>> of IOSTAT to 5001. I discovered this situation after having written a
> >>>>>> program expecting subsequent READs to all return a negative number, and
> >>>>>> of course the program didn't behave as expected until I investigated the
> >>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
> >>>>>> set to something other than -1 on any READ attempt after the end-of-file
> >>>>>> condition has been triggered. Isn't the file pointer still just beyond
> >>>>>> the last record in the file, even after the first failed READ?
>
> >>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
> >>>>> You are not expected to continue reading the file.
> >>>> I actually do test for IOSTAT and deal with the end-of-file condition by
> >>>> exiting the DO loop with the READ statement. But once the DO loop has
> >>>> been exited, the program is in an outer DO loop, which can also read the
> >>>> file.
> >>> You need to fix the logic.
> >> Apparently I haven't been clear.
> >> The program has already been fixed.
> > But it hasn't been fixed.
> Incorrect; reread my previous statement.
>
> > You admitted it.
>
> Incorrect.
> > You said that after detecting end of file using IOSTAT, you exited the loop,
> > and the outer loop then proceeded to execute READs on the same file.
> That was before the fix. Now it is using a named DO loop, and when
> the data-reading DO loop encounters the end-of-file condition, it
> exits the outer DO loop, so the outer DO loop's READ statement is
> not encountered. That made the value of IOSTAT in the outer DO loop
> irrelevant. However, that I succeeded in making my program work
> properly did not change my curiosity as to why the value of IOSTAT
> changed upon the second end-of-file READ.
> >> It was fixed before I even posted on this forum. The question is not
> >> how to fix the problem. The question is why the value of IOSTAT was
> >> changed.
> > That has already been explained to you.
> Where? The closest anyone has come is Steve Lionel, who said he could
> make an argument for the second instance not being -1, but he didn't
> present that argument. That ifort returns -1 upon a second READ
> demonstrates that the situation is "processor dependent".
> >> There must be some rationale.
>
> >>>> It also tests IOSTAT to deal with the end-of-file condition,
>
> >>> The program has already hit the end of file.
> >>> Now you are trying to read read the same file at the same place.
>
> >> That's just it: same file, same place, same action, yet a different
> >> IOSTAT result. Not what I had expected.
> > Again, that has been explained to you.
> Again, where?
> >>>> but
> >>>> it tested for a negative number, which IOSTAT was no longer set to.
>
> >>> You need to avoid reading after end of file has been detected.
>
> >> Why? IOSTAT was added so that a program could handle error conditions.
..
IOSTAT was added so that end-of-file could be detected and dealt with.
IOSTAT was also added so that error conditions could be detected.
[Note that end-of-file is not considered to be an error condition.]
..
> > Provided that you deal with the condition and do not do silly things like reading
> > the same file again, after detecting end-of file.

> Hardly silly, given that reading the same file again just triggers yet another
> error condition,
..
No it doesn't. Reading the same file again after end of file has been detected
is the first error condition raised.
..
IOSTAT returns a negative value for end of file.
IOSTAT returns a positive integer for an error condition.
..
> and IOSTAT was designed to allow programs to deal with the
> error condition.
>
> >> My program was designed to deal with the error condition. Had gfortran
> >> consistently returned a negative IOSTAT for every attempt to READ past
> >> the end-of-file, my program would have worked exactly as intended.
>
> > Listen up. Your program is in error. Fix it.
> Listen up: you are incorrect.
..
I am not incorrect. Read the standard.
..
> My program *WAS* incompatible with gfortran's
> handling of a second end-of-file READ.
..
No, your program was wrong.
..
Your program was in error when you tried to read the file again,
after receiving end-of-file notification.
That's why you received a positive number for IOSTAT when you tried
to read the file after receiving end-of-file.
..
> However, my program *WAS* compatible
> with ifort's handling of a second end-of-file READ. My program *IS* no longer
> incompatible with either compiler, and that was the case before I even
> posted on this forum. Get your tenses straight.
> >> Had I used ifort, my program would have worked exactly as intended.
>
> >>>> And if you must know why the nested DO loops, one loop dealt with data
> >>>> while the other loop dealt with metadata. It was easily fixable. My
> >>>> question here was not about how to fix the problem, but rather to find
> >>>> out the rationale for changing the value of IOSTAT in such a situation.
> >>>> My expectation was that IOSTAT would remain negative. My expectation
> >>>> was wrong, hence the curiosity.
>
> >>>>>> On a completely separate matter, I have a different program that
> >>>>>> didn't behave as expected, and that misbehavior was totally repeatable.
> >>>>>> In an attempt to debug the program, I added a WRITE statement to check
> >>>>>> on the value of a variable during execution. However, once the WRITE
> >>>>>> statement was added, the program started behaving properly, repeatably.
>
> >>>>> That suggests that there is a bug in your program.
>
> >>>> Brilliant.
>
> >>>>> Possibly something has
> >>>>> been overwritten.
>
> >>>> I already thought about that, hence the WRITE statement to examine
> >>>> the values of the array indices.
>
> >>> In every program unit, subroutine and function in the entire program?
>
> >> I usually sprinkle WRITE statements around to isolate where the crash
> >> is happening. As I narrow things down, unnecessary WRITE statements are
> >> either removed or commented out. In this particular case, I got it down
> >> to a single WRITE statement that could trigger proper behavior.
..
> > Your program has errors.
>
> Perhaps. I doubt anyone can guarantee that the optimizer has zero bugs.
..
It's common for people to blame the compiler when their program does not work,
especially when they do not bother to read the manual.

> It wouldn't be the first time I've encountered a bug in an optimizer.
> Clearly there is a bug somewhere, but you can't guarantee that it is in
> my code.
..
Can you guarantee that the bug(s) is not in your code?
..
> > Find them with the help of debug options
> > such as subscript bound errors, substring errors, and the like.
> The WRITE statements showed that my subscripts are within bounds.
..
That's not the same as enabling subscript checks and all other checks.
..
> Substrings are not involved.
> >>>> The values were within range and
> >>>> the program worked properly with the WRITE statement enabled. Comment
> >>>> out the WRITE statement, and the misbehavior returned. Hence the
> >>>> frustration.
>
> >>>>> Turn on subscript bounds checking (if available).
> >>>>> And while you're at it, turn on all checks.
> >>>> Easier said than done, as the program was built from dozens of separately
> >>>> written and compiled subprograms. It has not always been obvious to me
> >>>> whether a compilation flag needs to be applied to every single subprogram
> >>>> in order to accomplish a goal.
>
> >>> Obviously, program checking needs to be specified for every separate compilation,
> >>> as the apparent overwriting could be in any one or more subroutine and function.
>
> >> So, you're saying that if my program has subprograms A, B, C, D, and E,
> >> and the crash occurs while it's executing subprogram C, the apparent
> >> overwriting could be in A, B, D, or E?

On Sunday, July 17, 2022 at 12:18:18 PM UTC-4, Ron Shepard wrote:

> ..
> So for your question, it is because EOF is not treated as an error
> condition. It is intended to be a normal situation that a programmer can
> encounter, detect, and continue processing accordingly in a portable
> way. However, reading or writing past the EOF might or might not be
> allowed on a given device, or for a file on some device. If it is not
> allowed, then it is an error condition, an exceptional situation that
> should not be normally encountered. ..

@Dave Tholen,

You may want to take note with current Fortran, it will help to not deal directly with specific values of IOSTAT but to take a higher level view of them.

You may know Fortran now includes an intrinsic module ISO_FORTRAN_ENV which makes available via USE association among other entities also two named constants, IOSTAT_END and IOSTAT_EOR.

You may find it helpful to keep in mind the following when using data transfer statements and IOSTAT= optional parameter is being used; just an example, say with IOSTAT=ISTAT in READ and WRITE statements:

1. ISTAT == 0 applies to a data transfer when no exception condition has occurred, business as usual,
2. ISTAT == IOSTAT_END corresponds to an end-of-file condition. Note this is separate from an error,
3. ISTAT == IOSTAT_EOR corresponds to an end-of-record condition; particularly relevant to direct-access IO where RECL (record-length) is involved,
4. ISTAT equal to anything else corresponds to an IO error condition.

With above, whether the specific value of ISTAT following a data transfer is -1 or 5001 does not quite matter, what is relevant is where the ISTAT value falls in the above listed situations 1 thru' 4.

Someone with gfortran expertise can shed further light, but if you really wish to get into -1 vs 5001 in the READ you did with gfortran after the EOF was encountered, perhaps the following from the standard might help explain:

"If an end-of-file condition occurs during execution of an input/output statement that contains either an END= specifier or an IOSTAT= specifier, and an error condition does not occur then:

(1) processing of the input list, if any, terminates;
(2) if the statement is a data transfer statement or the end-of-file condition occurs during a wait operation,
all do-variables in the statement that initiated the transfer become undefined;
(3) if the statement is an input statement or the end-of-file condition occurs during a wait operation
for a transfer initiated by an input statement, all effective items resulting from the expansion of list
items or the namelist group in the statement that initiated the transfer become undefined;
(4) if the file specified in the input statement is an external record file, it is positioned after the endfile record;"

See point (4). Given such positioning, it does not make sense the subsequent READ you attempt on the same unit should lead to an EOF situation i.e., the ISTAT thus cannot be IOSTAT_END again.

I think about half of this would have been clarified by actually using the IOMSG= feature, which I did not find actually being used?
A simple example program demonstrates several of the points mentioned early on ....

cat /tmp/xp.f90
program testit
implicit none
integer :: iostat,i
character(len=256) :: iomsg,line
do i=1,10
read(*,*,iostat=iostat,iomsg=iomsg)line
if(iostat.ne.0)then
write(*,*)iostat,trim(iomsg)
else
write(*,*)'i=',i,trim(line)
endif
enddo

end program testit

$ xp </dev/null
-1 End of file
5001 Sequential READ or WRITE not allowed after EOF marker, possibly use REWIND or BACKSPACE

If you create a file with three lines
file1
ctrl-Z
file2

where ctrl-Z is literally the character ctrl-Z and try it with different compilers, you will find some compilers treat the ctrl-Z as an end-of-file that you can read past even on Linux platforms (an extension in my opinion); that after the first EOF the message changes and in some cases remains a -1 and in others does not. These are all acceptable behaviors if someone surprising in some cases as the standard is not specific about exactly what an EOF is; and at least one compiler appears to treat the ctrl-Z as an EOF only for formatted sequential I/O when the ctrl-Z is on a "line" by itself. All dependably return a value matching IOSTAT_END the first time they encounter what they define as an end-of-file.

But for the specific question about why -1 and 5001 tfor gfortran he IOMSG text seems quite self-explanatory and standard-conforming.
Remember that if you are not trapping the IOSTAT value the messages returned in IOMSG generally appear on stderr and the program ends
on error.

>>>>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
>>>>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>>>>>>>> of IOSTAT to 5001. I discovered this situation after having written a
>>>>>>>> program expecting subsequent READs to all return a negative number, and
>>>>>>>> of course the program didn't behave as expected until I investigated the
>>>>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
>>>>>>>> set to something other than -1 on any READ attempt after the end-of-file
>>>>>>>> condition has been triggered. Isn't the file pointer still just beyond
>>>>>>>> the last record in the file, even after the first failed READ?

>>>>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
>>>>>>> You are not expected to continue reading the file.

>>>>>> I actually do test for IOSTAT and deal with the end-of-file condition by
>>>>>> exiting the DO loop with the READ statement. But once the DO loop has
>>>>>> been exited, the program is in an outer DO loop, which can also read the
>>>>>> file.

>>>>> You need to fix the logic.

>>>> Apparently I haven't been clear.
>>>> The program has already been fixed.

>>> But it hasn't been fixed.

>> Incorrect; reread my previous statement.

>>> You admitted it.

>> Incorrect.

>>> You said that after detecting end of file using IOSTAT, you exited the loop,
>>> and the outer loop then proceeded to execute READs on the same file.

>> That was before the fix. Now it is using a named DO loop, and when
>> the data-reading DO loop encounters the end-of-file condition, it
>> exits the outer DO loop, so the outer DO loop's READ statement is
>> not encountered. That made the value of IOSTAT in the outer DO loop
>> irrelevant. However, that I succeeded in making my program work
>> properly did not change my curiosity as to why the value of IOSTAT
>> changed upon the second end-of-file READ.

>>>> It was fixed before I even posted on this forum. The question is not
>>>> how to fix the problem. The question is why the value of IOSTAT was
>>>> changed.

>>> That has already been explained to you.

>> Where? The closest anyone has come is Steve Lionel, who said he could
>> make an argument for the second instance not being -1, but he didn't
>> present that argument. That ifort returns -1 upon a second READ
>> demonstrates that the situation is "processor dependent".

>>>> There must be some rationale.

>>>>>> It also tests IOSTAT to deal with the end-of-file condition,

>>>>> The program has already hit the end of file.
>>>>> Now you are trying to read read the same file at the same place.

>>>> That's just it: same file, same place, same action, yet a different
>>>> IOSTAT result. Not what I had expected.

>>> Again, that has been explained to you.

>> Again, where?

>>>>>> but
>>>>>> it tested for a negative number, which IOSTAT was no longer set to.

>>>>> You need to avoid reading after end of file has been detected.

>>>> Why? IOSTAT was added so that a program could handle error conditions.
..
> IOSTAT was added so that end-of-file could be detected and dealt with.
> IOSTAT was also added so that error conditions could be detected.
> [Note that end-of-file is not considered to be an error condition.]

But another READ past the end-of-file apparently is considered to be an
error condition by gfortran, whereas ifort apparently considers it to be
an end-of-file condition.

>>> Provided that you deal with the condition and do not do silly things like reading
>>> the same file again, after detecting end-of file.

>> Hardly silly, given that reading the same file again just triggers yet another
>> error condition,

> No it doesn't. Reading the same file again after end of file has been detected
> is the first error condition raised.

You don't know that. In fact, if you read elsewhere in this thread, you'd
know that the data file uses the characters "null" in place of a missing
numerical value, which triggers multiple error conditions long before a
second end-of-file READ occurs. It's hardly the first error condition
raised.

> IOSTAT returns a negative value for end of file.
> IOSTAT returns a positive integer for an error condition.

Yet ifort returns -1 for a second end-of-file READ while gfortran returns
5001. Are you arguing that one of the two compilers is not standard
compliant?

>> and IOSTAT was designed to allow programs to deal with the
>> error condition.

>>>> My program was designed to deal with the error condition. Had gfortran
>>>> consistently returned a negative IOSTAT for every attempt to READ past
>>>> the end-of-file, my program would have worked exactly as intended.

>>> Listen up. Your program is in error. Fix it.

>> Listen up: you are incorrect.

> I am not incorrect. Read the standard.

You are incorrect. My program is standard-conforming and works properly.

>> My program *WAS* incompatible with gfortran's
>> handling of a second end-of-file READ.

> No, your program was wrong.

Looks like you've finally admitted (tacitly) that you had your tenses wrong.
Given that the different compilers return IOSTAT values with different signs
for the same second end-of-file READ, it can be argued that the situation is
"processor dependent". Forty years ago, programs that OPENed a file and
started READing worked just fine with compilers that set the file pointer to
the beginning of the file, until they were moved to a BSD UNIX system whose
compiler, by default, OPENed a file with the pointer at the end of the file.
The program wasn't "wrong"; rather, it was incompatible with a processor
dependency.

> Your program was in error when you tried to read the file again,
> after receiving end-of-file notification.

Rather, an error condition was triggered when the program tried to read
the file again, but the code included a test on the value of IOSTAT to
deal with the condition.

> That's why you received a positive number for IOSTAT when you tried
> to read the file after receiving end-of-file.

Yet ifort returns a negative number. Are you arguing that ifort is in
error?

>> However, my program *WAS* compatible
>> with ifort's handling of a second end-of-file READ. My program *IS* no longer
>> incompatible with either compiler, and that was the case before I even
>> posted on this forum. Get your tenses straight.

>>>> Had I used ifort, my program would have worked exactly as intended.

>>>>>> And if you must know why the nested DO loops, one loop dealt with data
>>>>>> while the other loop dealt with metadata. It was easily fixable. My
>>>>>> question here was not about how to fix the problem, but rather to find
>>>>>> out the rationale for changing the value of IOSTAT in such a situation.
>>>>>> My expectation was that IOSTAT would remain negative. My expectation
>>>>>> was wrong, hence the curiosity.

>>>>>>>> On a completely separate matter, I have a different program that
>>>>>>>> didn't behave as expected, and that misbehavior was totally repeatable.
>>>>>>>> In an attempt to debug the program, I added a WRITE statement to check
>>>>>>>> on the value of a variable during execution. However, once the WRITE
>>>>>>>> statement was added, the program started behaving properly, repeatably.

>>>>>>> That suggests that there is a bug in your program.

>>>>>> Brilliant.

>>>>>>> Possibly something has
>>>>>>> been overwritten.

>>>>>> I already thought about that, hence the WRITE statement to examine
>>>>>> the values of the array indices.

>>>>> In every program unit, subroutine and function in the entire program?

>>>> I usually sprinkle WRITE statements around to isolate where the crash
>>>> is happening. As I narrow things down, unnecessary WRITE statements are
>>>> either removed or commented out. In this particular case, I got it down
>>>> to a single WRITE statement that could trigger proper behavior.

>>> Your program has errors.

>> Can you provide an example?

> If your interest is in gaining some insight and answers as opposed to an open-ended discussion, you may also want to post at the Fortran Discourse site:
> https://fortran-lang.discourse.group/

Are you suggesting that fortran-lang.discourse.group is for insight and that
comp.lang.fortran is for open-ended discussions?

> And consider posting there first *your own small example* that illustrates what you are doing in your code.

You're assuming that the problem can be reproduced with a "small example".
The program isn't small. If the simple addition of a WRITE statement can
trigger correct behavior, shrinking of the task to a "small example" will
lead to massive changes that more than likely will also eliminate the bug,
especially if the bug is NOT in the subprogram where the crash occurs.

On 7/17/2022 4:13 AM, Dave Tholen wrote:
>>>> On a completely separate matter, I have a different program that
>>>> didn't behave as expected, and that misbehavior was totally
>>>> repeatable.
>>>> In an attempt to debug the program, I added a WRITE statement to check
>>>> on the value of a variable during execution.  However, once the WRITE
>>>> statement was added, the program started behaving properly,
>>>> repeatably.
>>>> Comment out the added WRITE statement, and the program once again
>>>> misbehaves, repeatedly.  Re-enable the WRITE statement, and everything
>>>> is once again hunky-dory.  Damned frustrating.  It's too easy to blame
>>>> the optimizer.  Anybody have any generic advice on what to look for in
>>>> such a situation?
>
>>> Remove the write statement and compile with -Wall -fcheck=all.  This
>>> might find where you are stomping on memory.
>
>> I would add -Werror. If you always compile without warnings, your
>> life will be simpler.
>
> Oh, that's a deadly option for me.  I use A LOT of implicit type
> conversion,
> such as doing a calculation internally using double precision, but saving
> the result in single precision, such as
>
>     snglz = dblex * dbley
>
> -Werror is happy if an explicit REAL() is used.  For the single precision
> version of the real component of a double complex number, one needs to
> use
> REAL(REAL(dblecmplx)) to make -Werror happy.
>

You might be able to silence conversion warnings. This code, for example:

===
subroutine s(xi, xo)

implicit none

double precision, intent(in) :: xi
real, intent(out) :: xo

xo = xi

return
end
===

compiles quietly with these options:

gfortran -c -Wall -Wno-conversion -Werror ...

>> If the presumably unlikely event that you're not already using
>> IMPLICIT NONE in every program unit, I would do that, too.
>
> I'm pretty religious about using IMPLICIT NONE in production code.
>
>> Among the other things mentioned, a WRITE statement might mask
>> problems caused by a variable that's not assigned a value before it's
>> used.
>
> Intriguing.  How would that work?  I've been trying to visualize how the
> addition of a WRITE statement could change the computation.

Don't even try to visualize it. Some things become clear only in
retrospect, and this could be one of them.

I have seen apparently unrelated code changes move variables from
registers to memory, and that can trigger the difference between an
uninitialized variable accidentally inheriting garbage instead of the
zero value the programmer was expecting. Sometimes there's a chain of
events.

I vaguely recall a post from a year or two ago in which a subprogram
ignored the value of an argument that had been declared INTENT(OUT). The
results were standard-compliant but surprising.

On the off chance that your compiler is misbehaving or perhaps not
issuing a warning when it should, are you running the latest version of
gfortran?

>>
>> If all else fails, the debugger is your friend. If you haven't
>> experienced the thrill of stepping through assembler instructions
>> finding where your wayward variable is stored and examining its value
>> and figuring out how it got to be what it is, this could be your chance.
>
> I haven't used a real GUI debugger since the days of WATCOM on OS/2.  If
> only gfortran had something like that on Windows.
>
>

On Sunday, July 17, 2022 at 11:48:12 AM UTC-7, FortranFan wrote:

(snip)

> "If an end-of-file condition occurs during execution of an input/output statement that contains either an END= specifier or an IOSTAT= specifier, and an error condition does not occur then:
> (1) processing of the input list, if any, terminates;
> (2) if the statement is a data transfer statement or the end-of-file condition occurs during a wait operation,
> all do-variables in the statement that initiated the transfer become undefined;
> (3) if the statement is an input statement or the end-of-file condition occurs during a wait operation
> for a transfer initiated by an input statement, all effective items resulting from the expansion of list
> items or the namelist group in the statement that initiated the transfer become undefined;
> (4) if the file specified in the input statement is an external record file, it is positioned after the endfile record;"
Those are useful, but there are also ones that apply for I/O errors.

I believe, like (3), in the case of I/O error that all the I/O list elements are undefined.

For comparison, in C all I/O elements before EOF or any other condition occurs are
not undefined, but have the appropriate value. That can be very convenient.

On Monday, July 18, 2022 at 9:49:52 AM UTC+10, Dave Tholen wrote:
> >>>>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
> >>>>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> >>>>>>>> of IOSTAT to 5001. I discovered this situation after having written a
> >>>>>>>> program expecting subsequent READs to all return a negative number, and
> >>>>>>>> of course the program didn't behave as expected until I investigated the
> >>>>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
> >>>>>>>> set to something other than -1 on any READ attempt after the end-of-file
> >>>>>>>> condition has been triggered. Isn't the file pointer still just beyond
> >>>>>>>> the last record in the file, even after the first failed READ?
>
> >>>>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
> >>>>>>> You are not expected to continue reading the file.
>
> >>>>>> I actually do test for IOSTAT and deal with the end-of-file condition by
> >>>>>> exiting the DO loop with the READ statement. But once the DO loop has
> >>>>>> been exited, the program is in an outer DO loop, which can also read the
> >>>>>> file.
>
> >>>>> You need to fix the logic.
>
> >>>> Apparently I haven't been clear.
> >>>> The program has already been fixed.
>
> >>> But it hasn't been fixed.
>
> >> Incorrect; reread my previous statement.
>
> >>> You admitted it.
>
> >> Incorrect.
>
> >>> You said that after detecting end of file using IOSTAT, you exited the loop,
> >>> and the outer loop then proceeded to execute READs on the same file.
>
> >> That was before the fix. Now it is using a named DO loop, and when
> >> the data-reading DO loop encounters the end-of-file condition, it
> >> exits the outer DO loop, so the outer DO loop's READ statement is
> >> not encountered. That made the value of IOSTAT in the outer DO loop
> >> irrelevant. However, that I succeeded in making my program work
> >> properly did not change my curiosity as to why the value of IOSTAT
> >> changed upon the second end-of-file READ.
>
> >>>> It was fixed before I even posted on this forum. The question is not
> >>>> how to fix the problem. The question is why the value of IOSTAT was
> >>>> changed.
>
> >>> That has already been explained to you.
>
> >> Where? The closest anyone has come is Steve Lionel, who said he could
> >> make an argument for the second instance not being -1, but he didn't
> >> present that argument. That ifort returns -1 upon a second READ
> >> demonstrates that the situation is "processor dependent".
>
> >>>> There must be some rationale.
>
> >>>>>> It also tests IOSTAT to deal with the end-of-file condition,
>
> >>>>> The program has already hit the end of file.
> >>>>> Now you are trying to read read the same file at the same place.
>
> >>>> That's just it: same file, same place, same action, yet a different
> >>>> IOSTAT result. Not what I had expected.
>
> >>> Again, that has been explained to you.
>
> >> Again, where?
>
> >>>>>> but
> >>>>>> it tested for a negative number, which IOSTAT was no longer set to.
>
> >>>>> You need to avoid reading after end of file has been detected.
>
> >>>> Why? IOSTAT was added so that a program could handle error conditions.
> .
> > IOSTAT was added so that end-of-file could be detected and dealt with.
> > IOSTAT was also added so that error conditions could be detected.
> > [Note that end-of-file is not considered to be an error condition.]
> But another READ past the end-of-file apparently is considered to be an
> error condition by gfortran, whereas ifort apparently considers it to be
> an end-of-file condition.
> >>> Provided that you deal with the condition and do not do silly things like reading
> >>> the same file again, after detecting end-of file.
>
> >> Hardly silly, given that reading the same file again just triggers yet another
> >> error condition,
> > No it doesn't. Reading the same file again after end of file has been detected
> > is the first error condition raised.
> You don't know that. In fact, if you read elsewhere in this thread, you'd
> know that the data file uses the characters "null" in place of a missing
> numerical value, which triggers multiple error conditions long before a
> second end-of-file READ occurs. It's hardly the first error condition
> raised.
> > IOSTAT returns a negative value for end of file.
> > IOSTAT returns a positive integer for an error condition.
> Yet ifort returns -1 for a second end-of-file READ while gfortran returns
> 5001. Are you arguing that one of the two compilers is not standard
> compliant?
> >> and IOSTAT was designed to allow programs to deal with the
> >> error condition.
>
> >>>> My program was designed to deal with the error condition. Had gfortran
> >>>> consistently returned a negative IOSTAT for every attempt to READ past
> >>>> the end-of-file, my program would have worked exactly as intended.
>
> >>> Listen up. Your program is in error. Fix it.
>
> >> Listen up: you are incorrect.
> > I am not incorrect. Read the standard.
> You are incorrect. My program is standard-conforming and works properly.
>
Since you have not read the standard, you do not know whether your program
is "standard conforming".
>
> >> My program *WAS* incompatible with gfortran's
> >> handling of a second end-of-file READ.
> > No, your program was wrong.
> Looks like you've finally admitted (tacitly) that you had your tenses wrong.
> Given that the different compilers return IOSTAT values with different signs
> for the same second end-of-file READ, it can be argued that the situation is
> "processor dependent".
>
All the IOSTAT values are processor-dependent (except for zero).
However, the value returned for end-of-file is negative.
The value for an error condition is positive.
This infrioation is in the standard.
>
> Forty years ago, programs that OPENed a file and
> started READing worked just fine with compilers that set the file pointer to
> the beginning of the file, until they were moved to a BSD UNIX system whose
> compiler, by default, OPENed a file with the pointer at the end of the file.
> The program wasn't "wrong"; rather, it was incompatible with a processor
> dependency.
> > Your program was in error when you tried to read the file again,
> > after receiving end-of-file notification.
> Rather, an error condition was triggered when the program tried to read
> the file again,

And because your program contains an error, it is not standard conforming.

but the code included a test on the value of IOSTAT to
> deal with the condition.
> > That's why you received a positive number for IOSTAT when you tried
> > to read the file after receiving end-of-file.
>
> Yet ifort returns a negative number. Are you arguing that ifort is in
> error?
>
ifort is in error.
You still have not read the standard.
>
> >> However, my program *WAS* compatible
> >> with ifort's handling of a second end-of-file READ. My program *IS* no longer
> >> incompatible with either compiler, and that was the case before I even
> >> posted on this forum. Get your tenses straight.
>
> >>>> Had I used ifort, my program would have worked exactly as intended.
>
> >>>>>> And if you must know why the nested DO loops, one loop dealt with data
> >>>>>> while the other loop dealt with metadata. It was easily fixable. My
> >>>>>> question here was not about how to fix the problem, but rather to find
> >>>>>> out the rationale for changing the value of IOSTAT in such a situation.
> >>>>>> My expectation was that IOSTAT would remain negative. My expectation
> >>>>>> was wrong, hence the curiosity.
>
> >>>>>>>> On a completely separate matter, I have a different program that
> >>>>>>>> didn't behave as expected, and that misbehavior was totally repeatable.
> >>>>>>>> In an attempt to debug the program, I added a WRITE statement to check
> >>>>>>>> on the value of a variable during execution. However, once the WRITE
> >>>>>>>> statement was added, the program started behaving properly, repeatably.
>
> >>>>>>> That suggests that there is a bug in your program.
>
> >>>>>> Brilliant.
>
> >>>>>>> Possibly something has
> >>>>>>> been overwritten.
>
> >>>>>> I already thought about that, hence the WRITE statement to examine
> >>>>>> the values of the array indices.
>
> >>>>> In every program unit, subroutine and function in the entire program?
>
> >>>> I usually sprinkle WRITE statements around to isolate where the crash
> >>>> is happening. As I narrow things down, unnecessary WRITE statements are
> >>>> either removed or commented out. In this particular case, I got it down
> >>>> to a single WRITE statement that could trigger proper behavior.
> >>> Your program has errors.
>
> >> Perhaps. I doubt anyone can guarantee that the optimizer has zero bugs.
>
> > It's common for people to blame the compiler when their program does not work,
> > especially when they do not bother to read the manual.
>
> Hence why I wrote "It's too easy to blame the optimizer."
>
> >> It wouldn't be the first time I've encountered a bug in an optimizer.
> >> Clearly there is a bug somewhere, but you can't guarantee that it is in
> >> my code.
>
> > Can you guarantee that the bug(s) is not in your code?
>
> Obviously not, hence my request for "generic advice on what to look for in
> such a situation."
>
Then why don't you act on that advice?
That advice has been given to you several times by various people.
>
> Why continue to ask questions for which answers have already been provided?
>
Why do you continue to ignore the advice that you have been
given? And keep asking the same questions?
>
> >>> Find them with the help of debug options
> >>> such as subscript bound errors, substring errors, and the like.
>
> >> The WRITE statements showed that my subscripts are within bounds.
>
> > That's not the same as enabling subscript checks and all other checks.
>
> It is the same as checking for subscript bound errors.
>
But your kind of error can be caused by many things besides
subscript errors.
>
> >> Substrings are not involved.
>
> >>>>>> The values were within range and
> >>>>>> the program worked properly with the WRITE statement enabled. Comment
> >>>>>> out the WRITE statement, and the misbehavior returned. Hence the
> >>>>>> frustration.
>
> >>>>>>> Turn on subscript bounds checking (if available).
> >>>>>>> And while you're at it, turn on all checks.
> >>>>>> Easier said than done, as the program was built from dozens of separately
> >>>>>> written and compiled subprograms. It has not always been obvious to me
> >>>>>> whether a compilation flag needs to be applied to every single subprogram
> >>>>>> in order to accomplish a goal.
>
> >>>>> Obviously, program checking needs to be specified for every separate compilation,
> >>>>> as the apparent overwriting could be in any one or more subroutine and function.
>
> >>>> So, you're saying that if my program has subprograms A, B, C, D, and E,
> >>>> and the crash occurs while it's executing subprogram C, the apparent
> >>>> overwriting could be in A, B, D, or E?

On Sunday, July 17, 2022 at 7:57:56 PM UTC-4, Dave Tholen wrote:

> ..
> Are you suggesting that fortran-lang.discourse.group is for insight and that
> comp.lang.fortran is for open-ended discussions?

@Dave Tholen,

You have to know the primarily text-based platform of comp.lang.fortran and the intricacies with Google Groups and Usenet readers, etc. becomes challenging to a certain crucial set of Fortranners who have backgrounds in certain tools and techniques toward Fortran. This group now volunteers their time at a far greater rate at the Fortran Discourse site where one can also post Fortran code with syntax highlighting, insert images, etc. The latter facilities can be "worth more than a thousand words". Hence the suggestion to you to also consider the Fortran Discourse site. The text-driven nature of comp.lang.fortran does appear to me to lead toward open-ended discussions around Fortran's past, previous hardware, etc.

> > And consider posting there first *your own small example* that illustrates what you are doing in your code.
> You're assuming that the problem can be reproduced with a "small example"..
> The program isn't small. If the simple addition of a WRITE statement can trigger correct behavior ..

You write in your original post, "I have a different program that didn't behave as expected, and that misbehavior was totally repeatable. In an attempt to debug the program, I added a WRITE statement to check on the value of a variable during execution. However, once the WRITE statement was added, the program started behaving properly, repeatably. Comment out the added WRITE statement, and the program once again misbehaves, repeatedly. Re-enable the WRITE statement, and everything is once again hunky-dory. Damned frustrating. It's too easy to blame the optimizer. Anybody have any generic advice on what to look for in such a situation?"

Based on my own experience, a bug in code is what I would investigate thoroughly in such circumstances. Though a compiler problem cannot be discarded, I personally would only start looking toward the compiler after I have exhausted my analysis of the code. Chances are high that you will need to share more details if you would like further advice from readers.

>>>>>>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
>>>>>>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>>>>>>>>>> of IOSTAT to 5001. I discovered this situation after having written a
>>>>>>>>>> program expecting subsequent READs to all return a negative number, and
>>>>>>>>>> of course the program didn't behave as expected until I investigated the
>>>>>>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
>>>>>>>>>> set to something other than -1 on any READ attempt after the end-of-file
>>>>>>>>>> condition has been triggered. Isn't the file pointer still just beyond
>>>>>>>>>> the last record in the file, even after the first failed READ?

>>>>>>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
>>>>>>>>> You are not expected to continue reading the file.

>>>>>>>> I actually do test for IOSTAT and deal with the end-of-file condition by
>>>>>>>> exiting the DO loop with the READ statement. But once the DO loop has
>>>>>>>> been exited, the program is in an outer DO loop, which can also read the
>>>>>>>> file.

>>>>>>> You need to fix the logic.

>>>>>> Apparently I haven't been clear.
>>>>>> The program has already been fixed.

>>>>> But it hasn't been fixed.

>>>> Incorrect; reread my previous statement.

>>>>> You admitted it.

>>>> Incorrect.

>>>>> You said that after detecting end of file using IOSTAT, you exited the loop,
>>>>> and the outer loop then proceeded to execute READs on the same file.

>>>> That was before the fix. Now it is using a named DO loop, and when
>>>> the data-reading DO loop encounters the end-of-file condition, it
>>>> exits the outer DO loop, so the outer DO loop's READ statement is
>>>> not encountered. That made the value of IOSTAT in the outer DO loop
>>>> irrelevant. However, that I succeeded in making my program work
>>>> properly did not change my curiosity as to why the value of IOSTAT
>>>> changed upon the second end-of-file READ.

>>>>>> It was fixed before I even posted on this forum. The question is not
>>>>>> how to fix the problem. The question is why the value of IOSTAT was
>>>>>> changed.

>>>>> That has already been explained to you.

>>>> Where? The closest anyone has come is Steve Lionel, who said he could
>>>> make an argument for the second instance not being -1, but he didn't
>>>> present that argument. That ifort returns -1 upon a second READ
>>>> demonstrates that the situation is "processor dependent".

>>>>>> There must be some rationale.

>>>>>>>> It also tests IOSTAT to deal with the end-of-file condition,

>>>>>>> The program has already hit the end of file.
>>>>>>> Now you are trying to read read the same file at the same place.

>>>>>> That's just it: same file, same place, same action, yet a different
>>>>>> IOSTAT result. Not what I had expected.

>>>>> Again, that has been explained to you.

>>>> Again, where?

>>>>>>>> but
>>>>>>>> it tested for a negative number, which IOSTAT was no longer set to.

>>>>>>> You need to avoid reading after end of file has been detected.

>>>>>> Why? IOSTAT was added so that a program could handle error conditions.

>>> IOSTAT was added so that end-of-file could be detected and dealt with.
>>> IOSTAT was also added so that error conditions could be detected.
>>> [Note that end-of-file is not considered to be an error condition.]

>> But another READ past the end-of-file apparently is considered to be an
>> error condition by gfortran, whereas ifort apparently considers it to be
>> an end-of-file condition.

>>>>> Provided that you deal with the condition and do not do silly things like reading
>>>>> the same file again, after detecting end-of file.

>>>> Hardly silly, given that reading the same file again just triggers yet another
>>>> error condition,

>>> No it doesn't. Reading the same file again after end of file has been detected
>>> is the first error condition raised.

>> You don't know that. In fact, if you read elsewhere in this thread, you'd
>> know that the data file uses the characters "null" in place of a missing
>> numerical value, which triggers multiple error conditions long before a
>> second end-of-file READ occurs. It's hardly the first error condition
>> raised.

>>> IOSTAT returns a negative value for end of file.
>>> IOSTAT returns a positive integer for an error condition.

>> Yet ifort returns -1 for a second end-of-file READ while gfortran returns
>> 5001. Are you arguing that one of the two compilers is not standard
>> compliant?

>>>> and IOSTAT was designed to allow programs to deal with the
>>>> error condition.

>>>>>> My program was designed to deal with the error condition. Had gfortran
>>>>>> consistently returned a negative IOSTAT for every attempt to READ past
>>>>>> the end-of-file, my program would have worked exactly as intended.

>>>>> Listen up. Your program is in error. Fix it.

>>>> Listen up: you are incorrect.

>>> I am not incorrect. Read the standard.

>> You are incorrect. My program is standard-conforming and works properly.

> Since you have not read the standard, you do not know whether your program
> is "standard conforming".

You wrote "the" standard as if there has been only one. In reality, there
have been multiple Fortran standards. I have a hardcopy of one of the
earlier standards and have read it. I have reference manuals to multiple
more recent compilers and I have read them as well. Since you do not know
the standard to which my program was written, and you also don't know which
of the standards or reference manuals I have read, you're not in a position
to make the claim that you did.

>>>> My program *WAS* incompatible with gfortran's
>>>> handling of a second end-of-file READ.

>>> No, your program was wrong.

>> Looks like you've finally admitted (tacitly) that you had your tenses wrong.
>> Given that the different compilers return IOSTAT values with different signs
>> for the same second end-of-file READ, it can be argued that the situation is
>> "processor dependent".

> All the IOSTAT values are processor-dependent (except for zero).
> However, the value returned for end-of-file is negative.
> The value for an error condition is positive.
> This infrioation is in the standard.

Irrelevant, given that my program does not test for specific processor-dependent
values; rather, it tests for negative values. The outer DO loop's IOSTAT negative
value test has been superfluous since the program now exits the outer named DO loop
when the inner DO loop encounters the end-of-file condition.

>> Forty years ago, programs that OPENed a file and
>> started READing worked just fine with compilers that set the file pointer to
>> the beginning of the file, until they were moved to a BSD UNIX system whose
>> compiler, by default, OPENed a file with the pointer at the end of the file.
>> The program wasn't "wrong"; rather, it was incompatible with a processor
>> dependency.

>>> Your program was in error when you tried to read the file again,
>>> after receiving end-of-file notification.

>> Rather, an error condition was triggered when the program tried to read
>> the file again,

> And because your program contains an error, it is not standard conforming.

My program does not contain an error. The version that existed prior to me
even starting this thread contained an incompatibility with a gfortran
processor-dependent feature. The current version is compatible with
gfortran.

>> but the code included a test on the value of IOSTAT to
>> deal with the condition.

>>> That's why you received a positive number for IOSTAT when you tried
>>> to read the file after receiving end-of-file.

>> Yet ifort returns a negative number. Are you arguing that ifort is in
>> error?

On 7/17/22 10:22 PM, gah4 wrote:
> On Sunday, July 17, 2022 at 11:48:12 AM UTC-7, FortranFan wrote:
>
> (snip)
>
>> "If an end-of-file condition occurs during execution of an input/output statement that contains either an END= specifier or an IOSTAT= specifier, and an error condition does not occur then:
>
>> (1) processing of the input list, if any, terminates;
>> (2) if the statement is a data transfer statement or the end-of-file condition occurs during a wait operation,
>> all do-variables in the statement that initiated the transfer become undefined;
>> (3) if the statement is an input statement or the end-of-file condition occurs during a wait operation
>> for a transfer initiated by an input statement, all effective items resulting from the expansion of list
>> items or the namelist group in the statement that initiated the transfer become undefined;
>> (4) if the file specified in the input statement is an external record file, it is positioned after the endfile record;"
>
> Those are useful, but there are also ones that apply for I/O errors.
>
> I believe, like (3), in the case of I/O error that all the I/O list elements are undefined.
>
> For comparison, in C all I/O elements before EOF or any other condition occurs are
> not undefined, but have the appropriate value. That can be very convenient.
>
>
There isn't much that can be done in recovering from I/O errors.

read(unit, IOSTAT = JJJ, ...) N, A(1,N), N, A(7,N), N, B(1,N)

There is no possible way to know what array elements are undefined if
jjj is non-zero.

And, if a magnetic tape goes into "stretch and then break" mode there's
no reason to think that the last few values read in are any good.

Dick Hendrickson

On 7/20/2022 5:00 AM, Dave Tholen wrote:
> <snip>

> The use of WRITE statements to check on the values of variables has
> served
> me well for debugging purposes for decades.  However, when the
> addition of a
> WRITE statement makes a problem go away, one wonders how that can happen.

Arguing with Robin isn't going to help you fix the problem.

Wondering what might have gone wrong isn't going to help.

There is a path to a solution, painful as it might be:

1. Compile your code for debug. If the problem persists, that's a good
thing. If the problem goes away, you'll have to debug whatever version
fails.

2. Find a debugger. Even a command-line debugger will do. Step through
the code. Set breakpoints. Set watchpoints. Follow the leads, the
hunches, the blind alleys. This will be time-consuming and tedious. You
will hate your life. Everyone around you will hate your life. They may
even hate you. Don't give up.

3. If the code and the data are suitable for public viewing, and if
packaging everything so someone else can have a go at it is less painful
than step 2 above, then do that. There are people out there with the
requisite knowledge, experience and high tolerance for pain. When
someone figures it out, we'll all learn something.

Louis

On 7/23/22 9:18 PM, Louis Krupp wrote:
> On 7/20/2022 5:00 AM, Dave Tholen wrote:
>> <snip>
>
>> The use of WRITE statements to check on the values of variables has
>> served
>> me well for debugging purposes for decades.  However, when the
>> addition of a
>> WRITE statement makes a problem go away, one wonders how that can happen.
>
> Arguing with Robin isn't going to help you fix the problem.
>
> Wondering what might have gone wrong isn't going to help.
>
> There is a path to a solution, painful as it might be:
>
> 1. Compile your code for debug. If the problem persists, that's a good
> thing. If the problem goes away, you'll have to debug whatever version
> fails.
>
> 2. Find a debugger. Even a command-line debugger will do. Step through
> the code. Set breakpoints. Set watchpoints. Follow the leads, the
> hunches, the blind alleys. This will be time-consuming and tedious. You
> will hate your life. Everyone around you will hate your life. They may
> even hate you. Don't give up.
>
> 3. If the code and the data are suitable for public viewing, and if
> packaging everything so someone else can have a go at it is less painful
> than step 2 above, then do that. There are people out there with the
> requisite knowledge, experience and high tolerance for pain. When
> someone figures it out, we'll all learn something.
>
> Louis

Here is another item to add to the list.

4. When you finally find the bug, it will be obvious, and you will
wonder why on earth you didn't see it immediately.

Dave Tholen <tholen@antispam.ham> schrieb:

> However, when the addition of a WRITE statement makes a problem
> go away, one wonders how that can happen.

That one is actually a textbook symptom of uninitialized variables
and/or buffer overruns. Adding a print statement will change the
memory layout of the program and the values of some stack variables
in unpredictable ways.

If you have that problem, try running your code on every compiler
you have with every debugging option turned on (including, but
not limited to, the -fsanitize options and the -finit-integer=
with gfortran and, if you have it, nagfor's -C=check=all and
-Ccheck=undefined), or run your code under valgrind.

On Wednesday, July 20, 2022 at 9:00:10 PM UTC+10, Dave Tholen wrote:
>>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
>>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>>>>> of IOSTAT to 5001. I discovered this situation after having written a
>>>>> program expecting subsequent READs to all return a negative number, and
>>>>> of course the program didn't behave as expected until I investigated the
>>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
>>>>> set to something other than -1 on any READ attempt after the end-of-file
>>>>> condition has been triggered. Isn't the file pointer still just beyond
>>>>> the last record in the file, even after the first failed READ?

>>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
>>>> You are not expected to continue reading the file.

>>> I actually do test for IOSTAT and deal with the end-of-file condition by
>>> exiting the DO loop with the READ statement. But once the DO loop has
>>> been exited, the program is in an outer DO loop, which can also read the
>>> file.

>> You need to fix the logic.

> Apparently I haven't been clear.
> The program has already been fixed.

It had NOT been fixed when you wrote the first paragraph above.
You wrote: "But once the DO loop has been exited, the program is in
an outer DO loop, which can also read the file."

>>>> The closest anyone has come is Steve Lionel, who said he could
>>>> make an argument for the second instance not being -1, but he didn't
>>>> present that argument. That ifort returns -1 upon a second READ
>>>> demonstrates that the situation is "processor dependent".

The situation is NOT "processor dependent". The actual numerical value
is processor dependent", but the sign of the value is NOT. The sign
is required to be positive. ifort is not standard-conforming.

> You wrote "the" standard as if there has been only one.

There have been a number of standards published.
All say the same thing about IOSTAT since the keyword
was introduced in FORTRAN 77 -- in regard to the
value returned for no errors (zero), end of file (a negative
value) and an error (a positive value).

> In reality, there
> have been multiple Fortran standards. I have a hardcopy of one of the
> earlier standards and have read it. I have reference manuals to multiple
> more recent compilers and I have read them as well. Since you do not know
> the standard to which my program was written, and you also don't know which
> of the standards or reference manuals I have read, you're not in a position
> to make the claim that you did.

What you say is irrelevant with regard to IOSTAT.

>>>> My program *WAS* incompatible with gfortran's
>>>> handling of a second end-of-file READ.

>>> No, your program was wrong.

>> Looks like you've finally admitted (tacitly) that you had your tenses wrong.
>> Given that the different compilers return IOSTAT values with different signs
>> for the same second end-of-file READ, it can be argued that the situation is
>> "processor dependent".

> All the IOSTAT values are processor-dependent (except for zero).
> However, the value returned for end-of-file is negative.
> The value for an error condition is positive.
> This information is in the standard.

> Irrelevant,

It's very relevant, since you were testing for an IOSTAT value
of the wrong sign.

>>> given that my program does not test for specific processor-dependent
>>> values; rather, it tests for negative values. The outer DO loop's IOSTAT negative
>>> value test has been superfluous since the program now exits the outer named DO loop
>>> when the inner DO loop encounters the end-of-file condition.

>>> Forty years ago, programs that OPENed a file and
>>> started READing worked just fine with compilers that set the file pointer to
>>> the beginning of the file, until they were moved to a BSD UNIX system whose
>>> compiler, by default, OPENed a file with the pointer at the end of the file.
>>> The program wasn't "wrong"; rather, it was incompatible with a processor
>>> dependency.

>> Your program was in error when you tried to read the file again,
>> after receiving end-of-file notification.

>> Rather, an error condition was triggered when the program tried to read
>> the file again,

>> And because your program contains an error, it is not standard conforming.

> My program does not contain an error. The version that existed prior to me
> even starting this thread contained an incompatibility with a gfortran
> processor-dependent feature.
..
Your program contained an error. It first tested for a negative
value indicating end-of-file, and then attempted to read another
record that caused gfortran to return a positive value, indicating
an error condition. Your program expected another negative value,
instead of the positive vaue that was require by the Standard.
..
>>>> The current version is compatible with gfortran.

>>> but the code included a test on the value of IOSTAT to
>>> deal with the condition.

>>> That's why you received a positive number for IOSTAT when you tried
>>> to read the file after receiving end-of-file.

>> Yet ifort returns a negative number. Are you arguing that ifort is in
>> error?

> ifort is in error.

> File a bug report for ifort,

No, if you don't like it, you file a bug report with ifort.

> if you think it is in error. Don't be surprised
> if the response is that when you read past the end-of-file, it is an
> end-of-file condition, regardless of the number of times you do it.

You still have not read the standard. The Standard requires that
a Fortran compiler to return a positive value when a program attempts
to read past the end of file.

> You still have not read the standard.

>> You are still incorrect. There are all sorts of situations that the standard
>> writers did not anticipate when the standards were written, hence the need for
>> "interpretations" from the standards committee.

There is no need for an "interpretation". The standard is clear and unequivocal.
The standard has been around for 40 years.
..
>>>>> Find them with the help of debug options
>>>>> such as subscript bound errors, substring errors, and the like.

>>>> The WRITE statements showed that my subscripts are within bounds.

>>> That's not the same as enabling subscript checks and all other checks.

>> It is the same as checking for subscript bound errors.

No it's not. You could have missed printing one critial value,
or not realized that a subscript is out-of-range.

> But your kind of error can be caused by many things besides
> subscript errors.

> My WRITE statements were not limited to the display of only subscript values.

That's irrelevant.
> The use of WRITE statements to check on the values of variables has served
> me well for debugging purposes for decades. However, when the addition of a
> WRITE statement makes a problem go away, one wonders how that can happen.

I repeat: your kind of error can be caused by many things
besides subscript errors. You need to specify options that
check for ALL errors, including subscript errors.

On 7/25/22 1:03 AM, Robin Vowels wrote:
> The situation is NOT "processor dependent". The actual numerical value
> is processor dependent", but the sign of the value is NOT. The sign
> is required to be positive. ifort is not standard-conforming.

Can you quote the standard, any version of the standard, that specifies
this? I do not think it is true for the reasons I explained previously
in this thread. If this were true, then there would be no way to read
multiple files from a magnetic tape. Yet at one time, that was a
routine, everyday, occurrence for fortran programmers.

$.02 -Ron Shepard

On Monday, July 25, 2022 at 8:33:04 AM UTC-7, Ron Shepard wrote:
> On 7/25/22 1:03 AM, Robin Vowels wrote:
> > The situation is NOT "processor dependent". The actual numerical value
> > is processor dependent", but the sign of the value is NOT. The sign
> > is required to be positive. ifort is not standard-conforming.
> Can you quote the standard, any version of the standard, that specifies
> this? I do not think it is true for the reasons I explained previously
> in this thread. If this were true, then there would be no way to read
> multiple files from a magnetic tape. Yet at one time, that was a
> routine, everyday, occurrence for fortran programmers.
>
> $.02 -Ron Shepard