If you read (sequentially) past the end-of-file, gfortran sets the value
of IOSTAT to -1, but if you read a second time, gfortran sets the value
of IOSTAT to 5001. I discovered this situation after having written a
program expecting subsequent READs to all return a negative number, and
of course the program didn't behave as expected until I investigated the
problem. So I'm curious as to why the choice was made to have IOSTAT
set to something other than -1 on any READ attempt after the end-of-file
condition has been triggered. Isn't the file pointer still just beyond
the last record in the file, even after the first failed READ?

On a completely separate matter, I have a different program that
didn't behave as expected, and that misbehavior was totally repeatable.
In an attempt to debug the program, I added a WRITE statement to check
on the value of a variable during execution. However, once the WRITE
statement was added, the program started behaving properly, repeatably.
Comment out the added WRITE statement, and the program once again
misbehaves, repeatedly. Re-enable the WRITE statement, and everything
is once again hunky-dory. Damned frustrating. It's too easy to blame
the optimizer. Anybody have any generic advice on what to look for in
such a situation?

On Friday, July 15, 2022 at 9:47:37 PM UTC+10, Dave Tholen wrote:
> If you read (sequentially) past the end-of-file, gfortran sets the value
> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> of IOSTAT to 5001. I discovered this situation after having written a
> program expecting subsequent READs to all return a negative number, and
> of course the program didn't behave as expected until I investigated the
> problem. So I'm curious as to why the choice was made to have IOSTAT
> set to something other than -1 on any READ attempt after the end-of-file
> condition has been triggered. Isn't the file pointer still just beyond
> the last record in the file, even after the first failed READ?

You are expected to test for IOSTAT and deal with the end-of-file condition.
You are not expected to continue reading the file.
>
>
> On a completely separate matter, I have a different program that
> didn't behave as expected, and that misbehavior was totally repeatable.
> In an attempt to debug the program, I added a WRITE statement to check
> on the value of a variable during execution. However, once the WRITE
> statement was added, the program started behaving properly, repeatably.

That suggests that there is a bug in your program. Possibly something has
been overwritten.
Turn on subscript bounds checking (if available).
And while you're at it, turn on all checks.

> Comment out the added WRITE statement, and the program once again
> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
> is once again hunky-dory. Damned frustrating. It's too easy to blame
> the optimizer. Anybody have any generic advice on what to look for in
> such a situation?

Dave Tholen wrote:

> If you read (sequentially) past the end-of-file, gfortran sets the value
> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> of IOSTAT to 5001. I discovered this situation after having written a
> program expecting subsequent READs to all return a negative number, and
> of course the program didn't behave as expected until I investigated the
> problem. So I'm curious as to why the choice was made to have IOSTAT
> set to something other than -1 on any READ attempt after the end-of-file
> condition has been triggered. Isn't the file pointer still just beyond
> the last record in the file, even after the first failed READ?

What version of the compiler and operating system?

What does IOMSG tell you? Likely, you have hit two different errors.

> On a completely separate matter, I have a different program that
> didn't behave as expected, and that misbehavior was totally repeatable.
> In an attempt to debug the program, I added a WRITE statement to check
> on the value of a variable during execution. However, once the WRITE
> statement was added, the program started behaving properly, repeatably.
> Comment out the added WRITE statement, and the program once again
> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
> is once again hunky-dory. Damned frustrating. It's too easy to blame
> the optimizer. Anybody have any generic advice on what to look for in
> such a situation?

Remove the write statement and compile with -Wall -fcheck=all. This
might find where you are stomping on memory.

--
steve

On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
> If you read (sequentially) past the end-of-file, gfortran sets the value
> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> of IOSTAT to 5001. I discovered this situation after having written a
> program expecting subsequent READs to all return a negative number, and
> of course the program didn't behave as expected until I investigated the
> problem. So I'm curious as to why the choice was made to have IOSTAT
> set to something other than -1 on any READ attempt after the end-of-file
> condition has been triggered. Isn't the file pointer still just beyond
> the last record in the file, even after the first failed READ?

I haven't thought about this in a lot of years, but in the days of
magnetic tapes, some systems allowed one to read the EOF,
and then go on to the next file on the tape.

Though Fortran 66 has no method for detecting EOF, you are just
supposed to stop reading before it happens.

I believe for tapes in Unix, when you get to the end, the system
call return 0 bytes, as the sign of the EOF, and then you go onto
the next file. (That is, past the tape mark.)

Disk files don't do that, though.

On Fri, 15 Jul 2022 07:45:56 -0700, Robin Vowels wrote:

> You are expected to test for IOSTAT and deal with the end-of-file
> condition.
> You are not expected to continue reading the file.

MIL-STD-1753 specified that you could, when reading magnetic tape (see
https://stevelionel.com/drfortran/2020/05/16/doctor-fortran-in-military-
strength/).

The Fortran standard says (12.11.1), that when an "endfile record is
encountered during the reading of a file connected for sequential access",
"if the file specified in the input statement is an external record file,
it is positioned after the endfile record;" (12.11.3). What happens after
that is not specified, though you are allowed to BACKSPACE over the
endfile record. Note that there doesn't need to be any physical
representation of an endfile record.

gfortran is within its rights to give you a different IOSTAT value in this
case, and I could make an argument that -1 is NOT the correct thing to
return here. I tried ifort and nagfor - ifort returns -1 on the read past
the EOF, nagfor uses 210, corresponding to "READ/WRITE attempted after
ENDFILE on unit 1". (I can't get too worked up over ifort returning -1,
however...)

--
Steve Lionel
ISO/IEC JTC1/SC22/WG5 (Fortran) Convenor
Retired Intel Fortran developer/support
Email: firstname at firstnamelastname dot com
Twitter: @DoctorFortran
LinkedIn: https://www.linkedin.com/in/stevelionel
Blog: https://stevelionel.com/drfortran
WG5: https://wg5-fortran.org

>> If you read (sequentially) past the end-of-file, gfortran sets the value
>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>> of IOSTAT to 5001. I discovered this situation after having written a
>> program expecting subsequent READs to all return a negative number, and
>> of course the program didn't behave as expected until I investigated the
>> problem. So I'm curious as to why the choice was made to have IOSTAT
>> set to something other than -1 on any READ attempt after the end-of-file
>> condition has been triggered. Isn't the file pointer still just beyond
>> the last record in the file, even after the first failed READ?

> You are expected to test for IOSTAT and deal with the end-of-file condition.
> You are not expected to continue reading the file.

I actually do test for IOSTAT and deal with the end-of-file condition by
exiting the DO loop with the READ statement. But once the DO loop has
been exited, the program is in an outer DO loop, which can also read the
file. It also tests IOSTAT to deal with the end-of-file condition, but
it tested for a negative number, which IOSTAT was no longer set to.

And if you must know why the nested DO loops, one loop dealt with data
while the other loop dealt with metadata. It was easily fixable. My
question here was not about how to fix the problem, but rather to find
out the rationale for changing the value of IOSTAT in such a situation.
My expectation was that IOSTAT would remain negative. My expectation
was wrong, hence the curiosity.

>> On a completely separate matter, I have a different program that
>> didn't behave as expected, and that misbehavior was totally repeatable.
>> In an attempt to debug the program, I added a WRITE statement to check
>> on the value of a variable during execution. However, once the WRITE
>> statement was added, the program started behaving properly, repeatably.

> That suggests that there is a bug in your program.

Brilliant.

> Possibly something has
> been overwritten.

I already thought about that, hence the WRITE statement to examine
the values of the array indices. The values were within range and
the program worked properly with the WRITE statement enabled. Comment
out the WRITE statement, and the misbehavior returned. Hence the
frustration.

> Turn on subscript bounds checking (if available).
> And while you're at it, turn on all checks.

Easier said than done, as the program was built from dozens of separately
written and compiled subprograms. It has not always been obvious to me
whether a compilation flag needs to be applied to every single subprogram
in order to accomplish a goal.

>> Comment out the added WRITE statement, and the program once again
>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
>> is once again hunky-dory. Damned frustrating. It's too easy to blame
>> the optimizer. Anybody have any generic advice on what to look for in
>> such a situation?

>> If you read (sequentially) past the end-of-file, gfortran sets the value
>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>> of IOSTAT to 5001. I discovered this situation after having written a
>> program expecting subsequent READs to all return a negative number, and
>> of course the program didn't behave as expected until I investigated the
>> problem. So I'm curious as to why the choice was made to have IOSTAT
>> set to something other than -1 on any READ attempt after the end-of-file
>> condition has been triggered. Isn't the file pointer still just beyond
>> the last record in the file, even after the first failed READ?

> What version of the compiler and operating system?

Unfortunately, I'm not sitting in front of the computer in question at
the moment, but I am sitting in front of a different computer that was
purchased at the same time and loaded with the same software, so there's
a good chance it's gfortran 7.1.0. The operating system is Windows 10 Pro
version 21H2. I'm expecting Windows Update to reboot the system any day
now to install the July patches (as happened last night to the system
I'm using at the moment).

> What does IOMSG tell you?

I'm unfamiliar with IOMSG. Sounds like an intrinsic that returns a
text description of an IOSTAT numerical value, but I don't see it listed
among the intrinsic procedures for gfortran. Could you elaborate?

> Likely, you have hit two different errors.

Yeah, the first being end-of-file and the second being past-end-of-file.
As I noted in an earlier response, my expectation was that both would
return negative numbers, but that expectation was wrong, at least for
gfortran. I would have been just fine with ifort, thanks to Steve
Lionel's test.

>> On a completely separate matter, I have a different program that
>> didn't behave as expected, and that misbehavior was totally repeatable.
>> In an attempt to debug the program, I added a WRITE statement to check
>> on the value of a variable during execution. However, once the WRITE
>> statement was added, the program started behaving properly, repeatably.
>> Comment out the added WRITE statement, and the program once again
>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
>> is once again hunky-dory. Damned frustrating. It's too easy to blame
>> the optimizer. Anybody have any generic advice on what to look for in
>> such a situation?

> Remove the write statement and compile with -Wall -fcheck=all. This
> might find where you are stomping on memory.

I'll give that a try over the weekend.

On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
> If you read (sequentially) past the end-of-file, gfortran sets the value
> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> of IOSTAT to 5001. I discovered this situation after having written a
> program expecting subsequent READs to all return a negative number, and
> of course the program didn't behave as expected until I investigated the
> problem. So I'm curious as to why the choice was made to have IOSTAT
> set to something other than -1 on any READ attempt after the end-of-file
> condition has been triggered. Isn't the file pointer still just beyond
> the last record in the file, even after the first failed READ?

If I understand it right, you should test for not zero, and act accordingly.

It is negative for EOF, but positive for I/O errors, which most likely you
should also exit your loop, and/or handle appropriately.

Otherwise, it makes some sense. In the first case, there is actual EOF.
In the second, reading after EOF, is an I/O error, and so reported.

Unless you plan to test for and take appropriate action for different
I/O errors, taking action for any non-zero value makes sense.

I believe this is a not unusual error in some other languages,
which I won't mention.

On Saturday, July 16, 2022 at 2:04:18 PM UTC+12, gah4 wrote:
> On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
> > If you read (sequentially) past the end-of-file, gfortran sets the value
> > of IOSTAT to -1, but if you read a second time, gfortran sets the value
> > of IOSTAT to 5001. I discovered this situation after having written a
> > program expecting subsequent READs to all return a negative number, and
> > of course the program didn't behave as expected until I investigated the
> > problem. So I'm curious as to why the choice was made to have IOSTAT
> > set to something other than -1 on any READ attempt after the end-of-file
> > condition has been triggered. Isn't the file pointer still just beyond
> > the last record in the file, even after the first failed READ?
> If I understand it right, you should test for not zero, and act accordingly.
>
> It is negative for EOF, but positive for I/O errors, which most likely you
> should also exit your loop, and/or handle appropriately.
>
> Otherwise, it makes some sense. In the first case, there is actual EOF.
> In the second, reading after EOF, is an I/O error, and so reported.
>
> Unless you plan to test for and take appropriate action for different
> I/O errors, taking action for any non-zero value makes sense.
>
> I believe this is a not unusual error in some other languages,
> which I won't mention.

In an open, inquire, read or write statement you may include iomsg=msg as well as iostat=ios if you have a scalar variable of type character that's long enough for the error messge, and you write msg if ios was nonzero. The length of msg will depend on what compiler you used, what ios was, and how long the name of the offending file was. Iomsg and iostat are not intrinsics but specifiers in those 4 kinds of statement.

On Saturday, July 16, 2022 at 6:33:31 AM UTC+10, Steve Lionel wrote:
> On Fri, 15 Jul 2022 07:45:56 -0700, Robin Vowels wrote:
>
> > You are expected to test for IOSTAT and deal with the end-of-file
> > condition.
> > You are not expected to continue reading the file.
> MIL-STD-1753 specified that you could, when reading magnetic tape
..
That's irrelevant.

> (see
> https://stevelionel.com/drfortran/2020/05/16/doctor-fortran-in-military-
> strength/).
> The Fortran standard says (12.11.1), that when an "endfile record is
> encountered during the reading of a file connected for sequential access",
> "if the file specified in the input statement is an external record file,
> it is positioned after the endfile record;" (12.11.3). What happens after
> that is not specified, though you are allowed to BACKSPACE over the
> endfile record. Note that there doesn't need to be any physical
> representation of an endfile record.
>
> gfortran is within its rights to give you a different IOSTAT value in this
> case, and I could make an argument that -1 is NOT the correct thing to
> return here. I tried ifort and nagfor - ifort returns -1 on the read past
> the EOF, nagfor uses 210, corresponding to "READ/WRITE attempted after
> ENDFILE on unit 1". (I can't get too worked up over ifort returning -1,
> however...)

On Saturday, July 16, 2022 at 11:30:41 AM UTC+10, Dave Tholen wrote:
> >> If you read (sequentially) past the end-of-file, gfortran sets the value
> >> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> >> of IOSTAT to 5001. I discovered this situation after having written a
> >> program expecting subsequent READs to all return a negative number, and
> >> of course the program didn't behave as expected until I investigated the
> >> problem. So I'm curious as to why the choice was made to have IOSTAT
> >> set to something other than -1 on any READ attempt after the end-of-file
> >> condition has been triggered. Isn't the file pointer still just beyond
> >> the last record in the file, even after the first failed READ?
>
> > You are expected to test for IOSTAT and deal with the end-of-file condition.
> > You are not expected to continue reading the file.
..
> I actually do test for IOSTAT and deal with the end-of-file condition by
> exiting the DO loop with the READ statement. But once the DO loop has
> been exited, the program is in an outer DO loop, which can also read the
> file.
..
You need to fix the logic.
..
> It also tests IOSTAT to deal with the end-of-file condition,
..
The program has already hit the end of file.
Now you are trying to read read the same file at the same place.
..
> but
> it tested for a negative number, which IOSTAT was no longer set to.
..
You need to avoid reading after end of file has been detected.
..
> And if you must know why the nested DO loops, one loop dealt with data
> while the other loop dealt with metadata. It was easily fixable. My
> question here was not about how to fix the problem, but rather to find
> out the rationale for changing the value of IOSTAT in such a situation.
> My expectation was that IOSTAT would remain negative. My expectation
> was wrong, hence the curiosity.
> >> On a completely separate matter, I have a different program that
> >> didn't behave as expected, and that misbehavior was totally repeatable.
> >> In an attempt to debug the program, I added a WRITE statement to check
> >> on the value of a variable during execution. However, once the WRITE
> >> statement was added, the program started behaving properly, repeatably.
>
> > That suggests that there is a bug in your program.
> Brilliant.
> > Possibly something has
> > been overwritten.
> I already thought about that, hence the WRITE statement to examine
> the values of the array indices.
..
In every program unit, subroutine and function in the entire program?
..
> The values were within range and
> the program worked properly with the WRITE statement enabled. Comment
> out the WRITE statement, and the misbehavior returned. Hence the
> frustration.
> > Turn on subscript bounds checking (if available).
> > And while you're at it, turn on all checks.
..
> Easier said than done, as the program was built from dozens of separately
> written and compiled subprograms. It has not always been obvious to me
> whether a compilation flag needs to be applied to every single subprogram
> in order to accomplish a goal.
..
Obviously, program checking needs to be specified for every separate compilation,
as the apparent overwriting could be in any one or more subroutine and function.
..
> >> Comment out the added WRITE statement, and the program once again
> >> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
> >> is once again hunky-dory. Damned frustrating. It's too easy to blame
> >> the optimizer. Anybody have any generic advice on what to look for in
> >> such a situation?

On Saturday, July 16, 2022 at 6:20:07 AM UTC+10, gah4 wrote:
> On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
> > If you read (sequentially) past the end-of-file, gfortran sets the value
> > of IOSTAT to -1, but if you read a second time, gfortran sets the value
> > of IOSTAT to 5001. I discovered this situation after having written a
> > program expecting subsequent READs to all return a negative number, and
> > of course the program didn't behave as expected until I investigated the
> > problem. So I'm curious as to why the choice was made to have IOSTAT
> > set to something other than -1 on any READ attempt after the end-of-file
> > condition has been triggered. Isn't the file pointer still just beyond
> > the last record in the file, even after the first failed READ?
> I haven't thought about this in a lot of years, but in the days of
> magnetic tapes, some systems allowed one to read the EOF,
> and then go on to the next file on the tape.
>
> Though Fortran 66 has no method for detecting EOF, you are just
> supposed to stop reading before it happens.
..
IBM FORTRAN IV of 1966 had EOF and ERR options in the
..
> I believe for tapes in Unix, when you get to the end, the system
> call return 0 bytes, as the sign of the EOF, and then you go onto
> the next file. (That is, past the tape mark.)
>
> Disk files don't do that, though.

On Saturday, July 16, 2022 at 6:20:07 AM UTC+10, gah4 wrote:
> On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
> > If you read (sequentially) past the end-of-file, gfortran sets the value
> > of IOSTAT to -1, but if you read a second time, gfortran sets the value
> > of IOSTAT to 5001. I discovered this situation after having written a
> > program expecting subsequent READs to all return a negative number, and
> > of course the program didn't behave as expected until I investigated the
> > problem. So I'm curious as to why the choice was made to have IOSTAT
> > set to something other than -1 on any READ attempt after the end-of-file
> > condition has been triggered. Isn't the file pointer still just beyond
> > the last record in the file, even after the first failed READ?
> I haven't thought about this in a lot of years, but in the days of
> magnetic tapes, some systems allowed one to read the EOF,
> and then go on to the next file on the tape.
>
> Though Fortran 66 has no method for detecting EOF, you are just
> supposed to stop reading before it happens.
..
IBM FORTRAN IV of 1966 had EOF= and ERR= options
for the READ statement.
..
PL/I (F) of 1966 raised the ENDFILE condition on attempting to
read after the final data item in the file.

On 7/15/2022 11:48 AM, steve kargl wrote:
> Dave Tholen wrote:
<snip>
>> On a completely separate matter, I have a different program that
>> didn't behave as expected, and that misbehavior was totally repeatable.
>> In an attempt to debug the program, I added a WRITE statement to check
>> on the value of a variable during execution. However, once the WRITE
>> statement was added, the program started behaving properly, repeatably.
>> Comment out the added WRITE statement, and the program once again
>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
>> is once again hunky-dory. Damned frustrating. It's too easy to blame
>> the optimizer. Anybody have any generic advice on what to look for in
>> such a situation?
> Remove the write statement and compile with -Wall -fcheck=all. This
> might find where you are stomping on memory.
>

I would add -Werror. If you always compile without warnings, your life
will be simpler.

If the presumably unlikely event that you're not already using IMPLICIT
NONE in every program unit, I would do that, too.

Among the other things mentioned, a WRITE statement might mask problems
caused by a variable that's not assigned a value before it's used.

If all else fails, the debugger is your friend. If you haven't
experienced the thrill of stepping through assembler instructions
finding where your wayward variable is stored and examining its value
and figuring out how it got to be what it is, this could be your chance.

Louis

>>>> If you read (sequentially) past the end-of-file, gfortran sets the value
>>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>>>> of IOSTAT to 5001. I discovered this situation after having written a
>>>> program expecting subsequent READs to all return a negative number, and
>>>> of course the program didn't behave as expected until I investigated the
>>>> problem. So I'm curious as to why the choice was made to have IOSTAT
>>>> set to something other than -1 on any READ attempt after the end-of-file
>>>> condition has been triggered. Isn't the file pointer still just beyond
>>>> the last record in the file, even after the first failed READ?

>>> You are expected to test for IOSTAT and deal with the end-of-file condition.
>>> You are not expected to continue reading the file.

>> I actually do test for IOSTAT and deal with the end-of-file condition by
>> exiting the DO loop with the READ statement. But once the DO loop has
>> been exited, the program is in an outer DO loop, which can also read the
>> file.

> You need to fix the logic.

Apparently I haven't been clear. The program has already been fixed.
It was fixed before I even posted on this forum. The question is not
how to fix the problem. The question is why the value of IOSTAT was
changed. There must be some rationale.

>> It also tests IOSTAT to deal with the end-of-file condition,

> The program has already hit the end of file.
> Now you are trying to read read the same file at the same place.

That's just it: same file, same place, same action, yet a different
IOSTAT result. Not what I had expected.

>> but
>> it tested for a negative number, which IOSTAT was no longer set to.

> You need to avoid reading after end of file has been detected.

Why? IOSTAT was added so that a program could handle error conditions.
My program was designed to deal with the error condition. Had gfortran
consistently returned a negative IOSTAT for every attempt to READ past
the end-of-file, my program would have worked exactly as intended. Had
I used ifort, my program would have worked exactly as intended.

>> And if you must know why the nested DO loops, one loop dealt with data
>> while the other loop dealt with metadata. It was easily fixable. My
>> question here was not about how to fix the problem, but rather to find
>> out the rationale for changing the value of IOSTAT in such a situation.
>> My expectation was that IOSTAT would remain negative. My expectation
>> was wrong, hence the curiosity.

>>>> On a completely separate matter, I have a different program that
>>>> didn't behave as expected, and that misbehavior was totally repeatable.
>>>> In an attempt to debug the program, I added a WRITE statement to check
>>>> on the value of a variable during execution. However, once the WRITE
>>>> statement was added, the program started behaving properly, repeatably.

>>> That suggests that there is a bug in your program.

>> Brilliant.

>>> Possibly something has
>>> been overwritten.

>> I already thought about that, hence the WRITE statement to examine
>> the values of the array indices.

> In every program unit, subroutine and function in the entire program?

I usually sprinkle WRITE statements around to isolate where the crash
is happening. As I narrow things down, unnecessary WRITE statements are
either removed or commented out. In this particular case, I got it down
to a single WRITE statement that could trigger proper behavior.

>> The values were within range and
>> the program worked properly with the WRITE statement enabled. Comment
>> out the WRITE statement, and the misbehavior returned. Hence the
>> frustration.

>>> Turn on subscript bounds checking (if available).
>>> And while you're at it, turn on all checks.
..
>> Easier said than done, as the program was built from dozens of separately
>> written and compiled subprograms. It has not always been obvious to me
>> whether a compilation flag needs to be applied to every single subprogram
>> in order to accomplish a goal.

> Obviously, program checking needs to be specified for every separate compilation,
> as the apparent overwriting could be in any one or more subroutine and function.

So, you're saying that if my program has subprograms A, B, C, D, and E,
and the crash occurs while it's executing subprogram C, the apparent
overwriting could be in A, B, D, or E?

>>>> Comment out the added WRITE statement, and the program once again
>>>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
>>>> is once again hunky-dory. Damned frustrating. It's too easy to blame
>>>> the optimizer. Anybody have any generic advice on what to look for in
>>>> such a situation?

>> You are expected to test for IOSTAT and deal with the end-of-file
>> condition.
>> You are not expected to continue reading the file.
>
> MIL-STD-1753 specified that you could, when reading magnetic tape (see
> https://stevelionel.com/drfortran/2020/05/16/doctor-fortran-in-military-
> strength/).
>
> The Fortran standard says (12.11.1), that when an "endfile record is
> encountered during the reading of a file connected for sequential access",
> "if the file specified in the input statement is an external record file,
> it is positioned after the endfile record;" (12.11.3). What happens after
> that is not specified, though you are allowed to BACKSPACE over the
> endfile record. Note that there doesn't need to be any physical
> representation of an endfile record.
>
> gfortran is within its rights to give you a different IOSTAT value in this
> case, and I could make an argument that -1 is NOT the correct thing to
> return here.

That argument you could make is precisely what I was hoping to get out of
this thread.

> I tried ifort and nagfor - ifort returns -1 on the read past
> the EOF, nagfor uses 210, corresponding to "READ/WRITE attempted after
> ENDFILE on unit 1". (I can't get too worked up over ifort returning -1,
> however...)

>> If you read (sequentially) past the end-of-file, gfortran sets the value
>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>> of IOSTAT to 5001. I discovered this situation after having written a
>> program expecting subsequent READs to all return a negative number, and
>> of course the program didn't behave as expected until I investigated the
>> problem. So I'm curious as to why the choice was made to have IOSTAT
>> set to something other than -1 on any READ attempt after the end-of-file
>> condition has been triggered. Isn't the file pointer still just beyond
>> the last record in the file, even after the first failed READ?

> If I understand it right, you should test for not zero, and act accordingly.
>
> It is negative for EOF, but positive for I/O errors, which most likely you
> should also exit your loop, and/or handle appropriately.

Not in this particular case. In older versions of the data file I'm
processing, they used csv format to present dozens and dozens of
quantities for each object in the database, and when a value wasn't
available, they simply omitted it, leading to consecutive commas,
and gfortran's list-directed READ statement handled it splendidly.
However, in the latest version of the data file, a missing value is
now represented as "null". For example, five values might be in
this file as:

18,97.43,null,-4.31,102

but attempting to read the characters "null" into an INTEGER or REAL
variable triggers an I/O error. I do NOT want to exit the READ loop.
However, once the end-of-file is reached, I do want to exit the READ
loop.

> Otherwise, it makes some sense. In the first case, there is actual EOF.
> In the second, reading after EOF, is an I/O error, and so reported.

So, prior to the READ that triggers the end-of-file condition, the
file pointer is after the last record of the file, but before the EOF,
and after the READ that triggers the end-of-file condition, the file
pointer is now after some imaginary EOF? There is no physical EOF
associated with the file. Does the compiler give the file a logical
EOF marker and is able to position the file pointer before and after it?

> Unless you plan to test for and take appropriate action for different
> I/O errors, taking action for any non-zero value makes sense.

I've written numerous programs where different actions are taken for
different I/O errors. A hypothetical (and trivial) example: A
program that converts Fahrenheit to Celsius. DO loop with WRITE
statement that prompts the user to enter a temperature in Fahrenheit
and a READ statement that accepts the user's input. Type in "abc"
and rather than crashing or exiting the loop, the program can say
"try again" and wait for another entry. But enter a CTRL-Z (an
EOF on Windows), and the loop can exit gracefully.

> I believe this is a not unusual error in some other languages,
> which I won't mention.

On Fri, 15 Jul 2022 15:46:36 -1000, Dave Tholen wrote:

> I'm unfamiliar with IOMSG. Sounds like an intrinsic that returns a text
> description of an IOSTAT numerical value, but I don't see it listed
> among the intrinsic procedures for gfortran. Could you elaborate?

It's not an intrinsic, it's another keyword for the I/O statement, like
IOSTAT. It was new in Fortran 2018.

--
Steve Lionel
Email: firstname at firstnamelastname dot com
Twitter: @DoctorFortran
Blog: https://stevelionel.com/drfortran

On Saturday, July 16, 2022 at 10:15:12 AM UTC-4, Steve Lionel wrote:

> On Fri, 15 Jul 2022 15:46:36 -1000, Dave Tholen wrote:
>
> > I'm unfamiliar with IOMSG. Sounds like an intrinsic that returns a text
> > description of an IOSTAT numerical value, but I don't see it listed
> > among the intrinsic procedures for gfortran. Could you elaborate?
> It's not an intrinsic, it's another keyword for the I/O statement, like
> IOSTAT. It was new in Fortran 2018.
>

You must mean Fortran 2003 standard revision given it is the data transfer (READ) statement being discussed here?

Fortran 2018 introduced an optional argument `ERRMSG` to fetch the messages involving intrinsics that interact with the processor environment, correct? Such as get_command, get_environment_variable, etc.

On 7/15/2022 9:04 PM, gah4 wrote:
> On Friday, July 15, 2022 at 4:47:37 AM UTC-7, Dave Tholen wrote:
>> If you read (sequentially) past the end-of-file, gfortran sets the value
>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
>> of IOSTAT to 5001. I discovered this situation after having written a
>> program expecting subsequent READs to all return a negative number, and
>> of course the program didn't behave as expected until I investigated the
>> problem. So I'm curious as to why the choice was made to have IOSTAT
>> set to something other than -1 on any READ attempt after the end-of-file
>> condition has been triggered. Isn't the file pointer still just beyond
>> the last record in the file, even after the first failed READ?
>
> If I understand it right, you should test for not zero, and act accordingly.
>
> It is negative for EOF, but positive for I/O errors, which most likely you
> should also exit your loop, and/or handle appropriately.
>
> Otherwise, it makes some sense. In the first case, there is actual EOF.
> In the second, reading after EOF, is an I/O error, and so reported.
>
> Unless you plan to test for and take appropriate action for different
> I/O errors, taking action for any non-zero value makes sense.
>
> I believe this is a not unusual error in some other languages,
> which I won't mention.
>
>
>
>
>
>
I know of at least one operating system that included the concept of EOT
within disk files as well. So you could have multiple files in the same
"disk area". You got and EOF return at end of the current file and if
you read again and there wasn't another file present, you got an EOT
return value. It was convenient for some things.

On Saturday, July 16, 2022 at 4:41:04 AM UTC-7, Dave Tholen wrote:

(snip, I wrote)
> > If I understand it right, you should test for not zero, and act accordingly.
> > It is negative for EOF, but positive for I/O errors, which most likely you
> > should also exit your loop, and/or handle appropriately.

> Not in this particular case. In older versions of the data file I'm
> processing, they used csv format to present dozens and dozens of
> quantities for each object in the database, and when a value wasn't
> available, they simply omitted it, leading to consecutive commas,
> and gfortran's list-directed READ statement handled it splendidly.
> However, in the latest version of the data file, a missing value is
> now represented as "null". For example, five values might be in
> this file as:
> 18,97.43,null,-4.31,102
> but attempting to read the characters "null" into an INTEGER or REAL
> variable triggers an I/O error. I do NOT want to exit the READ loop.
> However, once the end-of-file is reached, I do want to exit the READ
> loop.

On either EOF or I/O error, there is no guarantee as to what
is actually stored, or where the file is positioned.

> > Otherwise, it makes some sense. In the first case, there is actual EOF.
> > In the second, reading after EOF, is an I/O error, and so reported.

> So, prior to the READ that triggers the end-of-file condition, the
> file pointer is after the last record of the file, but before the EOF,
> and after the READ that triggers the end-of-file condition, the file
> pointer is now after some imaginary EOF? There is no physical EOF
> associated with the file. Does the compiler give the file a logical
> EOF marker and is able to position the file pointer before and after it?

The standard does have some discussion on that imaginary
EOF record, while indicating that it might not be an actual
physical anything. I am not sure if some historical system
had a physical EOF record that the standard emulates.

In any case, EOF is not an error, and so actual error is different.

> > Unless you plan to test for and take appropriate action for different
> > I/O errors, taking action for any non-zero value makes sense.

> I've written numerous programs where different actions are taken for
> different I/O errors. A hypothetical (and trivial) example: A
> program that converts Fahrenheit to Celsius. DO loop with WRITE
> statement that prompts the user to enter a temperature in Fahrenheit
> and a READ statement that accepts the user's input. Type in "abc"
> and rather than crashing or exiting the loop, the program can say
> "try again" and wait for another entry. But enter a CTRL-Z (an
> EOF on Windows), and the loop can exit gracefully.

I suppose you can do that, I am not sure that there is any
standard behavior, especially as to file position.

Reminds me, Unix, and at least Unix C compilers, all for one
to keep reading after EOF, such that tail -f works. That is,
if the file increases in length then the EOF goes away,
and you can keep reading. I do remember years ago
finding that C in OS/2 allowed that.

On Saturday, July 16, 2022 at 8:59:00 PM UTC+10, Dave Tholen wrote:
> >>>> If you read (sequentially) past the end-of-file, gfortran sets the value
> >>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> >>>> of IOSTAT to 5001. I discovered this situation after having written a
> >>>> program expecting subsequent READs to all return a negative number, and
> >>>> of course the program didn't behave as expected until I investigated the
> >>>> problem. So I'm curious as to why the choice was made to have IOSTAT
> >>>> set to something other than -1 on any READ attempt after the end-of-file
> >>>> condition has been triggered. Isn't the file pointer still just beyond
> >>>> the last record in the file, even after the first failed READ?
>
> >>> You are expected to test for IOSTAT and deal with the end-of-file condition.
> >>> You are not expected to continue reading the file.
> >> I actually do test for IOSTAT and deal with the end-of-file condition by
> >> exiting the DO loop with the READ statement. But once the DO loop has
> >> been exited, the program is in an outer DO loop, which can also read the
> >> file.
..
> > You need to fix the logic.
..
> Apparently I haven't been clear.
> The program has already been fixed.
..
But it hasn't been fixed. You admitted it.
You said that after detecting end of file using IOSTAT, you exited the loop,
and the outer loop then proceeded to execute READs on the same file.
..
> It was fixed before I even posted on this forum. The question is not
> how to fix the problem. The question is why the value of IOSTAT was
> changed.
..
That has already been explained to you.
..
> There must be some rationale.
> >> It also tests IOSTAT to deal with the end-of-file condition,
> > The program has already hit the end of file.
> > Now you are trying to read read the same file at the same place.
> That's just it: same file, same place, same action, yet a different
> IOSTAT result. Not what I had expected.
..
Again, that has been explained to you.
..
> >> but
> >> it tested for a negative number, which IOSTAT was no longer set to.
> > You need to avoid reading after end of file has been detected.
> Why? IOSTAT was added so that a program could handle error conditions.
..
Provided that you deal with the condition and do not do silly things like reading
the same file again, after detecting end-of file.
..
> My program was designed to deal with the error condition. Had gfortran
> consistently returned a negative IOSTAT for every attempt to READ past
> the end-of-file, my program would have worked exactly as intended.

Listen up. Your program is in error. Fix it.
..
> Had I used ifort, my program would have worked exactly as intended.
> >> And if you must know why the nested DO loops, one loop dealt with data
> >> while the other loop dealt with metadata. It was easily fixable. My
> >> question here was not about how to fix the problem, but rather to find
> >> out the rationale for changing the value of IOSTAT in such a situation.
> >> My expectation was that IOSTAT would remain negative. My expectation
> >> was wrong, hence the curiosity.
>
> >>>> On a completely separate matter, I have a different program that
> >>>> didn't behave as expected, and that misbehavior was totally repeatable.
> >>>> In an attempt to debug the program, I added a WRITE statement to check
> >>>> on the value of a variable during execution. However, once the WRITE
> >>>> statement was added, the program started behaving properly, repeatably.
>
> >>> That suggests that there is a bug in your program.
>
> >> Brilliant.
>
> >>> Possibly something has
> >>> been overwritten.
>
> >> I already thought about that, hence the WRITE statement to examine
> >> the values of the array indices.
> > In every program unit, subroutine and function in the entire program?
> I usually sprinkle WRITE statements around to isolate where the crash
> is happening. As I narrow things down, unnecessary WRITE statements are
> either removed or commented out. In this particular case, I got it down
> to a single WRITE statement that could trigger proper behavior.
..
You program has errors. Find them with the help of debug options
such as subscript bound errors, substring errors, and the like.
..
> >> The values were within range and
> >> the program worked properly with the WRITE statement enabled. Comment
> >> out the WRITE statement, and the misbehavior returned. Hence the
> >> frustration.
>
> >>> Turn on subscript bounds checking (if available).
> >>> And while you're at it, turn on all checks.
> .
> >> Easier said than done, as the program was built from dozens of separately
> >> written and compiled subprograms. It has not always been obvious to me
> >> whether a compilation flag needs to be applied to every single subprogram
> >> in order to accomplish a goal.
> > Obviously, program checking needs to be specified for every separate compilation,
> > as the apparent overwriting could be in any one or more subroutine and function.
> So, you're saying that if my program has subprograms A, B, C, D, and E,
> and the crash occurs while it's executing subprogram C, the apparent
> overwriting could be in A, B, D, or E?
> >>>> Comment out the added WRITE statement, and the program once again
> >>>> misbehaves, repeatedly. Re-enable the WRITE statement, and everything
> >>>> is once again hunky-dory. Damned frustrating. It's too easy to blame
> >>>> the optimizer. Anybody have any generic advice on what to look for in
> >>>> such a situation?

On Saturday, July 16, 2022 at 8:59:00 PM UTC+10, Dave Tholen wrote:
> >>>> If you read (sequentially) past the end-of-file, gfortran sets the value
> >>>> of IOSTAT to -1, but if you read a second time, gfortran sets the value
> >>>> of IOSTAT to 5001. I discovered this situation after having written a
> >>>> program expecting subsequent READs to all return a negative number, and
> >>>> of course the program didn't behave as expected until I investigated the
> >>>> problem. So I'm curious as to why the choice was made to have IOSTAT
> >>>> set to something other than -1 on any READ attempt after the end-of-file
> >>>> condition has been triggered. Isn't the file pointer still just beyond
> >>>> the last record in the file, even after the first failed READ?
>
> >>> You are expected to test for IOSTAT and deal with the end-of-file condition.
> >>> You are not expected to continue reading the file.
> >> I actually do test for IOSTAT and deal with the end-of-file condition by
> >> exiting the DO loop with the READ statement. But once the DO loop has
> >> been exited, the program is in an outer DO loop, which can also read the
> >> file.
> > You need to fix the logic.
> Apparently I haven't been clear. The program has already been fixed.
> It was fixed before I even posted on this forum. The question is not
> how to fix the problem. The question is why the value of IOSTAT was
> changed. There must be some rationale.
> >> It also tests IOSTAT to deal with the end-of-file condition,
> > The program has already hit the end of file.
> > Now you are trying to read read the same file at the same place.
> That's just it: same file, same place, same action, yet a different
> IOSTAT result. Not what I had expected.
> >> but
> >> it tested for a negative number, which IOSTAT was no longer set to.
> > You need to avoid reading after end of file has been detected.
> Why? IOSTAT was added so that a program could handle error conditions.
> My program was designed to deal with the error condition. Had gfortran
> consistently returned a negative IOSTAT for every attempt to READ past
> the end-of-file, my program would have worked exactly as intended. Had
> I used ifort, my program would have worked exactly as intended.
> >> And if you must know why the nested DO loops, one loop dealt with data
> >> while the other loop dealt with metadata. It was easily fixable. My
> >> question here was not about how to fix the problem, but rather to find
> >> out the rationale for changing the value of IOSTAT in such a situation.
> >> My expectation was that IOSTAT would remain negative. My expectation
> >> was wrong, hence the curiosity.
>
> >>>> On a completely separate matter, I have a different program that
> >>>> didn't behave as expected, and that misbehavior was totally repeatable.
> >>>> In an attempt to debug the program, I added a WRITE statement to check
> >>>> on the value of a variable during execution. However, once the WRITE
> >>>> statement was added, the program started behaving properly, repeatably.
>
> >>> That suggests that there is a bug in your program.
>
> >> Brilliant.
>
> >>> Possibly something has
> >>> been overwritten.
>
> >> I already thought about that, hence the WRITE statement to examine
> >> the values of the array indices.
> > In every program unit, subroutine and function in the entire program?
> I usually sprinkle WRITE statements around to isolate where the crash
> is happening. As I narrow things down, unnecessary WRITE statements are
> either removed or commented out. In this particular case, I got it down
> to a single WRITE statement that could trigger proper behavior.
> >> The values were within range and
> >> the program worked properly with the WRITE statement enabled. Comment
> >> out the WRITE statement, and the misbehavior returned. Hence the
> >> frustration.
>
> >>> Turn on subscript bounds checking (if available).
> >>> And while you're at it, turn on all checks.
> .
> >> Easier said than done, as the program was built from dozens of separately
> >> written and compiled subprograms. It has not always been obvious to me
> >> whether a compilation flag needs to be applied to every single subprogram
> >> in order to accomplish a goal.
> > Obviously, program checking needs to be specified for every separate compilation,
> > as the apparent overwriting could be in any one or more subroutine and function.
..
> So, you're saying that if my program has subprograms A, B, C, D, and E,
> and the crash occurs while it's executing subprogram C, the apparent
> overwriting could be in A, B, D, or E?
..
The error could be in any of A, B, C, D, E, and/or the main program.

On Sunday, July 17, 2022 at 3:40:55 AM UTC+10, gah4 wrote:
> On Saturday, July 16, 2022 at 4:41:04 AM UTC-7, Dave Tholen wrote:
>
> (snip, I wrote)
> > > If I understand it right, you should test for not zero, and act accordingly.
>
> > > It is negative for EOF, but positive for I/O errors, which most likely you
> > > should also exit your loop, and/or handle appropriately.
>
> > Not in this particular case. In older versions of the data file I'm
> > processing, they used csv format to present dozens and dozens of
> > quantities for each object in the database, and when a value wasn't
> > available, they simply omitted it, leading to consecutive commas,
> > and gfortran's list-directed READ statement handled it splendidly.
> > However, in the latest version of the data file, a missing value is
> > now represented as "null". For example, five values might be in
> > this file as:
>
> > 18,97.43,null,-4.31,102
>
> > but attempting to read the characters "null" into an INTEGER or REAL
> > variable triggers an I/O error.
..
Characters are characters, and cannot be read into an integer or real variable.
An appropriate variable would be a CHARACTER variable, but then the
characters NULL would need to be enclosed in apostrophes.
A better approach might be to read the entire line into a CHARACTER variable,
and then to scan it for 'null, and then to delete the word. Finally read the
altered line.
..
> I do NOT want to exit the READ loop.
> > However, once the end-of-file is reached, I do want to exit the READ
> > loop.
> On either EOF or I/O error, there is no guarantee as to what
> is actually stored, or where the file is positioned.
> > > Otherwise, it makes some sense. In the first case, there is actual EOF.
> > > In the second, reading after EOF, is an I/O error, and so reported.
>
> > So, prior to the READ that triggers the end-of-file condition, the
> > file pointer is after the last record of the file, but before the EOF,
> > and after the READ that triggers the end-of-file condition, the file
> > pointer is now after some imaginary EOF? There is no physical EOF
> > associated with the file. Does the compiler give the file a logical
> > EOF marker and is able to position the file pointer before and after it?
> The standard does have some discussion on that imaginary
> EOF record, while indicating that it might not be an actual
> physical anything. I am not sure if some historical system
> had a physical EOF record that the standard emulates.
>
> In any case, EOF is not an error, and so actual error is different.
> > > Unless you plan to test for and take appropriate action for different
> > > I/O errors, taking action for any non-zero value makes sense.

On 7/16/2022 12:04 PM, Robin Vowels wrote:
> On Sunday, July 17, 2022 at 3:40:55 AM UTC+10, gah4 wrote:
>> On Saturday, July 16, 2022 at 4:41:04 AM UTC-7, Dave Tholen wrote:
>>
>> (snip, I wrote)
>>>> If I understand it right, you should test for not zero, and act accordingly.
>>>> It is negative for EOF, but positive for I/O errors, which most likely you
>>>> should also exit your loop, and/or handle appropriately.
>>> Not in this particular case. In older versions of the data file I'm
>>> processing, they used csv format to present dozens and dozens of
>>> quantities for each object in the database, and when a value wasn't
>>> available, they simply omitted it, leading to consecutive commas,
>>> and gfortran's list-directed READ statement handled it splendidly.
>>> However, in the latest version of the data file, a missing value is
>>> now represented as "null". For example, five values might be in
>>> this file as:
>>> 18,97.43,null,-4.31,102
>>> but attempting to read the characters "null" into an INTEGER or REAL
>>> variable triggers an I/O error.
> .
> Characters are characters, and cannot be read into an integer or real variable.
> An appropriate variable would be a CHARACTER variable, but then the
> characters NULL would need to be enclosed in apostrophes.
> A better approach might be to read the entire line into a CHARACTER variable,
> and then to scan it for 'null, and then to delete the word. Finally read the
> altered line.
> .
>

If each READ statement is definitely reading one and only one line, then
reading each line into a character variable, deleting each instance of
the word 'null', and then reading data from the character variable seems
like the easiest solution by far.

Louis