Somehow, my compiler (GNU Fortran (MinGW.org GCC-6.3.0-1) 6.3.0) has issues reading ASCII-files in UTF8 format. The ASCII-files read fine in Notepad++, but my programs give undesired results. At the moment, I convert the UTF8-files to ANSI, but this requires my intervention. Is there a way to just read the ASCII-files without bothering about if they are UTF8 or ANSI?

On Wednesday, June 28, 2023 at 1:24:33 PM UTC+2, Arjan wrote:
> Somehow, my compiler (GNU Fortran (MinGW.org GCC-6.3.0-1) 6.3.0) has issues reading ASCII-files in UTF8 format. The ASCII-files read fine in Notepad++, but my programs give undesired results. At the moment, I convert the UTF8-files to ANSI, but this requires my intervention. Is there a way to just read the ASCII-files without bothering about if they are UTF8 or ANSI?

You should be able to open them with the keyword "ENCODING='UTF-8'". Mind you, on Windows many editors add a "BOM" to the file, that is a byte-order marker that the editors will hide but are two bytes at teh start of the file. This may mess up things if you do not skip over them.

Regards,

Arjen

> two bytes at the start of the file

Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark

On 6/28/2023 8:47 AM, eugene_...@yahoo.com wrote:
>> two bytes at the start of the file
>
> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark

That is gross. Any one opening any text file should expect UTF-8 now.

Lynn

On 6/28/2023 6:06 PM, Lynn McGuire wrote:
> On 6/28/2023 8:47 AM, eugene_...@yahoo.com wrote:
>>>   two bytes at the start of the file
>>
>> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark
>
> That is gross.  Any one opening any text file should expect UTF-8 now.
>
> Lynn
>
All of my text files will be plain 8-bit extended ascii, or else.

On 6/28/2023 6:42 PM, Gary Scott wrote:
> On 6/28/2023 6:06 PM, Lynn McGuire wrote:
>> On 6/28/2023 8:47 AM, eugene_...@yahoo.com wrote:
>>>>   two bytes at the start of the file
>>>
>>> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark
>>
>> That is gross.  Any one opening any text file should expect UTF-8 now.
>>
>> Lynn
>>
> All of my text files will be plain 8-bit extended ascii, or else.
....or ebcdic

On Wednesday, June 28, 2023 at 3:47:12 PM UTC+2, eugene wrote:
> > two bytes at the start of the file
>
> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark

So, simply opening a text file that is encoded in UTF-8 requires a delicate procedure:
- There may be a byte-order mark. There may not be one.
- If there is, it may be two bytes, in which case there are stil two possibilities you need to check.
- The third possibility is that there are three bytes that confer the same information about the byte ordering.

And there is no way to accurately predict which situation you are facing .... I think I am getting old.

Regards,

Arjen

On 6/29/2023 2:14 AM, Arjen Markus wrote:
> On Wednesday, June 28, 2023 at 3:47:12 PM UTC+2, eugene wrote:
>>> two bytes at the start of the file
>>
>> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark
>
> So, simply opening a text file that is encoded in UTF-8 requires a delicate procedure:
> - There may be a byte-order mark. There may not be one.
> - If there is, it may be two bytes, in which case there are stil two possibilities you need to check.
> - The third possibility is that there are three bytes that confer the same information about the byte ordering.
>
> And there is no way to accurately predict which situation you are facing ... I think I am getting old.
>

It's not so hard, just always read the first 3 bytes and figure out what
they are and proceed as required.

> Regards,
>
> Arjen

On 6/29/2023 8:42 AM, Gary Scott wrote:
> On 6/29/2023 2:14 AM, Arjen Markus wrote:
>> On Wednesday, June 28, 2023 at 3:47:12 PM UTC+2, eugene wrote:
>>>> two bytes at the start of the file
>>>
>>> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark
>>
>> So, simply opening a text file that is encoded in UTF-8 requires a
>> delicate procedure:
>> - There may be a byte-order mark. There may not be one.
>> - If there is, it may be two bytes, in which case there are stil two
>> possibilities you need to check.
>> - The third possibility is that there are three bytes that confer the
>> same information about the byte ordering.
>>
>> And there is no way to accurately predict which situation you are
>> facing ... I think I am getting old.
>>
>
> It's not so hard, just always read the first 3 bytes and figure out what
> they are and proceed as required.
>
>
>> Regards,
>>
>> Arjen

But those 3 bytes are not required to be there for UTF-8. So, those
three bytes will probably not be there.

We are assuming that all text files are UTF-8 now.

Lynn

On 6/29/2023 4:39 PM, Lynn McGuire wrote:
> On 6/29/2023 8:42 AM, Gary Scott wrote:
>> On 6/29/2023 2:14 AM, Arjen Markus wrote:
>>> On Wednesday, June 28, 2023 at 3:47:12 PM UTC+2, eugene wrote:
>>>>> two bytes at the start of the file
>>>>
>>>> Or three bytes for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark
>>>
>>> So, simply opening a text file that is encoded in UTF-8 requires a
>>> delicate procedure:
>>> - There may be a byte-order mark. There may not be one.
>>> - If there is, it may be two bytes, in which case there are stil two
>>> possibilities you need to check.
>>> - The third possibility is that there are three bytes that confer the
>>> same information about the byte ordering.
>>>
>>> And there is no way to accurately predict which situation you are
>>> facing ... I think I am getting old.
>>>
>>
>> It's not so hard, just always read the first 3 bytes and figure out
>> what they are and proceed as required.
>>
>>
>>> Regards,
>>>
>>> Arjen
>
> But those 3 bytes are not required to be there for UTF-8.  So, those
> three bytes will probably not be there.
>
> We are assuming that all text files are UTF-8 now.
Of course, once you've checked that they arent there, you can either
rewind and read again or prepend the 3 characters to the next read
operation.
>
> Lynn
>
>

On Friday, June 30, 2023 at 6:45:20 AM UTC-7, Gary Scott wrote:
> Of course, once you've checked that they arent there, you can either
> rewind and read again or prepend the 3 characters to the next read
> operation.

Hard to rewind a TTY or socket or pipe, though.

On 6/30/2023 11:11 AM, Peter Klausler US wrote:
> On Friday, June 30, 2023 at 6:45:20 AM UTC-7, Gary Scott wrote:
>> Of course, once you've checked that they arent there, you can either
>> rewind and read again or prepend the 3 characters to the next read
>> operation.
>
> Hard to rewind a TTY or socket or pipe, though.
Sure...there are ways to handle it