Let's say that I have a C program that is reading the contents of
a file, one line at a time, with fgets(). While this is ongoing, the
contents of the file are overwritten with new data.

Is it guaranteed that the C program will continue to have access
to the old contents of the file, as long as this program keeps the
original file descriptor open?

--

Clark Smith <noaddress@nowhere.net> wrote:
> Let's say that I have a C program that is reading the contents of
> a file, one line at a time, with fgets(). While this is ongoing, the
> contents of the file are overwritten with new data.
>
> Is it guaranteed that the C program will continue to have access
> to the old contents of the file, as long as this program keeps the
> original file descriptor open?

No. Or at least no when given your minimal conditions above (no
locking stated, no coordination between reader and writer stated).

At Sun, 26 Feb 2023 18:52:37 -0000 (UTC) Clark Smith <noaddress@nowhere.net> wrote:

>
> Let's say that I have a C program that is reading the contents of
> a file, one line at a time, with fgets(). While this is ongoing, the
> contents of the file are overwritten with new data.
>
> Is it guaranteed that the C program will continue to have access
> to the old contents of the file, as long as this program keeps the
> original file descriptor open?

Generally yes.

>
>
>
>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

On 2023-02-26 22:45, Robert Heller wrote:
> At Sun, 26 Feb 2023 18:52:37 -0000 (UTC) Clark Smith <noaddress@nowhere.net> wrote:
>
>>
>> Let's say that I have a C program that is reading the contents of
>> a file, one line at a time, with fgets(). While this is ongoing, the
>> contents of the file are overwritten with new data.
>>
>> Is it guaranteed that the C program will continue to have access
>> to the old contents of the file, as long as this program keeps the
>> original file descriptor open?
>
> Generally yes.

Or not.

Take a script that runs for long. Edit it while running, the script
usually errors out. Bash.

--
Cheers, Carlos.

Clark Smith <noaddress@nowhere.net> writes:

> Let's say that I have a C program that is reading the contents of
> a file, one line at a time, with fgets(). While this is ongoing, the
> contents of the file are overwritten with new data.
>
> Is it guaranteed that the C program will continue to have access
> to the old contents of the file, as long as this program keeps the
> original file descriptor open?

If the original file is deleted and a new file created any programs
that have the old file open won't see any changes. The file is really
deleted after all the programs using it close the file.

If the file is actually being updated in place the program will see data
change subject to buffering issues.

--
Dan Espen

On 26/02/2023 18:52, Clark Smith wrote:
> Let's say that I have a C program that is reading the contents of
> a file, one line at a time, with fgets(). While this is ongoing, the
> contents of the file are overwritten with new data.
>
> Is it guaranteed that the C program will continue to have access
> to the old contents of the file, as long as this program keeps the
> original file descriptor open?
>
>
I would say almost certainly not.

A file that is being simultaneously read and written too without some
form of semaphore will give indeterminate results depending on timings
of asynchronous processes.

In all probabilty the read will be heavily cached, yes, but I would
except that cache to be writeable to in real time before being read.

In short of you read an extent from a file, write the next extent, and
then read it, you will get new data, not old.

If you want to preserve all the old data, program a lock on the file,
explicitly load it into memory or duplicate the file and when don.,
remove the lock.

>
>
>

--
For in reason, all government without the consent of the governed is the
very definition of slavery.

Jonathan Swift

On 26/02/2023 21:45, Robert Heller wrote:
> At Sun, 26 Feb 2023 18:52:37 -0000 (UTC) Clark Smith <noaddress@nowhere.net> wrote:
>
>>
>> Let's say that I have a C program that is reading the contents of
>> a file, one line at a time, with fgets(). While this is ongoing, the
>> contents of the file are overwritten with new data.
>>
>> Is it guaranteed that the C program will continue to have access
>> to the old contents of the file, as long as this program keeps the
>> original file descriptor open?
>
> Generally yes.

I am pretty sure generally NO.

Otherwise databases would have a hard time working

--
For in reason, all government without the consent of the governed is the
very definition of slavery.

Jonathan Swift

On 26/02/2023 22:05, Carlos E.R. wrote:
> On 2023-02-26 22:45, Robert Heller wrote:
>> At Sun, 26 Feb 2023 18:52:37 -0000 (UTC) Clark Smith
>> <noaddress@nowhere.net> wrote:
>>
>>>
>>>     Let's say that I have a C program that is reading the contents of
>>> a file, one line at a time, with fgets(). While this is ongoing, the
>>> contents of the file are overwritten with new data.
>>>
>>>     Is it guaranteed that the C program will continue to have access
>>> to the old contents of the file, as long as this program keeps the
>>> original file descriptor open?
>>
>> Generally yes.
>
> Or not.
>
> Take a script that runs for long. Edit it while running, the script
> usually errors out. Bash.
>
The edit process probably writes a new file and erases the old.

I've certainly used and am using files on a small RAMdisk as a way for
two processes to communicate on a 'I write, you read' basis.
The writer is am icecast interpreter reading program data off an MP3
icecast stream, the reader is an Ajax process polling a piece of web
server to update the screen of an audioplayer app.

Ok in this case the reader opens the file reads it and closes it.

--
For in reason, all government without the consent of the governed is the
very definition of slavery.

Jonathan Swift

On 26/02/2023 22:35, Dan Espen wrote:
> Clark Smith <noaddress@nowhere.net> writes:
>
>> Let's say that I have a C program that is reading the contents of
>> a file, one line at a time, with fgets(). While this is ongoing, the
>> contents of the file are overwritten with new data.
>>
>> Is it guaranteed that the C program will continue to have access
>> to the old contents of the file, as long as this program keeps the
>> original file descriptor open?
>
> If the original file is deleted and a new file created any programs
> that have the old file open won't see any changes. The file is really
> deleted after all the programs using it close the file.
>
> If the file is actually being updated in place the program will see data
> change subject to buffering issues.
>
+1.

That's the neatest way of describing it 'subject to buffering issues'
hides a whole slew of complexity. :-)

--
Those who want slavery should have the grace to name it by its proper
name. They must face the full meaning of that which they are advocating
or condoning; the full, exact, specific meaning of collectivism, of its
logical implications, of the principles upon which it is based, and of
the ultimate consequences to which these principles will lead. They must
face it, then decide whether this is what they want or not.

Ayn Rand.

On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher <tnp@invalid.invalid> wrote:
> I am pretty sure generally NO.
> Otherwise databases would have a hard time working

Data base applications work by having one thread that does the physical i/o for
each file that contains the database. Multiple applications can use the database,
but they don't do the i/o to the disk.

For other files, applications that load the entire file into ram, and then write
a new copy, the old inode/file will remain until all applications that have the
file open have finished, so those files can be read ok regardless of buffering.

For applications that update the file in place, corruption can occur for other
applications reading that file using buffering.

Regards, Dave Hodgins

On 27/02/2023 00:06, David W. Hodgins wrote:
> On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher
> <tnp@invalid.invalid> wrote:
>> I am pretty sure generally NO. Otherwise databases would have a
>> hard time working
>
> Data base applications work by having one thread that does the
> physical i/o for each file that contains the database. Multiple
> applications can use the database, but they don't do the i/o to the
> disk.
>
> For other files, applications that load the entire file into ram, and
> then write a new copy, the old inode/file will remain until all
> applications that have the file open have finished, so those files
> can be read ok regardless of buffering.
Straw man.
That is not the case the OP was talking about. Its the one you happen to
know about, but put it away and don't show off

The issue is a file that has two file descriptors in two processes which
are reading and writing to its extents simultaneously.
How much is cached *in program* is the issue, because at the OS level
disk caches will be read and written by and shared by both processes.

It is entirely analogous to a database that doesn't serialise the
processes consistently.

So you cant tell if a given process will complete a write, before the
other process calls for a read.

Whether those reads and writes are actually to the cache or the disk is
supremely irrelevant as are inodes and file sand directories. We are
talking about a file that is open and not being closed by two concurrent
processes.
IIRC a C style fread() or fwrite() goes straight to the OS and so there
is no in program caching below that level, only OS disk caching which is
invisible to the application

So if process A does an fseek(X), fwrite() and process B then does
fseek(X), fread() of the same data extent, the last written value will
be read.

>
> For applications that update the file in place, corruption can occur
> for other applications reading that file using buffering.
>
What buffereing?
If you read the whole file into application RAM., of course you are
immune from changes made to it subsequently.

But if you are simply using fseek() and fread () to access extents of
the file, you will get whatever the last data put there is.

> Regards, Dave Hodgins

--
"Anyone who believes that the laws of physics are mere social
conventions is invited to try transgressing those conventions from the
windows of my apartment. (I live on the twenty-first floor.) "

Alan Sokal

On Sun, 26 Feb 2023 19:26:56 -0500, The Natural Philosopher <tnp@invalid.invalid> wrote:
> The issue is a file that has two file descriptors in two processes which
> are reading and writing to its extents simultaneously.
> How much is cached *in program* is the issue, because at the OS level
> disk caches will be read and written by and shared by both processes.
>
> It is entirely analogous to a database that doesn't serialise the
> processes consistently.
>
> So you cant tell if a given process will complete a write, before the
> other process calls for a read.
>
> Whether those reads and writes are actually to the cache or the disk is
> supremely irrelevant as are inodes and file sand directories. We are
> talking about a file that is open and not being closed by two concurrent
> processes.
> IIRC a C style fread() or fwrite() goes straight to the OS and so there
> is no in program caching below that level, only OS disk caching which is
> invisible to the application

Multiple applications can read/write the same file in place, but they must
have some way of co-ordinating which parts of the file each can update.

There are many ways it can safely be done, but if not done in a safe manner
will lead to data corruption.

> So if process A does an fseek(X), fwrite() and process B then does
> fseek(X), fread() of the same data extent, the last written value will
> be read.
>
>>
>> For applications that update the file in place, corruption can occur
>> for other applications reading that file using buffering.
>>
> What buffereing?

> If you read the whole file into application RAM., of course you are
> immune from changes made to it subsequently.

Buffering is the alternative to reading the whole file into ram. I.E. reading
one part at a time.

The example given earlier in this thread is bash scripts. Bash use buffering.
If the entire script fits in one buffer, great. If it's larger, then updating
the script while it's being run will most likely cause problems.

> But if you are simply using fseek() and fread () to access extents of
> the file, you will get whatever the last data put there is.

With unsafe handling, two applications read the same data, one makes changes and
writes it, the other makes other changes and writes it. The first applications
changes are lost.

At the hardware level, i/o is done with 512 or 4096 byte sectors. That is the
minimum size block two r/w applications must co-ordinate writing to ensure
changes are not lost. It can be ownership maps, serialized locks of blocks,
notification that a block is being changed, etc. There are many ways it can
be done, but if it isn't being done data loss is a risk.

Regards, Dave Hodgins

On 2/26/23 6:39 PM, The Natural Philosopher wrote:
> On 26/02/2023 22:35, Dan Espen wrote:
>> Clark Smith <noaddress@nowhere.net> writes:
>>
>>>     Let's say that I have a C program that is reading the contents of
>>> a file, one line at a time, with fgets(). While this is ongoing, the
>>> contents of the file are overwritten with new data.
>>>
>>>     Is it guaranteed that the C program will continue to have access
>>> to the old contents of the file, as long as this program keeps the
>>> original file descriptor open?
>>
>> If the original file is deleted and a new file created any programs
>> that have the old file open won't see any changes.  The file is really
>> deleted after all the programs using it close the file.
>>
>> If the file is actually being updated in place the program will see data
>> change subject to buffering issues.
>>
> +1.
>
> That's the neatest way of describing it 'subject to buffering issues'
> hides a whole slew of complexity. :-)

And thus we understand the often-complex file/record-locking
tricks things like multiuser databases do :-)

If your app (or subthread) loads a buffer from File-X and then
another app (or subthread) alters File-X, YOU don't really
KNOW about it ... you're working on an obsolete buffer. Some
smarter apps check if something about the file has changed ...
but any edits you were doing on the obsolete version HAVE to
be thrown out. If a bunch of apps/subthreads are working on
the file at the same time it becomes HOPELESS. Locks are
the easiest approach.

"Shared buffers" - something ALL users can access/edit at the
exact same time - live updating - are also possible but more
work. Some "collaborative" apps seem to use this approach. You
type in something, everybody sees their text/whatever change
too. Edit-wars are possible ... kinda like with WikiPedia :-)

The Natural Philosopher <tnp@invalid.invalid> writes:
> IIRC a C style fread() or fwrite() goes straight to the OS and so
> there is no in program caching below that level, only OS disk caching
> which is invisible to the application

You are thinking of read() and write() here - fread() & fwrite() will
use the process’s own buffer (unless you deliberately made the file
unbuffered).

A network filesystem (e.g. NFS with the async option) may also break
cache consistency.

--
https://www.greenend.org.uk/rjk/

On 2023-02-27 01:06, David W. Hodgins wrote:
> On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher
> <tnp@invalid.invalid> wrote:
>> I am pretty sure generally NO.
>> Otherwise databases would have a hard time working
>
> Data base applications work by having one thread that does the
> physical i/o for each file that contains the database. Multiple
> applications can use the database, but they don't do the i/o to the
> disk.

Not all database applications work that way.

For example, in MsDOS an application could lock for write a section of a
file, write that, then remove the lock. Another application could be
reading the same file, even writing in another section.

This functionality was an add on: you needed to run the optional
share.exe. But Windows had it natively.

I believe Microsoft Access works like that.

I am not sure this behaviour can be done in Linux. Long ago it could not.

--
Cheers, Carlos.

On 2023-02-26 23:35, Dan Espen wrote:
> Clark Smith <noaddress@nowhere.net> writes:
>
>> Let's say that I have a C program that is reading the contents of
>> a file, one line at a time, with fgets(). While this is ongoing, the
>> contents of the file are overwritten with new data.
>>
>> Is it guaranteed that the C program will continue to have access
>> to the old contents of the file, as long as this program keeps the
>> original file descriptor open?
>
> If the original file is deleted and a new file created any programs
> that have the old file open won't see any changes. The file is really
> deleted after all the programs using it close the file.

If the first program keeps the file opened, it can not be actually
deleted in a Linux filesystem. The first program keeps reading the old
file till it closes it.

lsof lists them, and marks "deleted" files.

> If the file is actually being updated in place the program will see data
> change subject to buffering issues.
>

--
Cheers, Carlos.

"Carlos E.R." <robin_listas@es.invalid> writes:
> For example, in MsDOS an application could lock for write a section of
> a file, write that, then remove the lock. Another application could be
> reading the same file, even writing in another section.
>
> This functionality was an add on: you needed to run the optional
> share.exe. But Windows had it natively.
>
> I believe Microsoft Access works like that.
>
>
> I am not sure this behaviour can be done in Linux. Long ago it could
> not.

Discretionary file locking has existed on Linux for as long as I can
remember. Mandatory locking existed in Linux at one point but apparently
was too buggy to be usable in practice, and was removed in Linux 5.15.

--
https://www.greenend.org.uk/rjk/

On 27/02/2023 09:03, Richard Kettlewell wrote:
> The Natural Philosopher <tnp@invalid.invalid> writes:
>> IIRC a C style fread() or fwrite() goes straight to the OS and so
>> there is no in program caching below that level, only OS disk caching
>> which is invisible to the application
>
> You are thinking of read() and write() here - fread() & fwrite() will
> use the process’s own buffer (unless you deliberately made the file
> unbuffered).
>
Well yes and no, I was not aware that processes had their own buffers
*below* fread/fwrite level.

> A network filesystem (e.g. NFS with the async option) may also break
> cache consistency.
>

Even more complexity. :)

--
“People believe certain stories because everyone important tells them,
and people tell those stories because everyone important believes them.
Indeed, when a conventional wisdom is at its fullest strength, one’s
agreement with that conventional wisdom becomes almost a litmus test of
one’s suitability to be taken seriously.”

Paul Krugman

On 27/02/2023 09:34, Carlos E.R. wrote:
> On 2023-02-27 01:06, David W. Hodgins wrote:
>> On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher
>> <tnp@invalid.invalid> wrote:
>>> I am pretty sure generally NO.
>>> Otherwise databases would have a hard time working
>>
>> Data base applications work by having one thread that does the
>> physical i/o for each file that contains the database. Multiple
>> applications can use the database, but they don't do the i/o to the
>> disk.
>
> Not all database applications work that way.
>
> For example, in MsDOS an application could lock for write a section of a
> file, write that, then remove the lock. Another application could be
> reading the same file, even writing in another section.
>
> This functionality was an add on: you needed to run the optional
> share.exe. But Windows had it natively.
>
> I believe Microsoft Access works like that.
>
>
> I am not sure this behaviour can be done in Linux. Long ago it could not.
>
File lockining always existed at the level of IIRC whole files.
But obliviously it is far far easier to have a databse engine accepting
all requests and using (at least) just one process, if not one thread,
to do all I/O

Even then things get tricky.
Consider you have placed an item on your 'basket' on a shipping app, on
the basies that there are 2 in stock, go make a cup of coffeee, come
back and try to checkout to find there are now none.

--
“People believe certain stories because everyone important tells them,
and people tell those stories because everyone important believes them.
Indeed, when a conventional wisdom is at its fullest strength, one’s
agreement with that conventional wisdom becomes almost a litmus test of
one’s suitability to be taken seriously.”

Paul Krugman

On 2023-02-27 10:49, Richard Kettlewell wrote:
> "Carlos E.R." <robin_listas@es.invalid> writes:
>> For example, in MsDOS an application could lock for write a section of
>> a file, write that, then remove the lock. Another application could be
>> reading the same file, even writing in another section.
>>
>> This functionality was an add on: you needed to run the optional
>> share.exe. But Windows had it natively.
>>
>> I believe Microsoft Access works like that.
>>
>>
>> I am not sure this behaviour can be done in Linux. Long ago it could
>> not.
>
> Discretionary file locking has existed on Linux for as long as I can
> remember. Mandatory locking existed in Linux at one point but apparently
> was too buggy to be usable in practice, and was removed in Linux 5.15.
>

In MsDOS was mandatory (if the exe had loaded).

--
Cheers, Carlos.

On 2023-02-27 11:02, The Natural Philosopher wrote:
> On 27/02/2023 09:34, Carlos E.R. wrote:
>> On 2023-02-27 01:06, David W. Hodgins wrote:
>>> On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher
>>> <tnp@invalid.invalid> wrote:
>>>> I am pretty sure generally NO.
>>>> Otherwise databases would have a hard time working
>>>
>>> Data base applications work by having one thread that does the
>>> physical i/o for each file that contains the database. Multiple
>>> applications can use the database, but they don't do the i/o to the
>>> disk.
>>
>> Not all database applications work that way.
>>
>> For example, in MsDOS an application could lock for write a section of
>> a file, write that, then remove the lock. Another application could be
>> reading the same file, even writing in another section.
>>
>> This functionality was an add on: you needed to run the optional
>> share.exe. But Windows had it natively.
>>
>> I believe Microsoft Access works like that.
>>
>>
>> I am not sure this behaviour can be done in Linux. Long ago it could not.
>>
> File lockining always existed at the level of IIRC whole files.
> But obliviously it is far far easier to have a databse engine accepting
> all requests and using (at least) just one process, if not one thread,
> to do all I/O

Easier, depends. MS Access is trivially easy to use (for simple databases).

>
> Even then things get tricky.
> Consider you have placed an item on your 'basket' on a shipping app, on
> the basies that there are 2 in stock, go make a cup of coffeee, come
> back and try to checkout to find there are now none.
>
>
>

--
Cheers, Carlos.

On 27/02/2023 10:13, Carlos E.R. wrote:
> On 2023-02-27 11:02, The Natural Philosopher wrote:
>> On 27/02/2023 09:34, Carlos E.R. wrote:
>>> On 2023-02-27 01:06, David W. Hodgins wrote:
>>>> On Sun, 26 Feb 2023 18:32:38 -0500, The Natural Philosopher
>>>> <tnp@invalid.invalid> wrote:
>>>>> I am pretty sure generally NO.
>>>>> Otherwise databases would have a hard time working
>>>>
>>>> Data base applications work by having one thread that does the
>>>> physical i/o for each file that contains the database. Multiple
>>>> applications can use the database, but they don't do the i/o to the
>>>> disk.
>>>
>>> Not all database applications work that way.
>>>
>>> For example, in MsDOS an application could lock for write a section
>>> of a file, write that, then remove the lock. Another application
>>> could be reading the same file, even writing in another section.
>>>
>>> This functionality was an add on: you needed to run the optional
>>> share.exe. But Windows had it natively.
>>>
>>> I believe Microsoft Access works like that.
>>>
>>>
>>> I am not sure this behaviour can be done in Linux. Long ago it could
>>> not.
>>>
>> File lockining always existed at the level of IIRC whole files.
>> But obliviously it is far far easier to have a databse engine
>> accepting all requests and using (at least) just one process, if not
>> one thread, to do all I/O
>
> Easier, depends. MS Access is trivially easy to use (for simple databases).
>
You call that a database? I dont. Any more than SqlLite is a
database...both are single user/process access only.

>>
>> Even then things get tricky.
>> Consider you have placed an item on your 'basket' on a shipping app,
>> on the basies that there are 2 in stock, go make a cup of coffeee,
>> come back and try to checkout to find there are now none.
>>
>>
>>
>

--
I would rather have questions that cannot be answered...
....than to have answers that cannot be questioned

Richard Feynman

On 2023-02-27 10:58, The Natural Philosopher wrote:
> On 27/02/2023 09:03, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> IIRC a C style fread() or fwrite() goes straight to the OS and so
>>> there is no in program caching below that level, only OS disk caching
>>> which is invisible to the application
>>
>> You are thinking of read() and write() here - fread() & fwrite() will
>> use the process’s own buffer (unless you deliberately made the file
>> unbuffered).
>>
> Well yes and no, I was not aware that processes had their own buffers
> *below* fread/fwrite level.

I don't remember that much detail in C, but in Pascal (Borland variety,
possibly Lazarus too) each file gets (got) a buffer of a hundred
something buffer by default, inside the application.

--
Cheers, Carlos.

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 27/02/2023 09:03, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> IIRC a C style fread() or fwrite() goes straight to the OS and so
>>> there is no in program caching below that level, only OS disk caching
>>> which is invisible to the application
>> You are thinking of read() and write() here - fread() & fwrite() will
>> use the process’s own buffer (unless you deliberately made the file
>> unbuffered).
>>
> Well yes and no, I was not aware that processes had their own buffers
> *below* fread/fwrite level.

All of fread(), fgetc(), fscanf(), fgets() &c read from an in-process
buffer refilled with read() when empty. Similarly all of fwrite(),
fputc(), fprintf(), fputs() &c write to an in-process buffer which is
flushed with write() when full (or in some cases, when a newline is
written).

Nuances to this include:
* You can change the in-process buffering policy with setvbuf.
* A reasonable implementation choice is for large reads and writes to
bypass the in-process buffer, as an optimization.

read() and write() are the kernel syscalls which transfer data from/to
the buffer cache (or from/to a pipe or socket’s internal buffer, &c).

--
https://www.greenend.org.uk/rjk/

On Mon, 27 Feb 2023 05:17:30 -0500, The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 27/02/2023 10:13, Carlos E.R. wrote:
>> Easier, depends. MS Access is trivially easy to use (for simple databases).
> You call that a database? I dont. Any more than SqlLite is a
> database...both are single user/process access only.

Database design does not consider the physical storage method. Once the data
requirements have been fully normalized
https://en.wikipedia.org/wiki/Database_normalization
it may need some level of denormalization for performance.

It doesn't matter whether it's stored in IMS DB, DB2, sqlite, spreadsheets, or flat
files. Any data that uses tables is logically a database, and the same design rules
apply, whether it's done formally or not.

Regards, Dave Hodgins