Hello all,

I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
that the whole thing slows down to a crawl (about 5 GByte consisting of
50.000 files in over eight hours). I've also tried a another, different USB
stick and see the same thing happen.

Windows task-manager shows my programs procesor usage as zero percent.

Remark: I've used the same program to copy even larger quantities of files
to an USB harddisk, and have not experienced any kind of slowdown there.

I know that an USB stick is rather slow in comparision to a harddisk, but
the above is just ludicrous. :-(

Does anyone have an idea what is going on and how to speed the whole thing
up ?

Regards,
Rudy Wieser

P.s.
I remember having had similar problems while drag-and-dropping the source
folder onto the USB stick. IOW, not a problem with the program itself.

On 1/12/2022 6:28 AM, R.Wieser wrote:
> Hello all,
>
> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
> that the whole thing slows down to a crawl (about 5 GByte consisting of
> 50.000 files in over eight hours). I've also tried a another, different USB
> stick and see the same thing happen.
>
> Windows task-manager shows my programs procesor usage as zero percent.
>
> Remark: I've used the same program to copy even larger quantities of files
> to an USB harddisk, and have not experienced any kind of slowdown there.
>
> I know that an USB stick is rather slow in comparision to a harddisk, but
> the above is just ludicrous. :-(
>
> Does anyone have an idea what is going on and how to speed the whole thing
> up ?
>
> Regards,
> Rudy Wieser
>
> P.s.
> I remember having had similar problems while drag-and-dropping the source
> folder onto the USB stick. IOW, not a problem with the program itself.

I don't have an answer, for what part of the system is screwing up.

As a workaround, try to package the files with 7-ZIP instead.

https://www.7-zip.org/

You can request 7-ZIP to tar the files, and it also has a
segmentation function, so can ensure that output files are less than
4GB to beat the FAT32 limitations on file size for the archive file.
There is a place in the GUI interface to set segment size.

some.zip \
some.zip1 \___ (Bad illustration of beating 4GB file size
some.zip2 / limit of FAT32... You probably know the file
extension for ZIP better than I do :-) )

Do the compression or packaging step on the hard drive. Then
copy across one or more of the large segment files to the USB stick.

That will give a better idea of the speed of the stick. Large
files will do the fewest directory updates and display more of
the sustained transfer rate of the device.

The reason you don't want to point 7-ZIP at the stick directly
for the output step, is to not give it any excuses this time.

7-ZIP can be used later, to do random access within the archive
on the stick, if so desired. If you need NOTE.txt from the 50,000
files, you can get it without unpacking the entire archive.

By preparing the archive file on the hard drive first, you'll be
doing the largest flash block copies possible that way.

You might also want to inspect the file system of the destination
device and make sure the cluster size is 32KB or whatever, rather
than just 512 bytes. An accidental too-small cluster size will
do stuff like that. Flash page sizes are pretty large, and asking
a USB stick to do things in 512 byte clusters, brings out the worst
(read-modify-write at the physical level) in it. That burns up
the flash write life, like crazy. The 32KB cluster size choice,
is getting closer in powers-of-2, to the true page size in the hardware.

This command, agrees with the value displayed when I select "format"
of the associated USB stick partition letter.

wmic volume get driveletter,blocksize

You can add EXFAT support to Windows XP, via an optional download
and install. EXFAT supports stuff like a 1MB cluster or so, and
EXFAT was designed with Flash storage devices in mind. But again,
there's really no excuse for the poor performance. EXFAT is native
to Windows 7, no installing it as an addon there.

I could use a Sandisk Extreme or Sandisk Extreme Pro and this transfer
would be finished in two minutes, tops.

You can put USB3 on a WinXP era machine, using a plugin card with
a NEC USB3 chip on it. They still sell PCIe versions of those
(a poster bought one a month ago). The NEC chip was one of the few
to include WinXP driver support.

The PCI card version of such things (100MB/sec limit) will be
very hard to find. The bridge chip company got bought up and crushed,
and bridge chips might cost 5x as much as they used to, which has
caused distortion of the computer industry products in response.
This is why they don't put bifurcation logic on motherboards any more.
One company made that happen, via piggish practice.

If you wanted USB3 on your Windows XP machine, and it only had PCI
slots, you needed to be upgrading around ten years ago. That
was back when the PCI to PCIe x1 bridge chip might have been
five bucks or so. And then the bridge chip feeds the NEC chip,
on the PCI card.

Paul

"R.Wieser" <address@not.available> wrote

| I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
| copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
| that the whole thing slows down to a crawl (about 5 GByte consisting of
| 50.000 files in over eight hours). I've also tried a another, different
USB
| stick and see the same thing happen.
|

I don't know any solution. I have noticed that XP doesn't
do USB well. Maybe because it's USB2? Win7 is much faster,
with particular effort. I assume the problem is the USB drivers.
The only other thing I can think of would be if you were using
some kind of AV program. Most of those seem to default to
scanning everything you touch.

Paul,

Thanks for the suggestions.

The reason why I copy files is that I can than access them using any kind of
filebrowser - even a simple one which doesn't understand the concept of ZIP
folders.

As for the cluster size, the second USB stick was new, and as such I assume
it was pre-formatted with the optimum block size. Also, it was NTFS (I had
hoped it would make a difference, but it looks like it doesn't)

And there is a problem with that suggestion : Years ago, when I thought of
the same (micro-SD connected to a DOS 'puter), I was unable to get that
optimum from the USB stick itself. No matter which drive interrogation
function I looked at, none of them returned the size of the FLASH chips
memory blocks . :-(

> I could use a Sandisk Extreme or Sandisk Extreme Pro and this transfer
> would be finished in two minutes, tops.

I assume my (rather old) machine has USB 2.0 tops, but even there a copy to
an USB HD takes just 7 minutes.

> If you wanted USB3 on your Windows XP machine,

:-) Yup.

But the question is not about the speed of the USB connection, but why the
(USB attached) memory-stick is so much slower (as in: more than 60 times)
than the (USB attached) HD.

I'll re-look into that cluster size though. Who knows ...

Regards,
Rudy Wieser

Mayayana,

> Maybe because it's USB2?

:-) Thats why I mentioned that I also tried an USB HD drive. If it would
be the USB connection itself it would also be slow. It isn't. When I
tried just now it copied the same thing in about 7 minutes. Thats over *60
times* faster. :-|

> The only other thing I can think of would be if you were
> using some kind of AV program.

In that case I should have seen it work its ass off in the task manager.
Which I didn't. 0% usage by any program, 99% idle time.

Regards,
Rudy Wieser

On 12.01.22 12:28, R.Wieser wrote:
> Hello all,
>
> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
> that the whole thing slows down to a crawl (about 5 GByte consisting of
> 50.000 files in over eight hours). I've also tried a another, different USB
> stick and see the same thing happen.
>
> Windows task-manager shows my programs procesor usage as zero percent.
>
> Remark: I've used the same program to copy even larger quantities of files
> to an USB harddisk, and have not experienced any kind of slowdown there.
>
> I know that an USB stick is rather slow in comparision to a harddisk, but
> the above is just ludicrous. :-(
>
> Does anyone have an idea what is going on and how to speed the whole thing
> up ?
>
> Regards,
> Rudy Wieser
>
> P.s.
> I remember having had similar problems while drag-and-dropping the source
> folder onto the USB stick. IOW, not a problem with the program itself.
>
>
Dump the files in a zip archive(is quite fast), and
put the zip file on the stick.
That also increases the life of the stick.

On Wed, 12 Jan 2022 12:28:35 +0100, "R.Wieser" <address@not.available>
wrote:

>Hello all,
>
>I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
>copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
>that the whole thing slows down to a crawl (about 5 GByte consisting of
>50.000 files in over eight hours). I've also tried a another, different USB
>stick and see the same thing happen.
>
>Windows task-manager shows my programs procesor usage as zero percent.
>
>Remark: I've used the same program to copy even larger quantities of files
>to an USB harddisk, and have not experienced any kind of slowdown there.
>
>I know that an USB stick is rather slow in comparision to a harddisk, but
>the above is just ludicrous. :-(
>
>Does anyone have an idea what is going on and how to speed the whole thing
>up ?

My totally unscientific observation is that writing to a USB2 device
starts out fast, and when it inevitably slows to a crawl I notice that
the device is blazing hot. If I allow it to cool down, the high write
speed returns until it gets hot again. Is there a correlation between
write speed and heat? *shrug* I don't know, but I can consistently
duplicate that behavior here on my devices.

Char,

> My totally unscientific observation is that writing to a USB2 device
> starts out fast, and when it inevitably slows to a crawl I notice
> that the device is blazing hot.

I tried checking the sticks temperature (removed the "sleeve") and regulary
touched both the processor as well as the flash chip on the other side. I
would not call it hot by any means, just a bit warm.

> If I allow it to cool down,

I have to try to add a keystroke to halt the copying temporarily and see
what happens.

(@all)

And by the way, it looks like the slowing-down is not gradually (as I would
expect when a device gets hotter), but suddenly. It took 45 minutes to
copy about 4 GByte, and after that each file takes 6 seconds.

Odd to say the least ...

Regards,
Rudy Wieser

On 1/13/2022 4:53 AM, R.Wieser wrote:
> Char,
>
>> My totally unscientific observation is that writing to a USB2 device
>> starts out fast, and when it inevitably slows to a crawl I notice
>> that the device is blazing hot.
>
> I tried checking the sticks temperature (removed the "sleeve") and regulary
> touched both the processor as well as the flash chip on the other side. I
> would not call it hot by any means, just a bit warm.
>
>> If I allow it to cool down,
>
> I have to try to add a keystroke to halt the copying temporarily and see
> what happens.
>
> (@all)
>
> And by the way, it looks like the slowing-down is not gradually (as I would
> expect when a device gets hotter), but suddenly. It took 45 minutes to
> copy about 4 GByte, and after that each file takes 6 seconds.
>
> Odd to say the least ...
>
> Regards,
> Rudy Wieser
>
>

I bet the problem does not occur if you use Robocopy or
if you do the copy from a Command Prompt window.

It might be a side-effect of how File Explorer handles
the file records it is reading. All it takes is around
50,000 files in a folder to "confuse" the File Explorer.

I discovered this, by converting movies to a series of
JPG files (for usage with AVIdemux 2.5). Attempts to
edit the movie, by removing the first thousand files and
deleting them in the trash, resulted in the other 49,000
not refreshing properly in the File Explorer window.

On Windows XP, Robocopy can be installed via a download.

On Win7 it would be a built-in command. Each OS has a
different version of Robocopy, with slight differences in
parameters to be passed.

Robocopy is a folder copying program, so it expects
two folders as arguments.

Paul

Paul,

> I bet the problem does not occur .... if you do the copy from a Command
> Prompt window.

Actually, my program is console based. :-)

> On Windows XP, Robocopy can be installed via a download.

I've heard of RoboCopy. But alas, I have much more fun trying to create my
own programs for it (as mentioned, using the CopyFile function outof
kernel32).

Also, as mentioned, the problem seems to be related to the USB stick, as
doing the same copy to an USB (spinning rust) disk doesn't show any kind of
slowdown problems.

Today I've been testing a number different things, including throwing quite
a number of files away and restart the copying (which skips files that are
already there). Alas, once the copying starts to get slow (a steady 6
seconds between files) it stays that way. Not even taking the stick out and
letting it "rest" for half an hour (and than restart). A reformat (of the
"quick" type) does allow it to become fast again though.

By the way, I said a second, NTFS formatted stick also got slow. I
just/might/ have been a bit too hasty there. Although /much/ slower than
the USB HD this morning it finished a copy in about 50 minutes.

I've just reformatted the origional stick to NTFS and have started another
copy. I'm currently waiting to see how it goes ...

.....

Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
5 GByte) having been copied. As such it does /much/ worse than the second
stick.

I'm currently assuming that I was just unlucky enough to have bought a wonky
USB stick. I think the trash is the correct place for it.

Just why is Murphy so nasty to me ? The last time I assumed that I did my
homework it turned out that I forgot something, and now I'm trying to put
the blame on me (my program not doing its thing) it turns out to be
something I have zero control over ...

Oh wait, IIRC that is one of Murpy's other rules. :-)

Regards,
Rudy Wieser

On 13.01.2022 17:16, R.Wieser wrote:

> Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
> 5 GByte) having been copied. As such it does/much/ worse than the second
> stick.

What is the setting for "Enfernungsrichtlinie" in properties of the disk?

[x] Schnelles Enfernen (Standard)
Deaktiviert den Schreibcache auf dem Datenträger und in
Window. Das Gerät kann jedoch auch ohne das Symbol "Sicheres
Entfernen" sicher entfert werden.

[ ] Bessere Leistung
Aktiviert den Schreibcache in Windows ....

Did you try to copy 1 byte files. Maybe without cache the writing
of the directory for each file slows down the copy more than the
data transfer itself.

On Thu, 13 Jan 2022 10:53:55 +0100, "R.Wieser" <address@not.available>
wrote:

>Char,
>
>> My totally unscientific observation is that writing to a USB2 device
>> starts out fast, and when it inevitably slows to a crawl I notice
>> that the device is blazing hot.
>
>I tried checking the sticks temperature (removed the "sleeve") and regulary
>touched both the processor as well as the flash chip on the other side. I
>would not call it hot by any means, just a bit warm.

Thanks for checking. On at least two of my USB2 sticks, they get too hot
to touch if I do extended writes, but it sounds like you've disproved my
theory.

Herbert,

> What is the setting for "Enfernungsrichtlinie" in properties of the disk?

When the memory stick was FAT32 formatted stick it was the first. Later I
formatted it to NTFS, which a choice you only get when the second is
enabled.

In short, I have had both for that stick. Didn't make much of a difference.

Regards,
Rudy Wieser

Char,

> Thanks for checking.

Thank you for mentioning it. I had not considered it as a possible cause.

> On at least two of my USB2 sticks, they get too hot to touch
> if I do extended writes, but it sounds like you've disproved
> my theory.

Too many different companies making them. I've had a a few of a brand which
would crap out after just a few uses. On the other hand I've still got a 1
GByte stick which got weekly use and now is over a decade old.

Regards,
Rudy Wieser

On 2022-01-13, R.Wieser <address@not.available> wrote:

> Herbert,
>
>> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
>
> When the memory stick was FAT32 formatted stick it was the first. Later I
> formatted it to NTFS, which a choice you only get when the second is
> enabled.
>
> In short, I have had both for that stick. Didn't make much of a difference.

I had a thumb drive formatted as NTFS for quite some time. The performance
really sucked, but I stuck with it because FAT32 only stores time stamps to
the nearest 2 seconds, making time stamp comparison (and makefiles) problematic.

However, as you noted earlier, I also noticed that copying large numbers of
small files to a flash drive runs like molasses through a pinhole in January,
so I resorted to the solution that has already been mentioned here: zip the
files on your hard drive and then copy the .zip file to the flash drive.
Since zip files preserve time stamps to the second, there was no longer
any reason not to revert the flash drive to FAT32 and get even better speed.

Zipping directly to the flash drive causes many small work files to be
created and scratched on it, again resulting in abominable performance.

--
/~\ Charlie Gibbs | Microsoft is a dictatorship.
\ / <cgibbs@kltpzyxm.invalid> | Apple is a cult.
X I'm really at ac.dekanfrus | Linux is anarchy.
/ \ if you read it the right way. | Pick your poison.

In alt.windows7.general Charlie Gibbs <cgibbs@kltpzyxm.invalid> wrote:
> On 2022-01-13, R.Wieser <address@not.available> wrote:

> > Herbert,
> >
> >> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
> >
> > When the memory stick was FAT32 formatted stick it was the first. Later I
> > formatted it to NTFS, which a choice you only get when the second is
> > enabled.
> >
> > In short, I have had both for that stick. Didn't make much of a difference.

> I had a thumb drive formatted as NTFS for quite some time. The performance
> really sucked, but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and makefiles) problematic.

What about exFAT? NTFS is a problem to use on devices that don't know it.
--
Slammy new week as expected. Lots of spams again! 2022 isn't any better and different so far. :(
Note: A fixed width font (Courier, Monospace, etc.) is required to see this signature correctly.
/\___/\ Ant(Dude) @ http://aqfl.net & http://antfarm.home.dhs.org.
/ /\ /\ \ Please nuke ANT if replying by e-mail.
| |o o| |
\ _ /
( )

Charlie,

> but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and
> makefiles) problematic.

I encountered a similar problem : even a just-copied file could be a number
of seconds different from the origional. As I'm not in the habit of running
backups while working on my machine I decided to allow for a 10 second
difference in timestamps.

That, and problems with summer/wintertime differences. :-\

> However, as you noted earlier, I also noticed that copying large numbers
> of small files to a flash drive runs like molasses through a pinhole in
> January,

Hmmm... Although comperativily it takes an USB stick quite a bit longer
than a USB HD, I do not really see any slow-down over the duration of the
copying itself.

> so I resorted to the solution that has already been mentioned here: zip
> the files on your hard drive and then copy the .zip file to the flash
> drive.

I also considered that, but that would have thrown quite a spanner in what
my program was actually build for : checking for changed files and
updating/deleting them on the copy.

Also, I wanted to be sure that regardless of how much space I had left on
the source I would always be able to make a backup or update it.

Next to that I also considered the possibility of a write or read error (bad
sector) could trash a compressed file, with little-to-no chance of recovery.

> Zipping directly to the flash drive causes many small work files to be
> created and scratched on it, again resulting in abominable performance.

Not only zipping to an USB stick. I also noticed a much worse performance
when extracting from a ZIP stored on an USB stick.

But yes, the effective need for an intermediate, to-be-transferred ZIP file
is what I ment with my "how much space I had left on the source" remark.

Regards,
Rudy Wieser

Wieser <address@not.available> wrote:
> Herbert,
>
> > What is the setting for "Enfernungsrichtlinie" in properties of the disk?

Google Translate broken!? :-) (Yes, GT can handle Herbert's small spelling
error, "Entfernungsrichtlinie".)

It means "removal policy".

<https://translate.google.com/?sl=de&tl=en&text=Enfernungsrichtlinie&op=translate>

But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
fast performance) removal policy, at least they don't on my Windows 8.1
system.

(Probably USB *disks* have a (fast removal versus fast performance)
removal policy, but I haven't checked.)

[...]

On Fri, 14 Jan 2022 09:57:44 +0100, "R.Wieser" <address@not.available>
wrote:

>Charlie,
>
>> but I stuck with it because FAT32 only stores time stamps to
>> the nearest 2 seconds, making time stamp comparison (and
>> makefiles) problematic.
>
>I encountered a similar problem : even a just-copied file could be a number
>of seconds different from the origional. As I'm not in the habit of running
>backups while working on my machine I decided to allow for a 10 second
>difference in timestamps.
>
>That, and problems with summer/wintertime differences. :-\
>
>> However, as you noted earlier, I also noticed that copying large numbers
>> of small files to a flash drive runs like molasses through a pinhole in
>> January,
>
>Hmmm... Although comperativily it takes an USB stick quite a bit longer
>than a USB HD, I do not really see any slow-down over the duration of the
>copying itself.
>
>> so I resorted to the solution that has already been mentioned here: zip
>> the files on your hard drive and then copy the .zip file to the flash
>> drive.
>
>I also considered that, but that would have thrown quite a spanner in what
>my program was actually build for : checking for changed files and
>updating/deleting them on the copy.
>
>Also, I wanted to be sure that regardless of how much space I had left on
>the source I would always be able to make a backup or update it.
>
>Next to that I also considered the possibility of a write or read error (bad
>sector) could trash a compressed file, with little-to-no chance of recovery.

One of the reasons that I like to use containers, such as zip or rar, is
that data corruption is much easier to spot. With bare files that aren't
in a container, bit rot can go undetected for a long period of time and
once found, it can be very difficult to correct.

My main tool for detecting bit rot, and then *correcting* it, is
Quickpar. http://www.quickpar.org.uk/
I use it almost daily and highly recommend it. It works equally well on
non-containerized files, but I mostly use it with multipart rar files.

<snip>

Frank,

> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.

On my XPsp3 machine I can choose either. Though when I set it to "fast
removal" I can't select NTFS when formatting (and vise-verse). Which, long
ago, confused the heck outof me. :-o

Regards,
Rudy Wieser

P.s.
The way you quoted it made it look as if /I/ posted that "what if" line ...

Char,

> One of the reasons that I like to use containers, such as zip or rar,
> is that data corruption is much easier to spot.

You mean, the first time you try to use /any/ file outof it it barfs and
refuses to let you access most, if not all of its contained files ? :-|

> With bare files that aren't in a container, bit rot can go undetected
> for a long period ...

True. But than I only lose the affected file(s), not the whole (or
many/most of) the container.

.... and once found, it can be very difficult to correct.

Harder than fixing an bit-rotted container ? :-p

But granted, the sooner you become aware of a problem with a backup the
better.

> My main tool for detecting bit rot, and then *correcting* it,
> is Quickpar.

:-) You make it sound as if you can just throw a file at it and it will
detect and correct bitrot in it. Which is ofcourse not quite the way it
works.

Though granted, pulling the files thru such a program (which adds
detecting/recovery information to the file) before saving them to a backup
does enable the files to survive a decent amount of bitrot or other similary
small damage.

Regards,
Rudy Wieser

On 14.01.2022 16:16, Frank Slootweg wrote:

> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.

in explorer right-click on the external usb drive
select "Properties"
on the Hardware tab, select your device and select "Properties"
select "Change Settings" on the General tab.
select the "Policies" tab

Herbert Kleebauer <klee@unibwm.de> wrote:
> On 14.01.2022 16:16, Frank Slootweg wrote:
>
> > But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> > fast performance) removal policy, at least they don't on my Windows 8.1
> > system.
>
>
> in explorer right-click on the external usb drive
> select "Properties"
> on the Hardware tab, select your device and select "Properties"
> select "Change Settings" on the General tab.
> select the "Policies" tab

Thanks!

I could remember that functionality, but could not find it again.

In hindsight, I apparently overlooked several things, both on the
'Hardware' tab and on the second level 'Properties' popup/window.

So thanks again.

R.Wieser <address@not.available> wrote:
> Frank,

[...]

> P.s.
> The way you quoted it made it look as if /I/ posted that "what if" line ...

Sorry, a case of sloppy reading! I thought you didn't understand
Herbert's German text.

I will now crawl back under my rock. :-)

On Fri, 14 Jan 2022 18:54:24 +0100, "R.Wieser" <address@not.available>
wrote:

>Char,
>
>> One of the reasons that I like to use containers, such as zip or rar,
>> is that data corruption is much easier to spot.
>
>You mean, the first time you try to use /any/ file outof it it barfs and
>refuses to let you access most, if not all of its contained files ? :-|

I suppose so, but I'd say that's preferable to the alternative, which is
not knowing. With individual files, I suppose each file would have to be
loaded in some way, checking for obvious corruption or error messages,
or in the case of system files like .dll and .exe just give it your best
shot, but all of the major container types give you an easy method to
detect corruption of the container itself. That's a huge benefit.

>> With bare files that aren't in a container, bit rot can go undetected
>> for a long period ...
>
>True. But than I only lose the affected file(s), not the whole (or
>many/most of) the container.

You don't actually have to lose any data. Quickpar can usually recover
that stuff, if it's used correctly.

>... and once found, it can be very difficult to correct.
>
>Harder than fixing an bit-rotted container ? :-p

Good point. Much harder to spot damage in individual files, but once
spotted, probably equally easy to correct if you've planned ahead.

>But granted, the sooner you become aware of a problem with a backup the
>better.
>
>> My main tool for detecting bit rot, and then *correcting* it,
>> is Quickpar.
>
>:-) You make it sound as if you can just throw a file at it and it will
>detect and correct bitrot in it. Which is ofcourse not quite the way it
>works.

Right, you have to plan ahead. You have to ask yourself, are these files
something that I care about? If so, how much damage do I want to be
prepared to correct? The answer to the second question, using Quickpar,
can be anywhere from 0% to 100%, but it's a decision that has to be made
before any damage occurs.

>Though granted, pulling the files thru such a program (which adds
>detecting/recovery information to the file) before saving them to a backup
>does enable the files to survive a decent amount of bitrot or other similary
>small damage.

Quickpar creates additional files (parity files) that you can use to
detect and correct damage. The actual files that you care about are not
touched in any way unless and until it becomes necessary to repair them.
You also don't have to store the parity files alongside the files that
they're protecting. You can, of course, but you can also store them, or
a second copy of them since they're relatively small, in another
location.